Fundamentals Of Database Engineering (updated 11/2022)

Learn ACID, Indexing, Partitioning, Sharding, Concurrency control, Replication, DB Engines, Best Practices and More!

What you'll learn

Learn and understand ACID Properties

Database Indexing

Database Partitioning

Database Replication

Database Sharding

Database Cursors

Concurrency Control (Optimistic, Pessimistic)

B-Trees in Production Database Systems

Database System Designs

Difference between Database Management System, Database Engine and Embedded database

Database Engines such as MyISAM, InnoDB, RocksDB, LevelDB and More

Benefits of Using one database engine over the other

Switching Database Engines with MySQL

Database Security

Homomorphic Encryption

Requirements

Have worked with databases before but wish to get deeper understanding

Basic SQL knowledge

Description

Database Engineering is a very interesting sector in software engineering. If you are interested in learning about database engineering you have come to the right place. I have curated this course carefully to discuss the Fundamental concepts of database engineering.This course will not teach you SQL or programming languages, however, it will teach you skillsets and patterns that you can apply in database engineering. A few of the things that you will learn are Indexing, Partitioning, Sharding, Replication, b-trees in-depth indexing, Concurrency control, database engines and security, and much more. I believe that learning the fundamentals of database engineering will equip you with the necessary means to tackle difficult and challenging problems yourself. I always compare engineering to math, you never memorize specific formulas and equations, you know the basic proves and derive and solve any equation one throws at you. Database engineering is similar, you can't possibly say MongoDB is better than MySQL or Postgres is better than Oracle. Instead, you learn your use case and by understanding how each database platform does its own trade-offs you will be able to make optimal decisions. One other thing you will learn in this course is the lowest database interface that talks to the OS which is the database engine. Database engines or storage engines or sometimes even called embedded databases is a software library that a database management software uses to store data on disk and do CRUD (create update delete) Embedded means move everything in one software no network client-server. In this video course, I want to go through the few popular database engines, explain the differences between them, and finally, I want to spin up a database and change its engine and show the different features of each engine. Enjoy the course.

Overview

Section 1: Course Updates

Lecture 1 Welcome to the Course

Lecture 2 Course Update Oct 2020

Lecture 3 Course Update April 2021

Lecture 4 Course Update December 2021

Lecture 5 Note about Docker

Section 2: ACID

Lecture 6 Introduction to ACID

Lecture 7 What is a Transaction?

Lecture 8 Atomicity

Lecture 9 Isolation

Lecture 10 Consistency

Lecture 11 Durability

Lecture 12 ACID by Practical Examples

Lecture 13 Phantom Reads

Lecture 14 Serializable vs Repeatable Read

Lecture 15 Eventual Consistency

Section 3: Understanding Database Internals

Lecture 16 How tables and indexes are stored on disk (MUST WATCH before continue)

Lecture 17 Row-Based vs Column-Based Databases

Lecture 18 Primary Key vs Secondary Key - What you probably didn't know

Section 4: Database Indexing

Lecture 19 Create Postgres Table with a million Rows (from scratch)

Lecture 20 Getting Started with Indexing

Lecture 21 Understanding The SQL Query Planner and Optimizer with Explain

Lecture 22 Bitmap Index Scan vs Index Scan vs Table Scan

Lecture 23 Key vs Non-Key Column Database Indexing

Lecture 24 Index Scan vs Index Only Scan

Lecture 25 Combining Database Indexes for Better Performance

Lecture 26 How Database Optimizers Decide to Use Indexes

Lecture 27 Create Index Concurrently - Avoid Blocking Production Database Writes

Lecture 28 Bloom Filters

Lecture 29 Working with Billion-Row Table

Lecture 30 Article - The Cost of Long running Transactions

Lecture 31 Article - Microsoft SQL Server Clustered Index Design

Section 5: B-Tree vs B+Tree in Production Database Systems

Lecture 32 B-Tree Section's Introduction & Agenda

Lecture 33 Full Table Scans

Lecture 34 Original B-Tree

Lecture 35 How the Original B-Tree Helps Performance

Lecture 36 Original B-Tree Limitations

Lecture 37 B+Tree

Lecture 38 B+Tree DBMS Considerations

Lecture 39 B+Tree Storage Cost in MySQL vs Postgres

Lecture 40 B-Tree Section's Summary

Section 6: Database Partitioning

Lecture 41 Introduction to Database Partitioning

Lecture 42 What is Partitioning?

Lecture 43 Vertical vs Horizontal Partitioning

Lecture 44 Partitioning Types

Lecture 45 The Difference Between Partitioning and Sharding

Lecture 46 Preparing: Postgres, Database, Table, Indexes

Lecture 47 Execute Multiple Queries on the Table

Lecture 48 Create and Attach Partitioned Tables

Lecture 49 Populate the Partitions and Create Indexes

Lecture 50 Class Project - Querying and Checking the Size of Partitions

Lecture 51 The Advantages of Partitioning

Lecture 52 The Disadvantages of Partitioning

Lecture 53 Section Summary - Partitioning

Lecture 54 How to Automate Partitioning in Postgres

Section 7: Database Sharding

Lecture 55 Introduction to Database Sharding

Lecture 56 What is Database Sharding?

Lecture 57 Consistent Hashing

Lecture 58 Horizontal partitioning vs Sharding

Lecture 59 Sharding with Postgres

Lecture 60 Spin up Docker Postgres Shards

Lecture 61 Writing to a Shard

Lecture 62 Reading from a Shard

Lecture 63 Advantages of Database Sharding

Lecture 64 Disadvantages of Database Sharding

Lecture 65 Database Sharding Section Summary

Lecture 66 When Should you consider Sharding your Database?

Section 8: Concurrency Control

Lecture 67 Shared vs Exclusive Locks

Lecture 68 Dead Locks

Lecture 69 Two-phase Locking

Lecture 70 Solving the Double Booking Problem (Code Example)

Lecture 71 Double Booking Problem Part 2 ( Alternative Solution and explination)

Lecture 72 SQL Pagination With Offset is Very Slow

Lecture 73 Database Connection Pooling

Section 9: Database Replication

Lecture 74 Introduction to Database Replication

Lecture 75 Master/Standby Replication

Lecture 76 Multi-master Replication

Lecture 77 Synchronous vs Asynchronous Replication

Lecture 78 Replication Demo with Postgres 13

Lecture 79 Pros and Cons of Replication

Section 10: Database System Design

Lecture 80 Twitter System Design Database Design

Lecture 81 Building a Short URL System Database Backend

Section 11: Database Engines

Lecture 82 Introduction

Lecture 83 What is a Database Engine?

Lecture 84 MyISAM

Lecture 85 InnoDB

Lecture 86 XtraDB

Lecture 87 SQLite

Lecture 88 Aria

Lecture 89 BerkeleyDB

Lecture 90 LevelDB

Lecture 91 RocksDB

Lecture 92 Popular Database Engines

Lecture 93 Switching Database Engines with mySQL

Section 12: Database Cursors

Lecture 94 What are Database Cursors?

Lecture 95 Server Side vs Client Side Database Cursors

Lecture 96 Inserting Million Rows with Python in Postgres using Client Side Cursor

Lecture 97 Querying with Client Side Cursor

Lecture 98 Querying with Server Side Cursor

Lecture 99 Pros and Cons of Server vs Client Side Cursors

Lecture 100 Article - Server Side Cursor Types in SQL Server

Section 13: Database Security

Lecture 101 How to Secure Your Postgres Database by Enabling TLS/SSL

Lecture 102 Deep Look into Postgres Wire Protocol with Wireshark

Lecture 103 Deep Look Into MongoDB Wire Protocol with Wireshark

Lecture 104 What is the Largest SQL Statement that You can Send to Your Database

Lecture 105 Best Practices Working with REST & Databases

Lecture 106 Database Permissions and Best Practices for Building REST API

Section 14: Homomorphic Encryption - Performing Database Queries on Encrypted Data

Lecture 107 Introduction to Homomorphic Encryption

Lecture 108 What is Encryption?

Lecture 109 Why Can't we always Encrypt?

Lecture 110 What is Homomorphic Encryption

Lecture 111 Homomorphic Encryption Demo

Lecture 112 Clone and Build the Code

Lecture 113 Going Through the Code and the Database

Lecture 114 Searching The Encrypted Database

Lecture 115 Is Homomorphic Encryption Ready?

Section 15: Answering your Questions

Lecture 116 Snapshot and Repeatable Read Isolation difference?

Lecture 117 I have an Index why is the database doing a full table scan?

Lecture 118 Why Databases Read Pages instead of Rows?

Lecture 119 How does Indexing a column with duplicate values work?

Lecture 120 Should I drop unused indexes?

Lecture 121 Why use serializable Isolation Level when we have SELECT FOR UPDATE?

Lecture 122 Can I use the same database connection for multiple clients?

Lecture 123 Do I need a transaction if I'm only reading?

Lecture 124 Why does an update in Postgres touches all indexes?

Lecture 125 What is the value of bitmap index scan?

Lecture 126 What does Explain Analyze actually do?

Section 16: Database Discussions

Lecture 127 SELECT COUNT (*) can impact your Backend Application performance, here is why

Lecture 128 How does the Database Store Data On Disk?

Lecture 129 Is QUIC a Good Protocol for Databases?

Lecture 130 What is a Distributed Transaction?

Lecture 131 Hash Tables and Consistent Hashing

Lecture 132 Indexing in PostgreSQL vs MySQL

Lecture 133 Why Uber Moved from Postgres to MySQL (Discussion)

Lecture 134 Can NULLs Improve your Database Queries Performance?

Lecture 135 Write Amplification Explained in Backend Apps, Database Systems and SSDs

Lecture 136 Optimistic vs Pessmistic Concurrency Control

Lecture 137 WAL, Redo and Undo logs

Section 17: Archived Lectures

Lecture 138 Introduction to ACID (Archived)

Lecture 139 What is a Transaction? (Archived)

Lecture 140 Atomicity (Archived)

Lecture 141 Isolation (Archived)

Lecture 142 Consistency (Archived)

Lecture 143 Durability (Archived)

Software Engineers and Database Engineers