PostgreSQL Replication, High Availability HA and Scalability

Solutions for Scaling Postgres with Master-Slave Replication, PgBouncer, PgPool II, HAProxy, Partitioning, Sharding

What you'll learn:
Assess your scaling needs
How to scale reads using Replication and Load-Balancing
Which is the best Replication solution for a certain use case
How to manage database connections with PgBouncer connection pooler
How to make use of multiple PostgreSQL instances in the cloud (Google Cloud)
How to achieve High-Availability
How to perform Automatic Failover using PgPool II
How to scale writes using Partitioning and Sharding

Requirements:
You need access to a Windows/Mac/Linux PC with 10GB of free disk space
Basic familiarity with database objects such as tables and indexes is expected
Some familiarity with Linux will be helpful

Description:
PostgreSQL is one of the most powerful and easy-to-use database management systems. It has strong support from the community and is being actively developed with a new release every year.
PostgreSQL supports the most advanced features included in SQL standards. It also provides NoSQL capabilities and very rich data types and extensions. All of this makes PostgreSQL a very attractive solution in software systems.

In this course, we discussed the problem of building scalable solutions based on PostgreSQL utilizing the resources of several servers. There is a natural limitation for such systems—basically, there is always a compromise between performance, reliability, and consistency. It's possible to improve one aspect, but others will suffer. In this course, we'll see how to find the best match for our use-cases so that we know eactly which aspects need scaling, and avoid the common trade-offs of distributed systems.

Scaling PostgreSQL is a journey. You should come out of this course more prepared to assess your scaling needs and understand how to scale reads and how to scale writes.

Each of this solution presented in this course will improve some aspect of the scalability topic, but each of them will add some complexity, and maybe some limitation or constraint.
We have to ask the right questions to get the system requirements, and this why we dedicated an entire lecture, so that we examine what questions we have to put ourself, before starting the Scaling Journey.
After this course, we should come out more prepared and understand how to scale reads.
We have several options for replication, depending on wether we favor performance or flexibility.
Replication can be used as a backup or a standby solution that would take over in case the main server crashes.
Replication can also be used to improve the performance of a software system by making it possible to distribute the load on several database servers.
Then, if we have one sort of replication in place, we could ask ourself if we want to allow several computers to serve the same data.
To achieve this, we should have a mechanism to distribute the requests. We’ll see here two of the most popular options available.
Next, if the number of database connections is great, then we’ll probably want to use a connection pooler. Again, we’ll cover two options here.
We’ll also see, how to scale writes, and how to make your traffic growth more predictable by adding queuing to your architecture.
Then, we’ll check partitioning for those cases when we have to deal with big tables.
Also, we’ll check sharding to scale writes, and all the complex decisions that come with it.
Finally, we’ll see shortly the multi-master solution, which is a relatively new concept that seems to be promising.
If our goal is to achieve only High availability, or the ability to continue working even in the situation where one part of the cluster fails, we can check out only those solutions.
The pre-requirements for HA is to put in place a replication strategy.
Then, we can use tools to allow a second server to take over quickly, if the primary server fails.

Introduction to Scaling PostgreSQL
Why scale PostgreSQL?
What is Vertical Scaling?
What is Horizontal Scaling?
Read Versus Write Bound Workloads
Why Statistics are essential?
How to enable and make us of Statistics? (Hands-on)
How to scale Postgres for Reads?
How replication helps to scale out?
What are the Load-Balancers?
How to scale Postgres for Writes?
How to make use of Queues?
How could Partitioning and Sharding help in scaling out?
What is the Multi-Master solution about?

Understanding the Limitations of Scaling out PostgreSQL
CAP Theorem Explained
PostgreSQL vs. Cassandra
Use case: CA Systems
Use case: AP Systems

How to use Streaming Replication?
What is Streaming Replication?
Asynchronous vs. Synchronous Replication
How to Initialise Primary Database? (Hands-on)
How to Configuring the Primary for Replication?  (Hands-on)
How to Configuring the Replica Instance?  (Hands-on)
Testing Replication Setup  (Hands-on)

How to use Logical Replication?
What is Logical Replication in Postgres?
Step by step Logical Replication setup
How to setup the servers for Logical Replication? (Hands-on)
How to make a selective Copy of the Data? (Hands-on)
How to Create the Publication? (Hands-on)
How to Create the Subscription? (Hands-on)
Postgres Limitations of Logical Replication
How to Monitoring Logical Replication? (Hands-on)
Best use-cases for using Logical Replication

How to make use of PgBouncer?
What is PgBouncer?
Fundamental concepts of connection pooling
How to build a PgBouncer Setup? (Hands-on)
How to install and configure PgBouncer? (Hands-on)
How to create a basic configuration file for PgBouncer? (Hands-on)
How to connect to PgBouncer? (Hands-on)
Explaining Advanced Settings for Performance
Which are the available Pool Modes?
Executing a benchmark with PgBouncer (Hands-on)

How to scale PostgreSQL in Google Cloud?
Introduction
Key Components on Google Cloud
Key Characteristics of the Architecture
How to create PostgreSQL Instances on Google Cloud?  (Hands-on)
How to create a Google Cloud Engine (GCE) for HAProxy? (Hands-on)
How to configure HAProxy for Load-Balancing? (Hands-on)
Testing Load-Balancing

How to make use of PostgreSQL Partitioning?
What is Partitioning?
Which Tables Need Partitioning?
How should the Tables be Partitioned? 
Declarative vs. Inheritance Partitioning
How to create a Partitioned Table? (Hands-on)
Partitioning Methods

How to Shard PostgreSQL?
What is Sharding?
Pain-Points of Sharding?
What is Second Level Sharding?
What is good Sharding?
How to query across multiple Shards?

How to setup High Availability (HA) on PostgreSQL?
Why High Availability?
Steps to achieve High Availability
Essential Questions to ask before setting-up High Availability
Log-Shipping Replication
Streaming Replication and Logical Replication
Cascading Replication
Synchronous vs. Asynchronous Replication
Automatic Failover and Always-on Strategy
Simple HA Solution Example
Better HA Solution Example

How to make use of PgPool II?
What is PgPool II?
Pgpool-II Features
How to Configure Pgpool-II with Streaming Replication? (Hands-on)
How to setup Streaming Replication? (Hands-on)
How to Configuring Pgpool-II for Load Balancing ? (Hands-on)
Testing load-balancing & read/write separation (Hands-on)
How to Configure Pgpool for PostgreSQL High-Availability? (Hands-on)
How to Configure PostgreSQL Primary Server? (Hands-on)
How to Configure Pgpool-II Server? (Hands-on)
How to Configure PostgreSQL Replica Server? (Hands-on)
Testing The Failover (Hands-on)
How to restore failed nodes? (Hands-on)

Who this course is for:
Software Engineers interested in designing Scalable and HA solutions on top of PostgreSQL
Database Administrators
Everyone interested in building better PostgreSQL applications

More Info