Designing Data Warehouse From Scratch: End To End Guide

Master the art of building a data warehouse from the ground up with this comprehensive end to end guide

What you'll learn

Begin with the basics of data warehousing, understanding its significance, and exploring data modeling concepts to design the logical view of your DW

Get hands-on with Amazon Redshift, learning how to set up, manage, and optimize your data warehouse cluster for performance and cost-efficiency.

Translate your logical model into a robust physical model in Redshift, mastering the art of schema design, table creation, and query optimization.

Explore various data ingestion methods and learn how to transform raw data into insightful, analytical information using ETL and ELT processes.

Implement the cutting-edge medallion architecture in your data warehouse to streamline data processing and enhance analytical capabilities.

Requirements

Basic Understanding of AWS Services

Basic SQL Technical Skills

Basic Understanding of Datawarehouse and Data Modelling

Description

This is the ultimate course for aspiring and experienced data engineers alike. This comprehensive course is meticulously designed to take you through the entire process of creating a data warehouse, from conceptualization to implementation, using Amazon Redshift and the medallion architecture.This course is designed for data professionals, analysts, and anyone interested in mastering data warehousing, this course will equip you with the skills and knowledge needed to create robust and scalable data warehouses from the ground up.Through hands-on lessons and real-world examples, you'll learn how to:Establish the logical architecture and design of a data warehouse, ensuring a solid foundation for your project.Create and configure a data warehouse in the cloud using Amazon Redshift, leveraging the power and scalability of cloud computing.Implement various data loading strategies to efficiently populate your data warehouse with relevant data from multiple sources.Convert the logical design into a physical implementation, setting up the necessary infrastructure and configurations on AWS.Implement the Medallion Architecture in your Redshift data warehouse, following industry best practices for data modeling and organization.Write comprehensive test cases and implement data quality measures to ensure the integrity and reliability of your data warehouse.Utilize DBT (Data Build Tool) for seamless data transformation, analysis, and documentation, streamlining the entire data warehousing process.This course is suitable for all regardless of your job function; a data analyst, business intelligence professional, or simply someone passionate about data, this course will provide you with the skills and knowledge required to design and build a robust data warehouse from scratch, enabling you to unlock valuable insights and drive data-driven decision-making within your organisation.Enroll now and embark on a journey to become a data warehousing expert with "Designing Data Warehouse From Scratch: End to End Guide".

Overview

Section 1: Introduction

Lecture 1 Prerequisites

Lecture 2 Steps Involved in Creating a Data Warehouse

Lecture 3 Discussing the business usecase

Section 2: Creating Logical View of Datawarehouse

Lecture 4 Creating the Logical Architecture View of the DW

Section 3: Creating a Datawarehouse in the Cloud

Lecture 5 Creating a Redshift Cluster on AWS

Lecture 6 Finalising the creation Redshift Cluster on AWS

Section 4: Data Loading into Datawarehouse

Lecture 7 Data Warehouse Data Loading Strategies

Section 5: Converting Logical View to Physical View

Lecture 8 Creating Security Groups on AWS

Lecture 9 Connecting and Managing your Redshift Cluster

Lecture 10 Creating Glue crawler to create table from S3

Lecture 11 Uploading data to s3 and crawling them with Glue crawler

Lecture 12 Verifying the data loaded into Data Warehouse

Lecture 13 Creating VPC, Subnets, Internet Gateways on AWS

Lecture 14 Creating Subnet Groups on Redshift for VPC

Section 6: Implementing Medallion Architecture in Redshift Datawarehouse

Lecture 15 Creating DBT Project and Implementing Medallion Architecture

Lecture 16 Finishing up on the Gold layer Facts

Section 7: Data Quality and Testing

Lecture 17 Writing Test Cases for Dimension Layers

Lecture 18 Writing Test Cases for Gold Layer

Lecture 19 Writing Data Quality Test Cases

Section 8: DBT Analyses

Lecture 20 Working with DBT Analyses

Section 9: DBT Documentation

Lecture 21 Writing DBT Documentation for the Project

Section 10: Course Resources

Lecture 22 Full Code

Suitable to everyone interested in Big Data and Data Warehouse Modelling