Build Data Engineering Pipelines With Azure Data Factory

Get hands on and build a modern data lake with Azure Data Factory, Data Lake Storage, Azure SQL and more!

What you'll learn

Overview and Background of Azure Big Data Analytics Solutions

Azure Data Factory

Azure Storage Accounts

Azure SQL Databases and Upsert Operations

Integrating Data From Various File Formats

Authentication and Access: Service Principals, Managed Identities and Azure Key Vault

Data Pipelines, Copy Activity, Data Flows, Control Flow and Transformation Activities

Implementing a Modern Data Lake

Control Flow, Parameters and Variables

Real World Application of Azure Data Factory

Requirements

Microsoft Azure Subscription

Basic SQL

Description

Hello students!I am pleased to present this course on one of the most in demand data engineering tools around… Azure Data Factory!As the demand for cloud-based data integration services continues to skyrocket, there is a huge demand for professionals with knowledge of services like Azure Data Factory. By learning Azure Data Factory, users can enhance their skills and increase their job prospects in the field of data engineering and analytics.In this course you will primarily be using Azure Data Factory on Microsoft Azure in addition to other services such as Azure Blob Storage, Azure Data Lake Storage Gen 2 and Azure SQL Database.The course is packed with lectures, code-along videos and a dedicated course project. As an added benefit you will also have lifetime access to all of the lectures…This course will cover the following topics:Azure Storage Solutions such as Azure Blob Storage and Azure Data Lake Gen2 StorageThe basics of Azure Data Factory including the core components such as Linked Services, Datasets, Activities, Data Flows, Pipelines and Integration RuntimesIntegrating data from various file formats such as CSV, JSON and ParquetThe Copy Activity in Azure Data FactoryData Flows and Control Flow and Transformation Activities in Azure Data FactoryOrchestrating Data Integration WorkflowsHow to create Schedules and Triggers to execute your pipelinesHow to use Parameters and Variables with your Linked Services, Datasets and PipelinesHow to use Azure Data Factory with SQL DatabasesAuthentication and Access including Managed Identities, Service Principals and Azure Key Vault

Overview

Section 1: Course Overview and Set Up

Lecture 1 Course Overview

Lecture 2 Azure Big Data Analytics

Lecture 3 Azure Account Set Up

Lecture 4 Azure UI Overview and Creating Your Subscription

Lecture 5 Services and Regions

Lecture 6 Naming Conventions and Creating the Course Resource Group

Lecture 7 Cost Management and Pricing

Lecture 8 Links and Resources

Section 2: Azure Storage

Lecture 9 Azure Storage Solutions Overview

Lecture 10 Classification of Data

Lecture 11 ADF Supported File Formats

Lecture 12 Customer Orders Dataset Download and Overview

Lecture 13 Blueprint for the Data Lake

Lecture 14 Storage Account Creation

Lecture 15 Container Creation and File Upload

Lecture 16 Links and Resources

Section 3: ADF - Overview and Set Up

Lecture 17 Azure Data Factory Overview

Lecture 18 ADF Resource Creation

Lecture 19 ADF User Interface and Components Overview

Lecture 20 Links and Resources

Section 4: Data Movement

Lecture 21 Copy Activity Overview

Lecture 22 Section Objective

Lecture 23 Copy Activity - Order Items Part 1 (Linked Services and Source/Sink Datasets)

Lecture 24 Copy Activity - Order Items Part 2 (Copy Activity and Pipeline)

Lecture 25 Copy Activity - Customers JSON

Lecture 26 Copy Activity - Orders Parquet

Lecture 27 Copy Activity - Stores and Products JSON

Lecture 28 Organising Data Factory Objects in Folders

Lecture 29 Chaining Activities

Lecture 30 Working with Multiple Files as a Source and Copy Behaviour

Lecture 31 Links and Resources

Section 5: Data Transformation

Lecture 32 Data Flow Overview

Lecture 33 Section Objective

Lecture 34 Data Flow Demo and User Interface

Lecture 35 Data Flow Debug (Pricing)

Lecture 36 Data Flow Debug (Row Limit)

Lecture 37 Source and Sink Partitioning

Lecture 38 Select Transformation

Lecture 39 Cast Transformation

Lecture 40 Import Projection

Lecture 41 Derived Column Transformation

Lecture 42 Raw to Cleansed Data Flow Requirement

Lecture 43 Raw to Cleansed Data Flow Walkthrough

Lecture 44 Partitioned Datasets

Lecture 45 Filter and Sort Transformations

Lecture 46 Aggregate Transformation

Lecture 47 Join Transformation

Lecture 48 Conditional Split and Union Transformations

Lecture 49 Cleansed to Structured Data Flow Requirement

Lecture 50 Cleansed to Structured Data Flow Walkthrough

Lecture 51 Structured to Analytics Data Flow Requirement

Lecture 52 Structured to Analytics Data Flow Walkthrough

Lecture 53 Links and Resources

Section 6: Scheduling and Chaining Pipelines for Execution

Lecture 54 Execute Pipeline Activity

Lecture 55 Schedule Triggers

Lecture 56 Storage Events Trigger

Lecture 57 Tumbling Window Triggers

Lecture 58 Links and Resources

Section 7: Control Flow, Parameters and Variables

Lecture 59 Data Lake Preparation

Lecture 60 Delete Activity

Lecture 61 Get Metadata Activity

Lecture 62 Lookup Activity

Lecture 63 If Condition and Switch Activities

Lecture 64 Introduction to Parameters in ADF

Lecture 65 Introduction to Variables in ADF

Lecture 66 ForEach Activity

Lecture 67 Metadata Based Approach (Example)

Lecture 68 Course Project Activity: Dynamic Datasets and Folder Structure

Lecture 69 Links and Resources

Section 8: Azure SQL Database for Structured Data

Lecture 70 Section Overview

Lecture 71 Creating our Azure SQL Resource

Lecture 72 Creating our Schema and Tables

Lecture 73 Adding a Unique Identifier Column to Structured Orders Data

Lecture 74 Creating Linked Service and Datasets for SQL

Lecture 75 Important Note Regarding the Dynamic Datasets

Lecture 76 Copy Activity with SQL Sink

Lecture 77 Data Flow with SQL Sink

Lecture 78 Course Project Activity: Storing our Structured Data in the SQL Database

Lecture 79 Links and Resources

Section 9: Tumbling Window Triggers

Lecture 80 Tumbling Window Triggers Overview

Lecture 81 Ingest Data for Specific Time Intervals

Lecture 82 Tumbling Window Trigger Dependencies

Lecture 83 Links and Resources

Section 10: Authentication and Access

Lecture 84 Overview

Lecture 85 Azure Key Vault

Lecture 86 System Assigned Managed Identities

Lecture 87 User Assigned Managed Identities

Lecture 88 Service Principals

Lecture 89 Fine Grained Access Control for Storage Accounts

Lecture 90 Links and Resources

Anyone interested in Azure or Cloud Data Engineering,Anyone interested in working with Big Data,Anyone working with cloud data