Master Microsoft Fabric: A Complete End-To-End Project- Cicd

Microsoft Fabric from Scratch includes Git, Lakehouse, Warehouse, Synapse Migration, end to end project with CICD- DP600

What you'll learn

Understand and implement Microsoft Fabric

Learn Data Engineering with Microsoft Fabric

Implement Continuous Integration and Continuous Deployment in project

Migrate Existing Synapse notebooks to Microsoft Fabric

How access control and permissions works in Microsoft Fabric

Enable Git and implement in Fabric project

Understand OneLake and its features

Implement Medallion Architecture with Microsoft Fabric

Build an end to end CICD pipeline in Fabric

Understand Fabric Lakehouse and its features

Use Capacity Metrics App to monitor your items in Workspace

Understand Fabric Warehouse and its features

Handle different scenarios with Power BI in Fabric

Row-level security , Column level security, object level security, Dynamic data masking in Warehouse

Incremental loading to ingest data in all layers

Update Microsoft Power BI from XMLA endpoint

Different ways to refresh semantic models of Power BI

Tenant level permissions, Capacity level permissions

Workspace level sharing, Item level sharing of Fabric Items

Row-level security , Column level security, object level security in Power BI

Capacity metrics app different throttling stages

Azure data factory vs Fabric data pipelines from On-premise data ingestion

Different Medallion Architectural patterns to implement in Fabric

Requirements

Basic knowledge on Python and SQL

Basic knowledge on Azure Cloud

An Azure account to perform hands-on practicals

No experience on Microsoft Fabric is needed, you will learn everything from scratch

Description

Unlock the full potential of Microsoft Fabric in this comprehensive course designed for data professionals at all levels. "Master Microsoft Fabric: A Complete End-to-End Project - CICD" takes you on an immersive journey through the core concepts and practical applications of Microsoft Fabric, enabling you to build and deploy advanced data solutions with confidence.The course begins with an in-depth exploration of foundational concepts such as Lakehouse, Warehouse, and OneLake, helping you understand the architecture and functionality of Microsoft Fabric. You'll learn how to leverage the Capacity Metrics App to monitor and manage your resources effectively, ensuring optimal performance for your data projects.As we delve deeper, you'll discover the seamless process of migrating Synapse notebooks to Fabric, gaining hands-on experience with its enhanced features. This migration will empower you to utilize existing resources while maximizing the benefits of the new environment.The course culminates in a capstone project that encompasses all the skills you've acquired. You'll implement a complete end-to-end data solution, focusing on building a robust Continuous Integration and Continuous Deployment (CICD) pipeline. This project not only solidifies your learning but also prepares you for real-world applications.By the end of this course, you will possess the expertise to navigate Microsoft Fabric and implement data solutions that meet today’s challenges. Join us and elevate your data skills to the next level!

Overview

Section 1: Course Introduction

Lecture 1 Introduction

Lecture 2 Pre-requisites and benefits

Lecture 3 Project Architecture and learning structure

Lecture 4 Course Slides

Section 2: Environment Setup

Lecture 5 Section Introduction

Lecture 6 Creating storage account in Azure

Lecture 7 Creating Azure Synapse Analytics Service in Azure

Section 3: Understanding Microsoft Fabric

Lecture 8 Section Introduction

Lecture 9 Evolution of Data Architectures

Lecture 10 Delta Lake Structure

Lecture 11 Why Microsoft Fabric is needed

Lecture 12 Microsoft's definition of Fabric

Lecture 13 How to enable and access Microsoft Fabric

Lecture 14 Fabric License and costing

Lecture 15 Experiences in Microsoft Fabric

Lecture 16 Fabric Terminology

Lecture 17 OneLake in Fabric

Lecture 18 One copy for all Computes in Microsoft Fabric

Section 4: Fabric Lakehouse

Lecture 19 Section Introduction

Lecture 20 Understanding Fabric Workspaces

Lecture 21 Enable Fabric Trail and Create workspace

Lecture 22 Purchasing Fabric Capacity from Azure

Lecture 23 Workspace roles in Microsoft Fabric

Lecture 24 Creating a Lakehouse

Lecture 25 What is inside lakehouse

Lecture 26 Uploading data to Lakehouse

Lecture 27 Uploading Folder into Lakehouse

Lecture 28 SQL analytics endpoint in Lakehouse

Lecture 29 Access SQL analytics endpoint using SSMS

Lecture 30 Visual Query in SQL endpoint

Lecture 31 Default Semantic Model

Lecture 32 OneLake File Explorer

Section 5: Fabric Data Factory

Lecture 33 Section Introduction

Lecture 34 Fabric Data Factory UI

Lecture 35 Ways to load data into Lakehouse

Lecture 36 Fabric Data Factory vs Azure Data Factory Scenario

Lecture 37 Gateway types in Microsoft Fabric

Lecture 38 Installing On-prem data gateway

Lecture 39 Create Connection to SQL Server

Lecture 40 Pipeline to ingest OnPrem SQL data to Lakehouse

Lecture 41 Scenario completed using Fabric data factory

Lecture 42 Dataflow Gen2 - Intro

Lecture 43 Creating DataFlow Gen2

Lecture 44 DataFlow Gen2 in Fabric vs Dataflows in ADF

Section 6: Fabric OneLake

Lecture 45 Section Introduction

Lecture 46 Loading data status

Lecture 47 Shortcuts in Fabric - Intro

Lecture 48 Prerequisites to Create a shortcut

Lecture 49 Creating a shortcut in Files of Lakehouse

Lecture 50 Criteria to create shortcuts in table section

Lecture 51 Uploading required files and access for synapse

Lecture 52 Right way to create a shortcut in table's section

Lecture 53 Creating delta file

Lecture 54 Creating shortcut in Table's section

Lecture 55 Scenario - Creating shortcut with delta in a subfolder

Lecture 56 Scenario - Creating shortcut with only parquet format

Lecture 57 Requirements to create shortcuts in Table and files section

Lecture 58 Updation Scenario 1 - Lakehouse to Datalake

Lecture 59 Updation Scenario 2 - Datalake to Lakehouse

Lecture 60 Shortcut deletion scenarios intro

Lecture 61 Deletion Scenario 1 - Delete in Lakehouse files

Lecture 62 Deletion Scenario 2 - Delete in ADLS

Lecture 63 Deletion Scenario 3 - Delete table data in Lakehouse

Lecture 64 Deletion Scenario 4 - Delete table data in ADLS

Lecture 65 Deletion Scenario 5 - Deleting entire shortcut

Lecture 66 Shortcut deleting scenario summary

Section 7: Fabric Synapse Data Engineering

Lecture 67 Section Introduction

Lecture 68 Ingestion to Lakehouse status

Lecture 69 Spark in Microsoft Fabric

Lecture 70 Spark pools in Microsoft Fabric

Lecture 71 Spark pool node size

Lecture 72 Customizing Starter pools

Lecture 73 Creating a custom pool in Workspace

Lecture 74 Standard vs High Concurrency Sessions

Lecture 75 Changing Spark Settings to StarterPool

Lecture 76 Understanding Notebooks UI

Lecture 77 Fabric Notebook basics

Lecture 78 MSSparkUtils - Intro

Lecture 79 MSSparkUtils - FS- Mount

Lecture 80 MSSparkUtils - FS - Other utils

Lecture 81 MSSparkUtils - FS - FastCp

Lecture 82 Creating Folders in Microsoft Fabric

Lecture 83 MSSparkUtils - Notebook Utils - Run exit

Lecture 84 MSSparkUtils - Notebook - RunMultiple

Lecture 85 Access ADLS data to Lakehouse - Intro

Lecture 86 Access ADLS using Entra ID

Lecture 87 Access ADLS using Service principal

Lecture 88 Access ADLS using SP with keyvault

Lecture 89 Call Fabric notebook from Fabric pipeline

Lecture 90 Managed vs External table - Intro

Lecture 91 Create a Managed Table

Lecture 92 Create an External Table

Lecture 93 Shortcut Table is an external or managed table

Lecture 94 Data Wrangler in Fabric Notebook

Lecture 95 Environments in Microsoft Fabric

Lecture 96 Understanding V-order optimization

Lecture 97 Inspire us with your Thoughts

Lecture 98 Spark Job Definition

Lecture 99 What is a data mesh

Lecture 100 Creating domains in Fabric

Section 8: Synapse Migration to Microsoft Fabric

Lecture 101 Section Introduction

Lecture 102 Manual import from Synapse to Fabric

Lecture 103 Automated way to import and export notebooks - Intro

Lecture 104 Migrate all notebooks from Synapse to fabric

Lecture 105 Possibility of Migration of Pipelines to Fabric pipelines

Lecture 106 Ways to migrate ADLS data to Fabric OneLake

Lecture 107 Migrate ADLS data to Onelake using Storage Explorer

Section 9: Fabric Capacity Metrics App

Lecture 108 Section Introduction

Lecture 109 Install Capacity Metrics App

Lecture 110 Understanding UI of Capacity Metrics App

Lecture 111 Capacity Units consumption

Lecture 112 Throttling vs Smoothing

Lecture 113 Throttling stage- Overage Protection Policy

Lecture 114 Other throttling stages

Lecture 115 Throttling stages Summary

Lecture 116 Overages in Fabric

Lecture 117 System Events in Fabric

Lecture 118 Matrix Visual

Section 10: Fabric Synapse Data Warehouse

Lecture 119 Section Introduction

Lecture 120 Creating a Warehouse in Fabric

Lecture 121 Warehouse vs SQL Analytics Endpoint

Lecture 122 Creating a table and Limitations

Lecture 123 Ways to Load Data into Warehouse

Lecture 124 Loading Data using COPY INTO Command

Lecture 125 Loading Data using Pipeline to Warehouse

Lecture 126 Loading Data using DataFlow Gen2

Lecture 127 Data Sharing - Lakehouse & Warehouse

Lecture 128 Cross Database Ingestion in Warehouse

Lecture 129 Lakehouse vs Warehouse when to choose what

Lecture 130 Different Medallion Architectural patterns

Lecture 131 Update Lakehouse data from WH and vice versa

Lecture 132 SQL query as session in Fabric

Lecture 133 Zero Copy clone within and across Schema

Lecture 134 Time Travel in Warehouse

Lecture 135 Benefits & Limitations of Zero Copy clones

Lecture 136 Cloning single or multiple tables using UI

Lecture 137 Query Insights in Warehouse

Section 11: Fabric Access Control and permissions

Lecture 138 Section Introduction

Lecture 139 Microsoft Fabric Structure

Lecture 140 Tenant Level permissions

Lecture 141 Capacity Level Permissions

Lecture 142 Creating new user in Entra ID

Lecture 143 Workspace roles- Workspace Administration

Lecture 144 Workspace roles - Data pipeline permissions

Lecture 145 Workspace Roles - Notebook, Spark jobs, etc

Lecture 146 Data Warehouse permissions - Intro

Lecture 147 Workspace Roles - Accessing shortcuts internal to fabric - Theory

Lecture 148 Workspace Roles - Accessing Shortcuts Internal to Fabric - Practical

Lecture 149 Workspace Roles - Accessing ADLS shortcuts - Theory

Lecture 150 Workspace Roles - Accessing ADLS shortcuts - Practical

Lecture 151 Workspace Roles - Lakehouse permissions

Lecture 152 Item level permissions - Intro

Lecture 153 Warehouse Sharing - No additional permissions

Lecture 154 Warehouse Sharing - ReadData permissions

Lecture 155 Warehouse Sharing - ReadAll permissions

Lecture 156 Warehouse Sharing - Build permissions

Lecture 157 Extend Microsoft Fabric Trail

Lecture 158 Lakehouse Sharing - All permissions

Lecture 159 Notebook - Item Sharing

Lecture 160 Manage OneLake data access

Lecture 161 Row-Level Security in Warehouse and SQL endpoint

Lecture 162 Dynamic Data Masking in Warehouse and SQL endpoint

Lecture 163 Column & Object level security in Warehouse and SQL endpoint

Section 12: Microsoft Power BI in Fabric

Lecture 164 Section Introduction

Lecture 165 Default Semantic Model - Intro

Lecture 166 Manage default Semantic Model

Lecture 167 Accessing Lakehouse or Warehouse using Power BI Desktop

Lecture 168 Automatically update semantic model in Lakehouse or warehouse

Lecture 169 Creating new Semantic Model

Lecture 170 Lineage view and Impact Analysis

Lecture 171 Connect using XMLA endpoint and readwrite using Tabular Editor

Lecture 172 What is a Direct Lake

Lecture 173 Direct Lake - Practical

Lecture 174 DirectLake vs Direct Query vs Import modes

Lecture 175 Refresh a Semantic Model - Manual

Lecture 176 Refresh Semantic model from Notebook

Lecture 177 Refresh Semantic Model from XMLA Endpoint

Lecture 178 Refresh Semantic Model from Data Pipeline

Lecture 179 Fallback to Direct Query Scenarios

Lecture 180 Handling Fallback behavior of Semantic Models

Lecture 181 Copy Multiple measures from One Model to another in Fabric.mp4

Lecture 182 Row-level security in Power BI

Lecture 183 Row-level security without Viewer Role

Lecture 184 Build permission to user with RLS

Lecture 185 Dynamic Row-level Security

Lecture 186 Object level Security with viewer role

Lecture 187 Object level security without Viewer role

Lecture 188 Column level security

Section 13: Fabric End to End project

Lecture 189 Section Introduction

Lecture 190 Different Medallion architectures in Fabric

Lecture 191 Understanding domain and dataset information

Lecture 192 Project Architecture

Lecture 193 Creating workspace for project and review dataset

Lecture 194 Get data from Raw to landing - theory

Lecture 195 Raw to landing zone

Lecture 196 Different incremental loading patterns

Lecture 197 Incrementally ingest from Raw to landing zone

Lecture 198 Automate ingest from Raw to Landing using pipeline

Lecture 199 Ingest data from Landing to Bronze layer - Theory

Lecture 200 Understanding UPSERT logic for Landing to Bronze ingestion

Lecture 201 Landing to Bronze layer - practical

Lecture 202 Reading landing to bronze from next partition

Lecture 203 UPSERT scenario practical - Landing to bronze

Lecture 204 Bronze layer to Silver layer - Theory

Lecture 205 Understanding data transformations and UPSERT logic for Silver table

Lecture 206 Silver table - Data cleaning

Lecture 207 Silver Layer - data transformations

Lecture 208 Gold Layer - Facts and dimensions table - Theory

Lecture 209 Gold Layer - Facts and dimension tables - Practical

Lecture 210 Data modelling and creating a report

Lecture 211 Orchestrate end to end pipeline and execute it

Section 14: Fabric Git Integration

Lecture 212 Section Introduction

Lecture 213 Creating data sources for PROD

Lecture 214 Changes made to support Git integration

Lecture 215 Executing to check if changes were working

Lecture 216 Sign up with Azure DevOps account

Lecture 217 Connect Fabric workspace to Azure DevOps

Lecture 218 Git integration permissions and Limitations

Lecture 219 Locking main branch with branch policy

Lecture 220 Understanding Continuous Integration (CI) in Fabric

Lecture 221 Continuous Integration in Fabric Workspace

Lecture 222 Status of workspace created for feature branch

Lecture 223 Understanding Continuous Deployment in Fabric

Lecture 224 Deploying Fabric items from Dev to Prod

Lecture 225 Deployment rules to Change data sources of Prod workspace

Lecture 226 End to End execution in PROD

Lecture 227 Git integration for Power BI developers

Lecture 228 Summary of version control for Power BI

Section 15: Fabric Course - Conclusion

Lecture 229 Course Completion

Lecture 230 My other Data Engineering Courses

Data Engineers who want to get real time experience using Microsoft Fabric,Data professionals who want to build an end to end project in Microsoft Fabric,Engineers who want to learn Microsoft Fabric from Scratch