Master Azure Databricks

Learn Databricks concepts, PySpark, Spark Structure Streaming, Delta lake, Databricks SQL Analytics, REST API & CLI

What you'll learn
Azure Databricks Fundamentals
RDD & PySpark DataFrame
Spark Structure Streaming
Databricks Advance concept (Delta lake,SQL Warehouse,Security,Devops,Administration)

Description
Module 1

What is Data Pipeline

What is Azure databricks

Azure Databricks Architecture

Azure Account Setup

WorkSpace Setup

Module 2

Navigate the Workspace

Runtimes

Clusters

Notebooks

Libraries

Repos

Databricks File System (DBFS)

DBUTILS

Widgets

Workflows

Metastore - Setup external Metastore

Module 3

What is RDD

Creating RDD

RDD transformations

RDD Actions

RDD Joins

Pair RDD

Broadcast Variables

Accumulators

Convert RDD to DataFrame

Import & Read data

Create a table using the UI

Create a table in a Notebook

Module 4

Create DataFrames

Define Schema

Functions

Casting Operations

Filter Transformation

Update, Update ALL & UpdateByName

OrderBy & SortBY

GroupBy

Remove Duplicates

Window Functions

Date and Timestamp Functions

UDF (User Defined Function)

JOIN

Handle corrupt records using the badRecordsPath

File metadata column

Module 5

Read Parquet File

Read CSV Files

Read JSON Files

Read XML Files

Read Excel file

SQL databases using JDBC

Azure blob storage

Module 6

What is Spark Structure Streaming

Data Source & Sink

Rate & File Source

Kafka Source

Sink : Console, Memory, File & Custom

Build Streaming ETL

Stream ETL 1 : Setup Event Hub

Streaming ETL 2 : Event Hub Producer

Streaming ETL 3 : Integrate Event Hubs with Data Bricks

Streaming ETL 4 : Transformation

Streaming ETL 5 : Ingest into Azure Data storage

Twitter Sentiment Analysis - Introduction

Setup Twitter Developer Account

Twitter Sentiment Analysis - II

Twitter Sentiment Analysis - III

Module 7

Components in Databricks SQL

Configuring a SQL Endpoint

Creating a Table from a CSV File

Create Queries

Parameterized Query

Query Profile

Building Visualization (Table, BAR & PIE )

Building Line Chart & Counter Chart

Adding Charts to Dashboards

Defining a Query Alert

Access Control on Databricks SQL Objects

Lab: Data Object Access Control

Transfer Ownership

Access SQL Endpoint from Python

Databricks SQL CLI

Databricks SQL CLI

Who this course is for
Data Engineering Students & Developers
Bigdata Developer
Python & SQL Developer

Requirements
Basic Python Skills
Basic SQL Skills
Azure Account


More info