Apache Kafka A-Z With Hands On Learning

Learn Kafka from scratch to advanced through hands on learning

What you'll learn
Learn to handle real-time data feeds using Kafka open-source messaging
Master important points - Topics, Partitions, Brokers, Producers, Consumers
Learn how to build robust streaming applications using Kafka for real-time messaging
Create Producers and Consumers
Write Kafka Streams application
Configure/run Kafka Source and Sink Connectors
Write your own customized Kafka Connector
Configure Standalone and Sink Connector
Build Standalone Application using Kafka and Storm
Create Flume agent for Sending data from Kafka to HDFS
Requirements
Basic knowledge of Big Data
Basics of Java/Scala programming
Description
[4-Sep-21 Update] Added code of demos as downloaded resourceApache Kafka is an open-source distributed stream processing platform that provides high-throughput and low latency real-time messaging. More than 80% of all Fortune 100 companies trust, and use Kafka. Companies like Airbnb, Netflix, Microsoft, Intuit, Target, etc use Kafka extensively.This course has been aligned with industry best practices and has been created by industry leaders.This is the an exhaustive course covering A-Z of Kafka:-Basic concepts and architecture of Kafka-Kafka Producer and consumer- Serializer/De-serializer-Kafka Streams-Kafka Connect-Cluster setup and Administrating Kafka-Kafka Monitoring and Schema registry-Integration of Kafka with Storm- Integration of Kafka with Spark and Flume- Kafka Security-and Many more concepts in detailThe course contains :-High quality engaging videos of 9.5 Hrs-18 Demos-Quizzes for each lesson-1 ProjectThe course will help you design Apache Kafka and learn how Apache Kafka is used to storing and processing multiple nonstop streams of information faster and all the more efficiently.Learn how to design and introduce Kafka groups, With simple to follow and step by step guidelines.This Apache Kafka course will help students:- To learn the all the required knowledge to undertake responsibility for their organization's Kafka group by arranging Kafka producer, consumer, streams, and connectors- Depicting the design of Kafka & clarify use cases in business- Start journey in Kafka

Overview

Section 1: Course Introduction

Lecture 1 Introduction

Section 2: Lesson 1: Getting Started with Big Data and Apache Kafka

Lecture 2 Learning Objective

Lecture 3 Data Evolution Over Time

Lecture 4 Big Data Overview

Lecture 5 Big Data Analytics

Lecture 6 Messaging System

Lecture 7 Introduction and Features of Kafka

Lecture 8 Kafka Usecases

Lecture 9 Kafka Terminologies

Lecture 10 Kafka Components

Lecture 11 Kafka Architecture

Lecture 12 Kafka Clusters

Lecture 13 Kafka Industry Usecases

Lecture 14 Demo - Install Zookeeper

Lecture 15 Demo - Install Kafka

Lecture 16 Demo - Setup a Single Node Single Broker Cluster

Lecture 17 Demo - Setup a Multi Node Single Broker Cluster

Lecture 18 Key Takeaway

Section 3: Lesson 2: Kafka Producer

Lecture 19 Learning Objective

Lecture 20 Overview of Producer and It's Architecture

Lecture 21 Kafka Producer Configuration

Lecture 22 Kafka Producer Optional Configuration

Lecture 23 Kafka Producer Configuration Objects

Lecture 24 Demo - Create a Kafka Producer

Lecture 25 Sending Messages

Lecture 26 Serializers and Custom Serializers

Lecture 27 Demo - Creating a Custom Serializer

Lecture 28 Serializers Challenges and Serializing using Apache Avro

Lecture 29 Demo - Serializing Using Apache Avro

Lecture 30 Partitions

Lecture 31 Demo - Setup Custom Partition

Lecture 32 Key Takeaway

Section 4: Lesson 3: Kafka Consumer

Lecture 33 Learning Objective

Lecture 34 Overview of Kafka Consumer

Lecture 35 Consumer Groups

Lecture 36 Partition Rebalance and Creating a Consumer

Lecture 37 Poll loop and Its Functioning

Lecture 38 Kafka Configuring Consumer - Part 1

Lecture 39 Kafka Consumer Configuration - Part 2

Lecture 40 Demo - Create Kafka Consumer

Lecture 41 Commit and Offset

Lecture 42 Ways of Commiting Offset - Automatic Offset

Lecture 43 Ways of Commiting Offset - Commit Current Offset

Lecture 44 Ways of Commiting Offset - Asynchronous Commit

Lecture 45 Ways of Commiting Offset - Combining Synchronous and Asynchronous Commits

Lecture 46 Ways of Commiting Offset - Commit Specified Offset

Lecture 47 Rebalance Listeners

Lecture 48 Consuming Records with Specific Offset

Lecture 49 Deserializers

Lecture 50 Demo - Create and Use Custom Deserializer

Lecture 51 Key Takeaway

Section 5: Lesson 4: Kafka Operations and Performance Tuning

Lecture 52 Learning Objective

Lecture 53 Kafka Internals Overview

Lecture 54 Replication and Replica Types

Lecture 55 Preffered Ladder, Request and Request Processing

Lecture 56 Types of Requests

Lecture 57 Partition Allocation, File Management and Segments

Lecture 58 File Format, Index and Compaction

Lecture 59 Kafka Reliability and Reliablity Methods

Lecture 60 Broker Configuration for Replication

Lecture 61 Producer in Reliable System

Lecture 62 Consumer in Reliable System

Lecture 63 Key Takeaway

Section 6: Lesson 5: Kafka Cluster Architectures and Administering Kafka

Lecture 64 Learning Objective

Lecture 65 Cluster Mirroring

Lecture 66 Multi Cluster Architecture - Hub, Spokes, and Active Active Architecture

Lecture 67 Multi Cluster Architecture - Active-Standby Architecture and Strech Clusters

Lecture 68 MirrorMaker Configuration

Lecture 69 MirrorMaker Deployment and Tuning

Lecture 70 Demo - Setting up MirrorMaker

Lecture 71 Administering Kafka - Topic Operations

Lecture 72 Administering Kafka - Consumer Group Operations

Lecture 73 Dynamic Configuration Changes

Lecture 74 Partition Management

Lecture 75 Console Producer Tool

Lecture 76 Console Consumer Tool

Lecture 77 Key Takeaway

Section 7: Lesson 6: Kafka Monitoring and Schema Registry

Lecture 78 Learning Objective

Lecture 79 Monitoring and Its Importance

Lecture 80 Server or Infrastructure Monitoring and Application Monitoring

Lecture 81 Kafka Monitoring

Lecture 82 Kafka Broker Metrics - Under Replicated Partitions

Lecture 83 Kafka Broker Metrics - Others

Lecture 84 Topic and Partition Specific Metrics

Lecture 85 Logging and Client Monitoring

Lecture 86 Producer and Consumer Metrics

Lecture 87 Quotas and Lag Monitoring

Lecture 88 Monitoring Dashboard

Lecture 89 Demo - Setting up Open Source Health Monitor

Lecture 90 Kafka Schema Registry

Lecture 91 Kafka Component and Architecture

Lecture 92 Kafka Schema Registry - Internal working and Use-cases

Lecture 93 Kafka Schema Registry Working

Lecture 94 Demo - Using Kafka Schema Registry With Kafka

Lecture 95 Key Takeaway

Section 8: Lesson 7: Kafka Streams and Kafka Connectors

Lecture 96 Learning Objective

Lecture 97 Kafka Stream Overview

Lecture 98 Kafka Stream

Lecture 99 Kafka Stream Architecture and Working

Lecture 100 Kafka Stream Components

Lecture 101 Kafka Stream Architecture Tasks, Threading Model and Local State Store

Lecture 102 Kafka Stream Architecture - Record Buffer

Lecture 103 Memory Management and Streaming Data Pipeline

Lecture 104 Kafka Stream DSL

Lecture 105 K-Stream Operations

Lecture 106 K-Table

Lecture 107 K-Table Operations

Lecture 108 Aggregation and Windowing

Lecture 109 Processor Topology and Stream Processor

Lecture 110 Stream and Processor APIs

Lecture 111 Processor APIs - Create Topology

Lecture 112 Kafka Connectors

Lecture 113 Standalone and Sink Connector Configuration

Lecture 114 Running Kafka Connect

Lecture 115 Kafka Connector Distributed Mode

Lecture 116 HTTP Rest Interface

Lecture 117 Demo - Kafka Connector

Lecture 118 Demo - Create an Application using Kafka Streams

Lecture 119 Key Takeaway

Section 9: Lesson 8: Integration of Kafka with Storm

Lecture 120 Learning Objective

Lecture 121 Apache Storm

Lecture 122 Real-time Analytics

Lecture 123 Apache Storm Architecture

Lecture 124 Apache Storm Components

Lecture 125 Apache Storm Topology Part - 1

Lecture 126 Apache Storm Topology Part - 2

Lecture 127 Kafka Spout

Lecture 128 Integration of Apache Storm and Kafka

Lecture 129 Demo - Simple Standalone Application using Kafka and Storm

Lecture 130 Key Takeaway

Section 10: Lesson 9: Kafka Integration with Spark and Flume

Lecture 131 Learning Objective

Lecture 132 Introduction to Spark

Lecture 133 Spark Components

Lecture 134 Resilient Distributed Dataset

Lecture 135 RDD Operations - Transformation - Map, FlatMap and Filter

Lecture 136 RDD Operations - Transformation - Join, Distinct, First and Take

Lecture 137 RDD Operations - Actions

Lecture 138 Data Sets and Spark Session

Lecture 139 Data Sets and Spark Session Operations

Lecture 140 Spark Stream

Lecture 141 Spark Integration with Kafka

Lecture 142 Demo - Running Small Standalone Application in Spark with Kafka

Lecture 143 Flume

Lecture 144 Flume Connectors

Lecture 145 Flume Kafka to HDFS Configuration

Lecture 146 Demo - Creating Flume agent Sending data from Kafka to HDFS

Lecture 147 Key Takeaway

Section 11: Lesson 10: Admin Client and Securing Kafka

Lecture 148 Learning Objective

Lecture 149 Admin Client

Lecture 150 Demo - Perform Various Admin Tasks using Admin Client

Lecture 151 Kafka Security

Lecture 152 Kafka Security Component

Lecture 153 SASL

Lecture 154 Configure SSL in Kafka

Lecture 155 Secure using ACLs

Lecture 156 Key Takeway

Section 12: Course End Project

Developers who are new to Kafka,Developers who want complete exhaustive understanding of Kafka with hand-on,Architects who need to integrate Kafka into their solution architecture