Kafka For Developers - Data Contracts Using Schema Registry

Learn to build a Kafka Produce/Consumer applications that uses AVRO data format and Confluent Schema Registry.

What you'll learn
Introduction to AVRO and its advantages of using them for sharing messages between applications
Publish AVRO records using Kafka Producer
Introduction to Schema Registry
Consume AVRO records using Kafka Producer
Use Schema Registry to register the AVRO Schema
Learn how Kafka Producer and Consumer interacts with the Schema Registry
Enforce Data Contracts using Schema Registry
Evolve AVRO schema using Schema Registry
Build Spring Boot Kafka Producer and Consumer applications that uses AVRO as a serialization format and interacts with Schema Registry
Requirements
Java 11 or Higher is needed (Java 17 recommended)
Docker must be installed
Intellij IDE or similar is needed
Experience build Kafka Producer and Consumer applications
Description
This course is structured to give you a theoretical and coding experience of Building Kafka Applications using AVRO and Schema Registry. If you are looking forward to learning the below-listed things:Techniques that are available to evolve the data between the applications that uses Kafka as a Streaming Platform Use a compact data format like AVRO to exchange data between the applicationsUse Schema Registry and its benefitsEnforcing Data Contracts between applications that uses Kafka as a Streaming PlatformHandle Data evolution gracefully using Schema RegistryThen this is the right course for you. This is a pure hands-on oriented course where you will be learning the concepts through code.By the end of this course, you will have a complete understanding of these concepts:Use AVRO as a data serialization formatEvolution of the data using Schema RegistryGetting Started with KafkaIn this section, I will give you all an introduction to the course and what to expect from this course.Data Contract & Serialization in KafkaLearn "How serialization is connected to Kafka" and how it benefits the overall Kafka architecture.We will look into different Serialization formats and the support for Schema in AVRO, Protobuf and ThriftIntroduction to AVRO - A data serialization systemAn introduction to AVRO and why AVRO is popular to work with Kafka and Schema Registry.Learn to build a simple AVRO schema.Kafka Setup & Demo in Local Using DockerIn this section , we will setup Kafka in local, Produce and Consume messages using Kafka Console Producer and Consumer.Greeting App - Base AVRO Project SetUp - GradleWe will set up the base project for the greeting app which we can use to generate the Java Classes from the Greetings schema using Gradle build tool.Greeting App - Base AVRO Project SetUp - MavenWe will set up the base project for the greeting app which we can use to generate the Java Classes from the Greetings schema using Maven build tool.Build AVRO Producer and Consumer in JavaWe will learn to build a Kafka Producer to publish AVRO records in to the Kafka topic.We will learn to build a Kafka Consumer to consume AVRO records from the Kafka topic.CoffeeShop Order Service Using AVRO - A Real time use CaseWe will build a AVRO schema for a real time use case and build Kafka Producers and Consumers to it.Logical Types in AVROI will cover the different logical types in AVRO and how to use them.TimeStampDecimalUUIDDateAVRO Record- Under the HoodAnatomy of an AVRO record when the data is published and consumed as AVRO recordSchema Changes in AVRODemonstration of how the consumer breaks with changing business requirementsData Evolution using Schema RegistryCover the different techniques of evolving a Schema with the changing business requirements.I will cover the different Compatibility techniques to share data between the producer and consumer applicationsBackward CompatibilityForward CompatibilityFull CompatibilityNone CompatibilitySchema Naming StrategiesI will cover the different naming strategies for Schema and how its impacts the application events.TopicName StrategyRecordName StrategyTopicRecordName StrategyBuild a Coffee Order Service using SpringBoot & Schema RegistryIn this section, we will code and build a Spring Boot Kafka application that exchanges the data in an AVRO format and interacts with Schema Registry for data evolution.Build a RestFul service to publish the events through which we receive events through the rest interface and then publish them to KafkaBy the end of this course, you will have a complete understanding of these concepts:Use AVRO as a data serialization formatEvolution of the data using Schema Registry

Overview

Section 1: Getting Started with the Course

Lecture 1 Introduction

Lecture 2 Pre-requestites

Section 2: Data Contract and Serialization in Kafka

Lecture 3 Data Contract & Serialization in Kafka

Lecture 4 Serialization Formats

Section 3: Introduction to AVRO - A data serialization system

Lecture 5 Introduction to AVRO - What is AVRO and Why AVRO?

Lecture 6 Build a simple AVRO Schema

Section 4: Kafka Setup & Demo in Local Using Docker

Lecture 7 Set up Kafka Broker and Zookeeper using Docker Compose

Lecture 8 Producer and Consumer Messages using CLI

Lecture 9 Produce and Consume using AVRO Console Producer & Consumer

Section 5: Greeting App - Base AVRO Project SetUp - Gradle

Lecture 10 Base Project SetUp for Greeting App

Lecture 11 Generate AVRO Java Records using AVRO Schema Files

Section 6: Greeting App - Base AVRO Project SetUp - Maven

Lecture 12 Base Project SetUp for Greeting App - Maven

Lecture 13 Generate AVRO Java Records using AVRO Schema Files - Maven

Section 7: Build AVRO Producer and Consumer in Java

Lecture 14 Lets build AVRO Kafka Producer

Lecture 15 Lets build AVRO Kafka Consumer

Section 8: CoffeeShop Order Service Using AVRO - A Real time Use Case

Lecture 16 Application Overview

Lecture 17 Project SetUp for CoffeeShop - Gradle

Lecture 18 Project SetUp for CoffeeShop - Maven

Lecture 19 Build a Coffee Order Schema using AVRO

Lecture 20 Generating AVRO classes using Gradle

Lecture 21 Generating AVRO classes using Maven

Lecture 22 Build a CoffeeShop Order Producer

Lecture 23 Build a CoffeeShop Order Consumer

Section 9: Logical Schema Types in AVRO

Lecture 24 Introduction to Logical Types in AVRO

Lecture 25 Add a timestamp, decimal logical type to the CoffeeOrder Schema

Lecture 26 Adding the UUID as Key for CoffeeOrder

Lecture 27 Date Logical Type

Section 10: AVRO Record- Under the Hood

Lecture 28 Whats inside an AVRO Record ?

Section 11: Schema Changes in AVRO - Issues without Schema Registry

Lecture 29 Evolving the Schema - Consumer Fails to Read the new Schema

Section 12: Introduction to Schema Registry

Lecture 30 Introduction to Schema Registry

Lecture 31 Publish and Consumer Record using Schema Registry

Lecture 32 Schema Registry internals & Interacting with Schema Registry using REST Endpoint

Lecture 33 Publish and Consume "Key" as an AVRO record

Section 13: Data Evolution using Schema Registry

Lecture 34 Data Evolution & Schema Evolution

Lecture 35 Update the code to interact with mavenLocal Repository - Gradle

Lecture 36 Update the code to interact with mavenLocal Repository - Maven

Lecture 37 Deleting a Field in Schema - BACKWARD Compatibility

Lecture 38 Adding a new Field in Schema - FORWARD Compatibility

Lecture 39 Add/Delete Optional Fields - FULL Compatibility

Lecture 40 Modify Field Names - NONE Compatibility

Section 14: Schema Naming Strategies

Lecture 41 Different Types of Naming Strategies

Lecture 42 Coffee Update Event AVRO Schema

Lecture 43 Publish and Consume CoffeeOrder UpdateEvent using RecordNameStrategy

Section 15: Build a Coffee Order Service using SpringBoot & Schema Registry

Lecture 44 Overview of the app

Lecture 45 Setting up the base project - Gradle

Lecture 46 Setting up the base project - maven

Lecture 47 Build the DTOs for the CoffeeOrderService

Lecture 48 Build the POST endpoint for the CoffeeOrderService - /coffee_orders

Lecture 49 Build the Service layer to map the DTO to AVRO domain object

Lecture 50 Configure the Kafka Producer properties in Coffee Order Service

Lecture 51 Build KafkaProducer to publish the CoffeeOrder Events

Lecture 52 Build the Coffee Order Consumer

Lecture 53 Build the PUT endpoint for the CoffeeOrderService - PUT /coffee_orders/{id}

Experienced Java Developers,Developers interested in learning AVRO and how to exchange data between applications using AVRO and Kafka,Developers who are interested in learning about Schema Registry and how it fits in to the Kafka,Developers who are interested in learning techniques to evolve the data