Elasticsearch Illuminated: A Practical Guide

Utilizing ELK Stack in Building Complex Architectures

What you'll learn
Understand the basics of Elasticsearch
Understand the ELK stack (Open Source)
Build Applications utilizing Elasticsearch
Elasticsearch as part of complex archietectures
Requirements
Programming experience in an object oriented Language
Ability to use command Line
Windows OS
Description
This course has a broader scope and covers not just the ELK stack but also integration of Elasticsearch  with other technologies that are commonly used in the industry to build efficient, scalable and powerful architectures. The course starts with the basics and guides the student on the ELK stack, provides useful information on how to install various components of the ELK stack. Certain parts of the lessons are intentially repeated to reinforce the ideas and help with improved learning. This course can be taken by both beginners and experienced Professionals to learn the technology or refresh previous knowledge. The course covers the latest version 8 of the ELK stack however do note the minor versions are being released almost every 2 months therefore, the version on the course may be slightly behind but all that is discussed in the course should work with the latest minor version of ELK stack.It is highly recommended that student has background in software development or possesses programming skills. For those who do not, they can follow along or just watch the lessons. The course has been desgined to enable student to learn the exciting stack and help them understand how they can adopt ELK stack in their current & future projects.

Overview

Section 1: Introduction

Lecture 1 Introduction

Lecture 2 Scope

Lecture 3 ELK Architecture

Lecture 4 Elasticsearch

Lecture 5 Why Elasticsearch

Lecture 6 Elasticsearch common use cases

Lecture 7 Configuration over code

Lecture 8 System Requirements

Section 2: Installations

Lecture 9 Download & Install Elasticsearch I

Lecture 10 Download & Install Elasticsearch II

Lecture 11 [Important] Choosing short paths (Windows)

Lecture 12 [Important] Firewalls and security settings (Windows)

Lecture 13 Running Elasticsearch (Local)

Lecture 14 Running Elasticsearch (non-Windows)

Lecture 15 [Optional] Running Elasticsearch (Docker)

Lecture 16 Important (Certificate Warning, Docker version)

Lecture 17 [Optional] Verify Elasticsearch is up (Docker)

Lecture 18 Elasticsearch Default Configuration (Local Machine)

Lecture 19 Important commands

Lecture 20 Verify Elasticsearch is up and running (Local)

Lecture 21 Important Information about the Certificate Warning

Lecture 22 [Optional] Sorting the Certificate Warning

Lecture 23 Elasticsearch Installation Review

Lecture 24 Download & Install Kibana I

Lecture 25 Download & Install Kinbana II [Enrollment]

Lecture 26 [Optional] Containerized Kibana

Lecture 27 Downloading Logstash

Lecture 28 Examining Logstash Installation

Lecture 29 Important About next lesson

Lecture 30 Verify Logstash (Simple Pipeline)

Lecture 31 Downloading FileBeats (Beat)

Lecture 32 FileBeats Installation

Lecture 33 Important about next lesson (Data Shippers)

Lecture 34 Verify FileBeat installation (Shipping Data via FileBeat)

Lecture 35 Section Review

Lecture 36 [Troubleshooting] Kibana -> Elasticsearch connection Issue

Lecture 37 [Troubleshooting] Problem Starting Zookeeper

Section 3: Elasticsearch Architecture

Lecture 38 Elasticsearch Modes (Single vs Cluster vs SAAS)

Lecture 39 Elasticsearch Rest APIs

Lecture 40 Restful API

Lecture 41 Understanding JSON I

Lecture 42 Understanding JSON II

Lecture 43 Optional [Starting Elasticsearch & Kibana]

Lecture 44 Client for Rest API (Kibana Dev Console)

Lecture 45 Inside Elasticsearch Cluster I

Lecture 46 Inside Elasticsearch Cluster II

Lecture 47 Inverted Index

Lecture 48 Review: Partitioning, Shards and Index

Lecture 49 Cluster API

Lecture 50 Segments

Lecture 51 Analyzers

Lecture 52 Mappings

Lecture 53 Relevance

Section 4: Elasticsearch CRUD Operations

Lecture 54 Data for this and next section [Important]

Lecture 55 Creating a document I

Lecture 56 Creating a document II [Avoid Duplicates]

Lecture 57 Creating documents III [PUT]

Lecture 58 Geting a document

Lecture 59 Bulk Create

Lecture 60 Search Query I

Lecture 61 Documents Relevance (Similarity)

Lecture 62 Controlling Relevancy [Boosting Parameter]

Lecture 63 Controlling Relevancy [Boosting Query]

Lecture 64 Term Query [Recap]

Lecture 65 Search Query II [Bool Query]

Lecture 66 Search Query III [Filters]

Lecture 67 Fuzzy Query

Lecture 68 Keyword vs Text

Lecture 69 Updating a document I

Lecture 70 Updating a document II

Lecture 71 Deleting a Document

Lecture 72 Creating an Index

Lecture 73 Reindexing Documents

Lecture 74 Node & NPM [Important]

Lecture 75 Using ElasticDump [Import to ES]

Lecture 76 Using ElasticDump [Export from ES]

Lecture 77 Loading CSV data into Elasticsearch via Kibana

Lecture 78 JSON & NDJSON

Lecture 79 Loading JSON data into Elasticsearch via Kibana Upload

Lecture 80 Kibana Data View [customers index]

Lecture 81 Loading Apache Log Data [GROK]

Lecture 82 Understanding GROK with example

Lecture 83 Data Ingestion Architecture

Lecture 84 Data Ingestion Bulk insert Demo [Python]

Lecture 85 Using SQL with Elasticsearch

Section 5: Advanced Elasticsearch

Lecture 86 Analyzers I [Built-in Analyzers]

Lecture 87 Analyzers II [Custom Analyzer]

Lecture 88 Ngram Tokenizer

Lecture 89 Edge Ngrams

Lecture 90 Runtime Fields

Lecture 91 Creating a policy [ILM]

Lecture 92 Apply a policy [ILM]

Lecture 93 Aggregations

Section 6: Kibana Visualization and Analytics

Lecture 94 What is Kibana

Lecture 95 Working with Data Views

Lecture 96 Visualizations & Dashboards

Lecture 97 Management

Section 7: Data Shippers

Lecture 98 Data Shippers

Lecture 99 Beats and Logstash

Lecture 100 Beats Configuration

Lecture 101 Uptime Monitoring using Heartbeat

Lecture 102 Download & Installation

Lecture 103 Configuration

Lecture 104 Testing Configuration

Lecture 105 Enabling Heartbeat Dashboards

Lecture 106 Running Heatbeat

Lecture 107 Elasticsearch and Kibana Status

Lecture 108 Running File Beat Review

Lecture 109 Data Streams

Section 8: Logstash

Lecture 110 What is Logstash

Lecture 111 Configuration

Lecture 112 Important for next lesson

Lecture 113 Logstash Example

Lecture 114 Pros & Cons

Section 9: NoSQL Data Modelling

Lecture 115 Why Data Modelling

Lecture 116 Data Modelling

Lecture 117 Data Modelling (SQL vs NoSQL)

Lecture 118 Elasticsearch Data Modelling Patterns

Lecture 119 Importing Data for this section

Lecture 120 Case Study

Lecture 121 Relationship Patterns: Application Joins

Lecture 122 Relationship Patterns: Application Joins Demo

Lecture 123 Loading Data for Inner Objects

Lecture 124 Inner Objects

Lecture 125 Problems of Inner Objects Demo

Lecture 126 Inner Objects Flattening

Lecture 127 Loading Data for Nested Objects

Lecture 128 Relationship Patterns: Nested Objects

Lecture 129 Relationship Patterns: Nested Objects Demo

Lecture 130 Relationship Patterns: Parent Child relationships Introduction

Lecture 131 Relationship Patterns: Parent Child relationship Intro

Lecture 132 Relationship Patterns: Parent Child relationship Demo

Lecture 133 Relationship Patterns: Denormalisation

Lecture 134 Relationship Patterns: Denormalisation Code

Lecture 135 Relationship Patterns: Denormalisation Demo

Lecture 136 Relationship Patterns: Inversion Pattern

Lecture 137 Factors to consider (Embedding vs Reference)

Section 10: Elasticsearch As Data Store (App Development)

Lecture 138 About this section

Lecture 139 Elasticsearch as Database

Lecture 140 Example architecture: Elasticsearch as a DataStore

Lecture 141 Load plant Data [CSV]

Lecture 142 Checking Plant Index mappings

Lecture 143 Code Editors

Lecture 144 Setting up and Running the Project

Lecture 145 Application Demo

Lecture 146 Dependencies (packages)

Lecture 147 Making a connection

Lecture 148 Server side code (NodeJS-ES integration)

Lecture 149 Front end (Pure Javascript Application)

Lecture 150 Application Review

Lecture 151 Binary Data

Lecture 152 What we will build

Lecture 153 Transforming raw data

Lecture 154 Creating Index and specifying custom mappings

Lecture 155 Important (installing all the required packages)

Lecture 156 Loading data via custom application (Node JS)

Lecture 157 Verifying Stored data

Lecture 158 Server side processing

Lecture 159 Client side Application

Lecture 160 Application Demo

Section 11: ELK as part of Complex Architecture

Lecture 161 Section Introduction

Lecture 162 Change data capture

Lecture 163 Data Pipelines

Lecture 164 Determine System OS

Lecture 165 Apache Kafka on Windows [Requirements]

Lecture 166 Download and Install JDK I

Lecture 167 Download and Install JDK II

Lecture 168 Setting Paths (Java/JDK)

Lecture 169 [Phase I] Postgres to Kafka Integration

Lecture 170 Downloading & Installing Apache Kafka (Windows)

Lecture 171 Exploring Apache Kafka Installation Directory [Windows]

Lecture 172 Starting Zookeeper

Lecture 173 [Troubleshooting] Zookeeper Warning

Lecture 174 Configuring & Starting Apache Kafka

Lecture 175 Setting up Postgres on Windows (Docker)

Lecture 176 Setting up Postgres Locally

Lecture 177 Running Postgres on Docker

Lecture 178 [Important] Pgadmin new connection

Lecture 179 Review What have we got

Lecture 180 Important for next lesson

Lecture 181 Download and copy Debezium Connector

Lecture 182 Kafka Connect

Lecture 183 Configuring Connector

Lecture 184 Putting it all together

Lecture 185 Commands used

Lecture 186 [Phase II] Kafka to Elasticsearch

Lecture 187 Ports in use

Lecture 188 Limitation of Confluent Kafka Sink Connector

Lecture 189 Options

Lecture 190 Important for next 2 lessons [Must read]

Lecture 191 Creating Logstash Pipeline (Configuration)

Lecture 192 Configuring Security for Logstash ES Integration

Lecture 193 Running Logstash and verifying results

Lecture 194 Testing the complete pipeline (real time)

Lecture 195 Final Review (What we have achieved)

Lecture 196 Troubleshooting/ FAQ

Section 12: ELK in Cloud

Lecture 197 Cloud SAAS Offerings

Section 13: Elasticsearch Optimization & Considerations

Lecture 198 Checking Elasticsearch installation using Opster

Lecture 199 Data Modelling

Lecture 200 Segments (Optimization)

Lecture 201 Sharding Strategy

Lecture 202 Elasticsearch Performance Optimizations

Section 14: Wrap Up

Lecture 203 Course Review

Software developers,Software Engineers,Programmers,Architects