Build An Aws Machine Learning Pipeline For Object Detection
Published 3/2023
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz
Language: English | Size: 7.34 GB | Duration: 16h 18m
Published 3/2023
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz
Language: English | Size: 7.34 GB | Duration: 16h 18m
Use AWS Step Functions + Sagemaker to Build a Scalable Production Ready Machine Learning Pipeline for Plastic Detection
What you'll learn
Learn how you can use Google's Open Images Dataset V7 to use any custom dataset you want
Create Sagemaker Domains
Upload and Stream data into you Sagemaker Environment
Learn how to set up secure IAM roles on AWS
Build a Production Ready Object detection Algorithm
Use Pandas, Numpy for Feature and Data Engineering
Understanding Object detection annotations
Visualising Images and Bounding Boxes with Matplotlib
Learn how Sagemaker's Elastic File System(EFS) works
Use AWS' built in Object detection detection algorithm with Transfer Learning
How to set up Transfer Learning with both VGG-16 and ResNet-50 in AWS
Learn how to save images to RecordIO format
Learn what RecordIO format is
Learn what .lst files are and why we need them with Object Detection in AWS
Learn how to do Data Augmentation for Object detection
Gain insights into how we can manipulate our input data with data augmentation
Learn AWS Pricing for SageMaker, Step Functions, Batch Transformation Jobs, Sagemaker EFS, and many more
Learn how to choose the ideal compute(Memory, vCPUs, GPUS and kernels) for your Sagemaker tasks
Learn how to install dependencies to a Sagemaker Notebook
Setup Hyperparameter Tuning Jobs in AWS
Set up Training Jobs in AWS
Learn how to Evaluate Object detection models with mAP(mean average precision) score
Set up Hyperparameter tuning jobs with Bayesian Search
Learn how you can configure Batch Size, Epochs, optimisers(Adam, RMSProp), Momentum, Early stopping, Weight decay, overfitting prevention and many more in AWS
Monitor a Training Job in Real time with Metrics
Use Cloudwatch to look at various logs
How to Test your model in a Sagemaker notebook
Learn what Batch Transformation is
Set up Batch Transformation Jobs
How to use Lambda functions
Saving outputs to S3 bucket
Prepare Training and Test Datasets
Data Engineering
How to build Complex Production Ready Machine Learning Pipelines with AWS Step Functions
Use any custom dataset to build an Object detection model
Use AWS Cloudformation with AWS Step Functions to set up a Pipeline
Learn how to use Prebuilt Pipelines to Configure to your own needs
Learn how you can Create any Custom Pipelines with Step Functions(with GUI as well)
Learn how to Integrate Lambda Functions with AWS Step Functions
Learn how to Create and Handle Asynchronous Machine Learning Pipelines
How to use Lambda to read and write from S3
AWS best practices
Using AWS EventBridge to setup CRON jobs to tell you Pipeline when to Run
Learn how to Create End-to-End Machine Learning Pipelines
Learn how to Use Sagemaker Notebooks in Production and Schedule Jobs with them
Learn Machine Learning Pipeline Design
Create a MERN stack web app to interact with our Machine Learning Pipeline
How to set up a production ready Mongodb database for our Web App
Learn how to use React, Nextjs, Mongodb, ExpressJs to build a web application
Create and Interact with JSON files
Put Convolutional Neural Networks into Production
Deep Learning Techniques
How to clean up an AWS account after you are done
Train Machine Learning models on AWS
How to use AWS' GPUs to speed up Machine Learning Training jobs
Learn what AWS Elastic Container Registry(ECS) is and how you can download Machine Learning Algorithms from it
AWS Security Best practices
Requirements
Laptop with Internet Access
AWS account
Knowledge of Python and basic Machine Learning
Spend 20-50 dollars on AWS if you want to follow along with me. Note that you can still follow along without having to pay any money
Description
Welcome to the ultimate course on creating a scalable, secure, complex machine learning pipeline with Sagemaker, Step Functions, and Lambda functions. In this course, we will cover all the necessary steps to create a robust and reliable machine learning pipeline, from data preprocessing to hyperparameter tuning for object detection.We will start by introducing you to the basics of AWS Sagemaker, a fully-managed service that provides developers and data scientists with the ability to build, train, and deploy machine learning models quickly and easily. You will learn how to use Sagemaker to preprocess and prepare your data for machine learning, as well as how to build and train your own machine learning models using Sagemaker's built-in algorithms.Next, we will dive into AWS Step Functions, which allow you to coordinate and manage the different steps of your machine learning pipeline. You will learn how to create a scalable, secure, and robust machine learning pipeline using Step Functions, and how to use Lambda functions to trigger your pipeline's different steps.In addition, we will cover deep learning related topics, including how to use neural networks for object detection, and how to use hyperparameter tuning to optimize your machine learning models for different use cases.Finally, we will walk you through the creation of a web application that will interact with your machine learning pipeline. You will learn how to use React, Next.js, Express, and MongoDB to build a web app that will allow users to submit data to your pipeline, view the results, and track the progress of their jobs.By the end of this course, you will have a deep understanding of how to create a scalable, secure, complex machine learning pipeline using Sagemaker, Step Functions, and Lambda functions. You will also have the skills to build a web app that can interact with your pipeline, opening up new possibilities for how you can use your machine learning models to solve real-world problems.
Overview
Section 1: What we are Building
Lecture 1 Let's look at our End Project
Section 2: Getting Started with AWS and Getting our Dataset
Lecture 2 Source Code for the Course
Lecture 3 Setting up IAM User
Lecture 4 Clarification about AWS S3
Lecture 5 Getting Data for our Project
Lecture 6 Getting dataset Part 1
Lecture 7 Getting dataset Part 2
Lecture 8 Getting dataset Part 3
Lecture 9 Getting dataset Part 4
Section 3: Setting up AWS SageMaker
Lecture 10 Create SageMaker Domain
Lecture 11 Create SageMaker Studio Notebook
Lecture 12 Learning how to Stop and Start SageMaker Notebooks
Lecture 13 Restarting our SageMaker Studio Notebook Kernel
Lecture 14 Upload and Extract Data in SageMaker
Lecture 15 Deleting Unused Files
Section 4: Exploratory Data Analysis
Lecture 16 Loading and Understanding our Data
Lecture 17 Counting total Images and getting Image ids
Lecture 18 Getting Classname Identifier
Lecture 19 Looking at Random Samples from our Dataframe
Lecture 20 Understanding Annotations
Lecture 21 Visualize Random Images Part 1
Lecture 22 Visualise Random Images Part 2
Lecture 23 Matplotlib difference between plt.show() and plt.imshow()
Lecture 24 Visualising Multiples Images at Once
Lecture 25 Correcting our Function
Lecture 26 Visualising Bounding Boxes Part 1
Lecture 27 Visualising Bounding Boxes Part 2 (Theory Lesson)
Lecture 28 Visualising Random Images with Bounding Boxes Part 1
Lecture 29 Wrong Print Statement
Lecture 30 Visualising Random Images with Bounding Boxes Part 2
Lecture 31 Read this Lesson if you have issues with Data Visualization
Section 5: Cleaning and Splitting our Data
Lecture 32 Clean our Train and Validation Dataframes
Lecture 33 Split Dataframe into Test and Train
Lecture 34 Get Images IDs
Lecture 35 Splitting IDs Theory Lesson
Lecture 36 Explanation Regarding Next video
Lecture 37 Moving Images to Appropriate Folders
Lecture 38 Count how many Train and Test Images we have
Lecture 39 Verifying that our Images have been moved Properly Part 1
Lecture 40 Verifying that our Images have been moved Properly Part 2
Section 6: Date Engineering
Lecture 41 Using Mxnet
Lecture 42 Additional Info regarding RecordIO format
Lecture 43 Using Mxnet RecordIO
Lecture 44 Correction Regarding Label width
Lecture 45 Preparing Dataframes to RecordIO format Part 1
Lecture 46 Preparing Dataframes to RecordIO format Part 2
Lecture 47 Moving Images To Correct Directory
Lecture 48 Explanation Regarding the Previous Video
Lecture 49 Verifying that all Images have been Moved Properly
Lecture 50 Read Before Proceeding to the next Lecture
Lecture 51 Creating Production .lst files (Optional)
Section 7: Data Augmentation
Lecture 52 Data Augmentation Theory
Lecture 53 Augmenting a Random Image
Lecture 54 Moving Images to new Folder structure
Lecture 55 Visualising Random Augmented Images Part 1
Lecture 56 Visualising Random Augmented Images Part 2
Lecture 57 Read this Lesson if you have issues visualising your images
Lecture 58 Creating Data Augmentation Function Part 1
Lecture 59 Creating Data Augmentation Function Part 2
Lecture 60 Checking Image Counts Before running the Function
Lecture 61 Correctional Video regarding our Function
Lecture 62 Augmenting Test Dataset and Creating test .lst Files
Lecture 63 Augmenting Train Dataset and Creating .lst File Part 1
Lecture 64 Augmenting Train Dataset and Creating .lst File Part 2
Lecture 65 Verifying that Data Augmentation has Worked
Section 8: Setting up and Creating our Training Job
Lecture 66 Increasing Service Quotas
Lecture 67 Installing dependencies and Packages
Lecture 68 Creating our RecordIO Files
Lecture 69 Uploading our RecordIO data to our S3 bucket
Lecture 70 Downloading Object Detection Algorithm from AWS ECR
Lecture 71 Setting up our Estimator Object
Lecture 72 Setting up Hyperparameters
Lecture 73 Additional Information for Hyperparameter Tuning in AWS
Lecture 74 Setting up Hyperparameter Ranges
Lecture 75 Setting up Hyperparameter Tuner
Lecture 76 Additional Information about mAP( mean average precision)
Lecture 77 Starting the Training Job Part 1
Lecture 78 Starting the Training Job Part 2
Lecture 79 More on mAP Scores
Lecture 80 Monitoring the Training Job
Lecture 81 Looking at our Finished Hyperparameter Tuning Job
Section 9: Analysing Training Job Results
Lecture 82 Deploying our Model in a Notebook
Lecture 83 Creating Visualization Function for Inferences
Lecture 84 Testing our Endpoint Part 1
Lecture 85 Testing out Endpoint Part 2
Lecture 86 Testing our Endpoint from Random Images from the Internet
Section 10: Setting up Batch Transformation
Lecture 87 Setting up Batch Transformation Job locally first
Lecture 88 Starting our Batch Transformation Job
Lecture 89 Analysing our Batch Transformation Job
Lecture 90 Visualising Batch Transformation Results
Lecture 91 Look at this lesson if you have trouble with the Visualisations
Section 11: Setting Up Our Machine Learning Pipeline
Lecture 92 Read this Before Watching the Next Lesson
Lecture 93 Setting up AWS Step Function
Lecture 94 Verify that CloudFormation has worked
Lecture 95 Configure Batch Transform Lambda Part 1
Lecture 96 Configure Batch Transform Lambda Part 2
Lecture 97 Create Check Batch Transform Job Lambda
Lecture 98 Fixing typos and Syntax Erros
Lecture 99 JSON output Format
Lecture 100 Creating Cleaning Batch output Lambda Function Part 1
Lecture 101 Creating Cleaning Batch output Lambda Function Part 2
Lecture 102 Configuring our Step Function Part 1
Lecture 103 Configuring our Step Function Part 2
Lecture 104 Configuring our Step Function Part 3
Lecture 105 Upload Test Data to S3
Lecture 106 Testing our Step Function
Lecture 107 Fixing Errors
Lecture 108 Testing our Step Function with the Corrections
Lecture 109 Verifying that our Step Function Ran Successfully
Lecture 110 Donwloading our JSON file from S3
Lecture 111 Using Event Bridge to set up Cron Job for our Machine Learning Pipeline
Lecture 112 Verify that the Cron Job works
Lecture 113 Verifying that our Pipeline Ran Successfully
Lecture 114 Setting up Production Notebook
Lecture 115 Extending Our Machine Learning Pipeline
Lecture 116 Coding our Process Job Notebook Part 1
Lecture 117 Coding our Process Job Notebook Part 2
Lecture 118 Coding our Process Job Notebook Part 3
Lecture 119 Coding our Process Job Notebook Part 4
Lecture 120 Verifying that the Images have been Saved Properly
Lecture 121 Productionizing our Notebook Part 1
Lecture 122 Productionizing our Notebook Part 2
Lecture 123 Verify that the Entire Machine Learning Pipeline works
Lecture 124 Deleted Unused items from Sagemaker EFS
Section 12: Creating our Web Application
Lecture 125 Clone the Web Application from Github
Lecture 126 Setup MongoDB
Lecture 127 Connect to MongoDB and get AWS Credentials
Lecture 128 Configuring Env file
Lecture 129 Install Node modules
Lecture 130 MERN app Walkthrough Part 1
Lecture 131 MERN app Walkthrough Part 2
Lecture 132 MERN app Walkthrough Part 3
Lecture 133 Output Images Explanation
Lecture 134 MERN app Walkthrough Part 4
Lecture 135 MERN app Walkthrough Part 5
Section 13: Outro
Lecture 136 Clean Up Resources
Lecture 137 Congratulations
For developers who want to take their machine learning skills to the next lever by being able to not only build machine learning models, but also incorporate them in a complex, secure production ready machine learning pipeline