Snowpark : Data Engineering With Snowflake.

Learn fundamentals of Snowflake-snowpark API.

What you'll learn

Connect Snowpark API with snowflake.

Basic read and write operations using Snowpark.

Do's and Don'ts while using Snowpark.

Building data components and basic data pipeline.

Requirements

Basic knowledge on python.

Description

What is Snowpark ?With Snowpark, Snowflake allows developers to write code in their preferred language.Along with Snowflake’s original SQL interface now snowflake allows you to write code in,         1. Python         2. Scala         3. JavaSome of the key features of Snowpark are,Your code will be pushed to snowflake leveraging the compute power of snowflake warehouses.You will not end up exporting data to different environment but rather your code is shipped to the data.You can build complex data pipelines or data products using SnowparkSnowpark also address below overheads in conventional data pipelines,Long startup time of node clusters: Systems like Hadoop and Spark requires cluster of nodes to process data. Most of the time it takes 5-10 minutes to just start the cluster. In case of Snowpark we will be using snowflake warehouse to process our data.Problem of small files , Problem of using right joins to shuffle data across nodes, Problem of garbage collection. uncertainty when the compute nodes goes down.All the above problems are well handled with Snowpark.What you will learn ?You will learn the basics of SNOWPARK API.     > Basic read and write operations.    > Read data from s3 and load that to snowflake table.    > We will do deep analysis of how SNOWPARK API works.    > Do's and Don'ts of SNOWPARK.    > Build data components to process data.    > Build data pipeline to process data.

Overview

Section 1: Introduction

Lecture 1 Snowpark introduction

Lecture 2 Test connection

Lecture 3 Snowpark-demo part 1

Lecture 4 Snowpark-demo part2

Section 2: Snowpark– Read operations

Lecture 5 Create dataframe–part1

Lecture 6 Create dataframe–part2

Lecture 7 Apply schema

Lecture 8 Create dataframe–part3

Lecture 9 Create dataframe–part4

Lecture 10 Read from table–part1

Lecture 11 Read from s3 csv

Lecture 12 Read from s3 json

Section 3: Assignments– Read operation

Section 4: Snowpark–-Write operations

Lecture 13 Basic write operation

Lecture 14 Write from s3 to table–json

Lecture 15 Write from s3 to table–csv

Section 5: Assignments – write operations

Section 6: Assignments– Read and write semistructured data.

Section 7: Snowpark–Copy commands

Lecture 16 Write using copy

Section 8: Transformations and query

Lecture 17 Aggregation in snowpark

Lecture 18 Group by –part 1

Lecture 19 Group by –part 2

Lecture 20 Window function

Lecture 21 Joins

Lecture 22 Using in clause

Section 9: Assignments–-Transformation and query

Lecture 23 Prepare data

Section 10: Building generic components

Lecture 24 Download resources

Lecture 25 Create snow connection component– part1

Lecture 26 Create snow connection component–part2

Lecture 27 Copy to snowflake table

Lecture 28 Creating configuration file

Lecture 29 Collect rejects

Lecture 30 Copy semi structured data part 1

Lecture 31 Copy semi structured data part 2

Lecture 32 Map columns

Lecture 33 Map columns solution

Lecture 34 Summary

Developers who want's to lean Snowflake-Snowpark.