Integrate External Services with Apache Airflow
Integrate External Services with Apache Airflow
Released/Updated: Aug 01, 2024
Duration: 23m 41s | .MP4 1920x1080, 30 fps(r) | AAC, 48000 Hz, 2ch | 82 MB
Genre: eLearning | Language: English
Released/Updated: Aug 01, 2024
Duration: 23m 41s | .MP4 1920x1080, 30 fps(r) | AAC, 48000 Hz, 2ch | 82 MB
Genre: eLearning | Language: English
This course will teach you how to connect Apache Airflow with Google Cloud Storage, Google Cloud BigQuery, and AWS S3 enabling data management and processing across cloud platforms.
What you'll learn
Apache Airflow provides built-in operators to connect to external cloud services. In this course, Integrate External Services with Apache Airflow, you’ll gain the ability to connect and manage various external services within your Airflow workflows. You'll connect your Airflow workflow to Google Cloud Storage and Cloud BigQuery as well as to AWS S3, enabling data management and processing across cloud platforms. First, you’ll explore how to set up permissions on Google Cloud using a service account to enable Airflow to access Google Cloud Storage buckets and BigQuery datasets. Next, you'll discover how to read files from your local machine and upload to Google Cloud, create a BigQuery dataset, and create an external table in BigQuery. Then, you'll integrate your workflow with AWS S3 buckets to transfer data from S3 to Google Cloud Storage. Finally, you’ll use hooks to work with S3, Cloud Storage, and BigQuery - hooks give you much more control over how you access your data from external services. When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to integrate and manage external services effectively within your data workflows.
More Info