This course was created with the
course builder. Create your online course today.
Start now
Create your course
with
Autoplay
Autocomplete
Previous Lesson
Complete and Continue
Apache Airflow: Complete Hands-On Beginner to Advanced Class
Introduction
Introduction (1:17)
Your Airflow Journey (2:36)
What is Apache Airflow? (1:40)
Comparing Airflow to Other Tools (0:45)
Course Prerequisites
Extra: Install Conda (Virtual Environment Manager) (2:43)
Quiz 1: Airflow Basics
Getting Started with Apache Airflow
Components of Airflow (2:12)
Install Airflow on MacOS (5:45)
Install Airflow on Linux
Install Airflow on Windows (19:30)
Install and Run Airflow with Docker
Run Airflow Locally (3:22)
Introduction to the Airflow UI (4:05)
Introduction to the Airflow CLI (2:37)
Quiz 2: Airflow Setup
Core Concepts in Apache Airflow
What are DAGs? (4:49)
What are Default Arguments? (2:00)
What are Tasks and Operators? (5:21)
How to Define Dependencies? (5:16)
Quiz 3: Core Concepts
Loading Data to a Data Warehouse
Use Case (2:06)
Set Up (4:55)
Connections (2:50)
Load Data from Storage to BigQuery (8:01)
Run SQL Query in BigQuery (9:47)
Use Hook to List Storage Objects (5:42)
Cross-Task Communication (XComs) (6:59)
Jinja Templating and Macros (5:33)
Variables (4:44)
Quiz 4: Advanced Concepts
Analysing Data using PySpark
Use Case (5:06)
Set Up (2:36)
Create Dataproc Hadoop Cluster (4:49)
Branching (4:33)
Submit a PySpark Job (4:13)
Subdags (9:19)
Trigger Rules (3:55)
DAG Documentation (6:39)
Quiz 5: Advanced Concepts
Extending Airflow with Custom Plugins
Create a Custom Operator (17:38)
Create a Custom Sensor (8:33)
Run Custom Plugins (7:20)
Quiz 6: Custom Plugins
Testing Airflow DAGs
Load Test DAGs (6:00)
Unit Test DAGs and Operators (12:08)
Unit Test Custom Operators (10:25)
Quiz 7: Testing
Airflow in Production
Executors (7:45)
Configure Local Executor (21:50)
Configure Celery Executor
Service Level Agreements (SLAs) (3:48)
Security: Authentication, Roles, Encryption (7:10)
Write Logs to a Remote Location (4:32)
Monitor Airflow with StatsD, Prometheus and Grafana (19:26)
Error Tracking with Sentry (3:29)
Managed Airflow Services
Quiz 8: Airflow in Production
Finale
Clean Up in Google Cloud
Additional Resources
What's Next?
Course Feedback
Write Logs to a Remote Location
Lesson content locked
If you're already enrolled,
you'll need to login
.
Enroll in Course to Unlock