Comparing Airflow to Other Tools | datastack.tv

Autoplay
Autocomplete

Previous Lesson Complete and Continue

Apache Airflow: Complete Hands-On Beginner to Advanced Class

Introduction

Introduction (1:17)
Your Airflow Journey (2:36)
What is Apache Airflow? (1:40)
Comparing Airflow to Other Tools (0:45)
Course Prerequisites
Extra: Install Conda (Virtual Environment Manager) (2:43)
Quiz 1: Airflow Basics

Getting Started with Apache Airflow

Components of Airflow (2:12)
Install Airflow on MacOS (5:45)
Install Airflow on Linux
Install Airflow on Windows (19:30)
Install and Run Airflow with Docker
Run Airflow Locally (3:22)
Introduction to the Airflow UI (4:05)
Introduction to the Airflow CLI (2:37)
Quiz 2: Airflow Setup

Core Concepts in Apache Airflow

What are DAGs? (4:49)
What are Default Arguments? (2:00)
What are Tasks and Operators? (5:21)
How to Define Dependencies? (5:16)
Quiz 3: Core Concepts

Loading Data to a Data Warehouse

Use Case (2:06)
Set Up (4:55)
Connections (2:50)
Load Data from Storage to BigQuery (8:01)
Run SQL Query in BigQuery (9:47)
Use Hook to List Storage Objects (5:42)
Cross-Task Communication (XComs) (6:59)
Jinja Templating and Macros (5:33)
Variables (4:44)
Quiz 4: Advanced Concepts

Analysing Data using PySpark

Use Case (5:06)
Set Up (2:36)
Create Dataproc Hadoop Cluster (4:49)
Branching (4:33)
Submit a PySpark Job (4:13)
Subdags (9:19)
Trigger Rules (3:55)
DAG Documentation (6:39)
Quiz 5: Advanced Concepts

Extending Airflow with Custom Plugins

Create a Custom Operator (17:38)
Create a Custom Sensor (8:33)
Run Custom Plugins (7:20)
Quiz 6: Custom Plugins

Testing Airflow DAGs

Load Test DAGs (6:00)
Unit Test DAGs and Operators (12:08)
Unit Test Custom Operators (10:25)
Quiz 7: Testing

Airflow in Production

Executors (7:45)
Configure Local Executor (21:50)
Configure Celery Executor
Service Level Agreements (SLAs) (3:48)
Security: Authentication, Roles, Encryption (7:10)
Write Logs to a Remote Location (4:32)
Monitor Airflow with StatsD, Prometheus and Grafana (19:26)
Error Tracking with Sentry (3:29)
Managed Airflow Services
Quiz 8: Airflow in Production

Finale

Clean Up in Google Cloud
Additional Resources
What's Next?
Course Feedback

Teach online with

Comparing Airflow to Other Tools

Complete and Continue

Discussion

1 comments