Apache Airflow: Understanding DAGs - Comprehensive Study Notes

Introduction to Apache Airflow DAGs

  • This video dives into the core of Apache Airflow: Directed Acyclic Graphs (DAGs).

  • It aims to explain what a DAG is, its importance, key components, how to write, load, and trigger DAGs, including examples.

  • The goal is to equip viewers to write their own Airflow workflows.

What is a DAG in Airflow?

  • DAG stands for Directed Acyclic Graph.

  • It represents the entire workflow structure in Airflow.

  • A DAG is fundamentally a combination of tasks.

  • Components of a DAG include:

    • Schedule (when it runs)

    • Tasks (individual units of work)

    • Task Dependencies (order of execution)

    • Trigger Rules (conditions for task execution)

    • Default Arguments (common parameters for tasks)

    • Connections

    • Variables

  • The DAG defines the execution order and control flow of these components.

Typical DAG Structure and Examples

  • A typical DAG is visualized as a network of interconnected boxes (tasks) with arrows (dependencies).

  • Example Workflow Illustration:

    • Start Task: A dummy operator marking the beginning.

    • Extract Data: A task to pull initial data.

    • Extract Master Data: (Assumed to be a parallel or subsequent task to extract data based on context, then dependencies kick in).

    • Parallel Tasks: After Extract Master Data completes, three tasks (Task 1, Task 2, Task 3) trigger independently and run in parallel.

    • Conditional End Task: Once Task 1, Task 2, and Task 3 are completed, it triggers either Notify Failure or Join Transform based on their defined trigger rules.

  • Airflow UI Demonstrations:

    • Simple Test DAG: Demonstrated in the Airflow UI with start_task, print_date, and print_hello tasks, showing parallel execution after start_task.

    • Sequential DAG: Illustrated an extract →\rightarrow transform →\rightarrow load pattern, where each task is dependent on the successful completion of the previous one.

      • start_task →\rightarrow extract_data →\rightarrow transform_data →\rightarrow load_data →\rightarrow end_status (dummy operator to mark completion).

  • DAG Schedule in UI:

    • The Airflow UI displays the schedule_interval for each DAG (e.g., None for manual trigger, Daily for daily execution).

    • DAGs can be enabled or disabled; only enabled DAGs will run on their scheduled time.

Key Components of a DAG

When writing a DAG definition file (a Python .py file), several components are defined:

  1. Default Arguments:

    • A Python dictionary holding common parameters for tasks within the DAG.

    • Examples: start_date, retries, retry_delay, email.

    • Purpose: To avoid code repetition across tasks and keep the DAG definition clean and concise.

  2. Tasks:

    • The individual units of work or operations that make up the workflow.

    • Examples provided: start_task (dummy), print_hello_task, print_date_task.

  3. Task Dependencies:

    • Define the execution order of tasks.

    • Can establish sequential or parallel execution flows.

    • Example: start_task must complete before print_hello_task and print_date_task can run in parallel. This is defined using bitshift operators, e.g., start_task >> [print_hello_task, print_date_task].

  4. Schedule Interval:

    • Controls when the DAG runs.

    • Defined using the schedule_interval parameter in the DAG constructor.

    • Can be specified using:

      • Cron expressions: e.g., `