Airflow UI Detailed Walkthrough

Airflow UI Walkthrough

This video provides a detailed walkthrough of the Airflow UI, primarily for beginners starting with Airflow, emphasizing its role in creating, troubleshooting, and monitoring data pipelines.

Airflow Setup and Access

  • Local Deployment: The instructor set up Airflow on a local system using Docker Desktop, accessible via port 80808080.

  • Web Server Component: The Airflow UI is hosted by the web server component. If the UI is not visible, it indicates an issue with the web server.

  • Logs: Various component logs (web server, trigger, scheduler) are visible through the Docker container logs, confirming the system's running state and resource utilization.

  • Credentials: Default username is admin. The password is found in a standalone_admin_password file (for local setups or virtual machines) generated during initial setup. For production environments (e.g., Google Composer, Kubernetes), credentials are obtained from the admin or through IAM.

Airflow Homepage and Main Menu

  • Default View: Upon successful login, the user lands on the DAGs page, which is the default homepage.

  • Warnings: Local setups might show warnings (e.g., regarding PostgreSQL or Cloud Composer) which can typically be ignored in non-production environments.

  • Main Menu Options: The top navigation bar includes:

    • DAGs

    • Cluster Activity

    • Dataset

    • Security

    • Browse

    • Admin

    • Docs

DAGs Page Details

  • Overview: Displays all available DAGs loaded by the web server or scheduler.

  • DAG Status Counts:

    • All DAX: Total number of DAGs (e.g., initially 17).

    • Active: Enabled/unpaused DAGs (e.g., initially 12).

    • Paused: Disabled DAGs (e.g., initially 5).

  • Pausing/Unpausing DAGs:

    • DAGs can be easily toggled between paused and unpaused states by clicking the blue button next to their name.

    • A blue toggled button indicates an active (unpaused) DAG, allowing it to be scheduled.

    • A gray toggled button indicates a paused DAG, preventing it from being scheduled.

    • Example: Unpausing two DAGs changed the counts from 12 active and 5 paused to 14 active and 3 paused (out of 17 total).

  • DAG Details View: Clicking a DAG name opens its detailed view.

    • Grid View: The default view, showing tasks but not explicit dependencies.

    • Graph View: Provides a comprehensive visual representation of the DAG's flow, including task dependencies (sequential, parallel) and their execution order.

  • Task Statuses (Color Coding):

    • Light Green: Running

    • Dark Green: Success

    • Blue: Up for Reschedule

    • Gray: Skewed

    • Red: Failed

    • Pink: Skipped

    • White (no color): No Status

  • Checking Task Logs:

    • Click on a specific task (e.g., a failed task) in the Graph View.

    • The Log button becomes enabled. Clicking it displays detailed execution logs.

    • Logs can be wrapped for easier readability or downloaded for offline analysis.

    • Errors and exceptions (e.g., random failures, HTTP connection errors, temporary service unavailable) are visible here.

  • Viewing DAG Code: The Code tab displays the full Python code of the DAG as uploaded, useful for quick reference.

  • DAG Frequency and Time Zones:

    • The Schedule of a DAG (e.g., every ten minutes, every day) is displayed.

    • The time zone for viewing run times can be changed (e.g., from UTC to IST) from the UI.

  • Manual DAG Triggering:

    • For DAGs not on a schedule (manual DAGs).

    • Navigate back to the main DAGs page, locate the desired DAG.

    • Click the Trigger DAG button (play icon) available in the DAX list view.

    • This initiates an immediate manual run of the DAG, with its status visible on the DAG's detail page.

  • Calendar View:

    • Shows the historical runs of a DAG, day by day, for an entire year.

    • Allows quick identification of failures or runs on specific dates.

    • Also indicates planned future runs.

  • Task Duration: Provides statistics on how much time each task within a DAG takes to complete, displayed in seconds.

Main Menu Options Deep Dive

Cluster Activity

This tab provides an overall health and status overview of the Airflow instance.

  • DAG Statistics:

    • Unpaused DAGs: Active DAG count (e.g., 14 out of 17).

    • Paused DAGs: Paused DAG count (e.g., 3 out of 17).

    • Top five longest DAG run to finish: Identifies DAGs with extended execution times.

  • Component Health: Displays the health status of core Airflow components:

    • Web Server: (e.g., healthy)

    • Metadata Database: (e.g., healthy)

    • Scheduler: (e.g., healthy)

    • Trigger: (e.g., unhealthy) - useful for troubleshooting issues.

  • DAG Run Stats: Visualizes the distribution of DAG run states (e.g., failed, running, queued). Useful for detecting issues like DAGs getting stuck in the queued state due to scheduler or worker problems.

  • Run Type Distribution: Shows how many DAGs were manually triggered, scheduled, or backfilled.

Security

This section is dedicated to user and role management (IAM).

  • List Users: Displays existing users (e.g., default admin user with Admin role).

  • Add New User: Allows creation of new users by providing First Name, Last Name, Username, Password, Email ID, and Role.

  • Roles:

    • Predefined roles include Admin, Public, Viewer, User, and Operational.

    • Each role has specific permissions. For example, Admin has all permissions, while Public has none.

  • Other Options: Includes User Statistics, Action, and Resource Permission, all related to access control and auditing.

Browse

This tab provides detailed insights into DAG and task instances.

  • DAG Runs:

    • Lists every execution of a DAG.

    • Details include start time, end time, logical date, run ID (scheduled run type, manual run ID for manual runs), and total execution time.

    • Allows filtering by DAG ID to view all runs for a specific DAG.

    • Example: A complex DAG ran seven times, with four scheduled and one manual run, and two more scheduled runs.

  • Task Instances:

    • Lists every execution of individual tasks within DAGs.

    • Provides a combined view of all task statuses (e.g., failed, succeed, skipped, upstream_failed) across all runs of a specified DAG.

    • Allows filtering by DAG ID or Task ID.

  • Other Options: Includes DAG Dependencies (for chained DAGs, not demonstrated) and SLA Misses.

Admin

This section allows administrative configurations within Airflow.

  • Variables:

    • Global key-value pairs that can be used across DAGs.

    • New variables can be created by clicking Add New Record, providing a Key (e.g., username) and a Value (e.g., airflow_some_variable), and saving.

  • Connections:

    • Manages connections to external systems.

    • Default connections are pre-populated (e.g., airflow_db, athena, aws, google_cloud, mysql, postgres, sftp).

    • New connections can be created by Add New Record and specifying a Connection Type (e.g., Azure, Docker, FTP, Email, Cloud SQL, Google BigQuery). Based on the type, parameters like Host, Schema, Login, Password, and Project ID (for Google BigQuery) will be requested.

  • Other Options: Includes Plugins, Providers, Pools, and XComs, which are covered in more advanced videos.

Docs

This section provides direct links to important Airflow resources.

  • External Links: Provides quick access to the Airflow GitHub repository (open-source code), Airflow Documentation, Airflow Website, and Swagger UI (for API exploration).

Advanced UI Operations

Rerunning Failed DAGs/Tasks (Clearing)

In Airflow, rerunning is referred to as