Apache Airflow Setup on Virtual Machine

Setting Up Apache Airflow on a Virtual Machine (Development/R&D)

This guide outlines the process of setting up Apache Airflow on a virtual machine, suitable for development and R&D purposes. The steps are demonstrated using Google Compute Engine but are largely applicable to other cloud VMs like AWS EC2 or Azure VM.

Virtual Machine Setup

  • Purpose: The setup targets a small virtual machine for development or R&D. For production use cases, additional robust steps and configurations are required.

  • Cloud Platform: Google Cloud Platform (GCP) is used for the demo, specifically Google Compute Engine.

  • Instance Creation:

    • Navigate to Google Cloud Console > Compute Engine > Create Instance.

    • Name: Airflow.

    • Machine Configuration: Default specifications are used, which include 1 shared core, 102102 CPU, and 44 GB of memory. This is sufficient for development.

    • Operating System: Debian Linux is chosen.

    • Access Scopes: Allow full access to all Cloud APIs to prevent permission issues when Airflow interacts with other Google Cloud services.

    • Firewall Rules: Traffic must be allowed on HTTP and HTTPS ports. Additionally, Airflow's web UI runs on port 80808080, so this port needs to be opened in the firewall.

    • Network: The default network is used for simplicity in the demo. However, for a production-grade setup, a custom network, subnet, and specific firewall rules are highly recommended for enhanced security.

  • Alternatives (Composer/GKE): Google Cloud Composer, which deploys Airflow on Google Kubernetes Engine (GKE), can be complex and often fails due to specific resource requirements, especially with free-tier accounts. This VM approach is a simpler, easier, and cheaper alternative.

Initial Setup and Dependencies

  • SSH Access: SSH into the newly created virtual machine. Ensure port 2222 is open in your firewall rules for SSH connectivity.

  • Update Packages: Before installing anything, update the package list:
    bash sudo apt update

  • Install Python and Dependencies: Airflow is Python-based, so Python 3, pip (Python package installer), and python3-venv (for virtual environments) are required:
    bash sudo apt install python3 python3-pip python3-venv

  • Create and Activate Virtual Environment:

    • It is a best practice to install Airflow within a Python virtual environment to manage dependenciesisolation. Create one named airflow_env and activate it:
      bash python3 -m venv airflow_env source airflow_env/bin/activate

    • Once activated, (airflow_env) will appear in your terminal prompt, indicating you are in the virtual environment.

Apache Airflow Installation

  • Install Airflow: Use pip to install the base Apache Airflow package. For a generic setup (not specific to a cloud provider at this stage), use: bash pip install apache-airflow

    • Components: Apache Airflow consists of several components: a database (to store metadata), a scheduler (to trigger tasks), and a web server (for the UI). All these are set up by this single command.

  • Verify Installation: Check the installed Airflow version and list available commands: bash airflow version # Should show Airflow 2.1 or similar airflow # Lists available Airflow CLI commands

    • Note the airflow standalone command, which starts all Airflow components for convenience.

Running Apache Airflow

  • Initial Run (airflow standalone):

    • Execute airflow standalone to start the web server, scheduler, and other components.

    • Output: The command will display critical information, including: Listening on port 8080, log in with username admin and a dynamically generated password (e.g., abcdefgh). Make a note of this password.

    • Access Web UI: Open a browser and navigate to http://[YOUR_EXTERNAL_IP]:8080. You will be prompted to log in with admin and the displayed password.

    • Initial UI View: The Airflow UI will display several example DAGs by default.

  • Persistence Issue: If you close the terminal where airflow standalone is running, Airflow components will shut down, making the UI inaccessible.

  • Running in Background (nohup):

    • To keep Airflow running even after closing the terminal, use nohup &.

    • First, stop any running airflow standalone processes.

    • Then, activate your virtual environment:
      bash source airflow_env/bin/activate

    • Run Airflow in the background, redirecting logs to a file:
      bash nohup airflow standalone > airflow_dag_log.txt 2>&1 &

    • You can now safely close the terminal, and Airflow will continue to run.

Airflow Configuration (airflow.cfg)

  • Airflow Directory: After running Airflow for the first time, an airflow directory is created in your home path (e.g., ~/airflow/).

  • Key Files/Folders:

    • airflow.cfg: The main configuration file for Airflow.

    • dags/: This directory is where your DAG (Directed Acyclic Graph) files will reside. If it doesn't exist, you'll need to create it manually.

    • standalone_admin_password.txt: If the initial password from airflow standalone was missed, this file contains the admin password.

  • Configuration Changes (airflow.cfg):

    • To hide the example DAGs from the UI, edit airflow.cfg:

      • Find the load_examples parameter (under [webserver] or [core] section depending on Airflow version).

      • Change load_examples = True to load_examples = False.

    • Important: After modifying airflow.cfg, you need to restart Airflow for changes to take effect (by killing the nohup process and restarting it).

Creating and Testing a Sample DAG

  • DAG Directory: Create a dags folder inside your airflow directory:
    bash mkdir ~/airflow/dags

  • Sample DAG (create_bucket_v1.py):

    • Create a Python file (e.g., create_bucket_v1.py) inside the ~/airflow/dags folder.

    • Purpose: This example DAG will create a Google Cloud Storage (GCS) bucket. It uses the GCSSimpleCreateBucketOperator.

    • DAG Parameters (within the Python file):

      • task_id: e.g., create_storage_bucket.

      • bucket_name: e.g., airflow-test-001.

      • project_id: Your GCP project ID (e.g., gcp-project-id).

  • Install Google Cloud Provider: The GCSSimpleCreateBucketOperator is part of the Google Cloud provider for Airflow. Install it:
    bash pip install apache-airflow-providers-google

  • Restart Airflow: After adding the DAG file and installing dependencies, restart Airflow (kill the nohup process and re-run the nohup command) to allow it to pick up the new DAG.

  • Verify and Trigger DAG in UI:

    • Go back to the Airflow UI (http://[YOUR_EXTERNAL_IP]:8080).

    • You should now see create_gcs_bucket listed among your DAGs (if load_examples is False, it might be the only one).

    • Trigger the DAG: Click the