Cloud Composer / Apache Airflow Complete Guide: DAGs, Operators, Workflows Comparison

Cloud Composer is GCP's managed Apache Airflow service and the de facto standard for orchestrating data pipelines, ETL, and ML workflows. You define DAGs (Directed Acyclic Graphs) in Python and Composer handles scheduling, dependency resolution, retries, and monitoring automatically.

Composer Version Comparison

DAG Example (BigQuery ETL)

from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-team',
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'daily_user_aggregation',
    default_args=default_args,
    schedule_interval='0 2 * * *',  # 02:00 JST daily
    start_date=datetime(2026, 1, 1),
    catchup=False,
) as dag:
    aggregate = BigQueryInsertJobOperator(
        task_id='aggregate_users',
        configuration={
            "query": {
                "query": """
                    INSERT INTO analytics.daily_users
                    SELECT DATE(event_timestamp) AS d, COUNT(DISTINCT user_id) AS dau
                    FROM raw.events
                    WHERE DATE(event_timestamp) = CURRENT_DATE() - 1
                    GROUP BY 1
                """,
                "useLegacySql": False,
            }
        },
    )

Key Operators

• BigQueryInsertJobOperator / BigQueryToGCSOperator
• DataflowStartFlexTemplateOperator
• DataprocSubmitJobOperator
• GCSToBigQueryOperator
• PubSubPublishMessageOperator
• CloudRunExecuteJobOperator (2023 onward)
• KubernetesPodOperator (runs any Pod)
• PythonOperator / BashOperator
• VertexAITrainingOperator / BatchPredictionOperator

Pricing Example (Composer 2, asia-northeast1)

Composer vs Workflows vs Dataform

Other Airflow / Orchestrator Comparisons

Typical Use Cases

• Daily batch ETL (Pub/Sub / API → BigQuery)
• Launching and managing dependencies for Dataflow / Dataproc jobs
• Vertex AI training and inference pipelines
• GCS file watching → transformation → BigQuery load
• Multi-cloud integration (AWS / on-prem DB → BigQuery)
• SaaS data integration (Salesforce / Marketo → BigQuery)

Best Practices

• Design DAGs to be idempotent (safely re-runnable)
• Control parallelism with Pools and Slots
• Manage Connections and Variables through the Airflow UI or Secret Manager
• Run Sensors in Reschedule mode to save resources
• Isolate long-running tasks with KubernetesPodOperator
• Sync DAG code via Cloud Storage and connect it to Git

* Google Cloud and Apache Airflow are trademarks of their respective owners. For the latest information, see the official Cloud Composer documentation.