Cloud Composer (Managed Airflow)
Fully managed Apache Airflow service on Google Cloud for orchestrating complex end-to-end data workflows.
Fully managed Apache Airflow service on Google Cloud for orchestrating complex end-to-end data workflows.
What is Cloud Composer (Managed Airflow)?
Google Cloud Composer is a fully managed workflow orchestration service built directly on Apache Airflow. Composer manages Airflow Webservers, Schedulers, Worker nodes, and underlying GKE clusters automatically, enabling data teams to author, schedule, and monitor complex pipelines as Python code (DAGs). Composer 2 features serverless autoscaling worker pools powered by GKE Autopilot.
Decision Guide: When to Choose This Service
Use Cloud Composer (Managed Airflow) when you need Python-authored DAGs (Directed Acyclic Graphs) to orchestrate multi-step data pipelines connecting BigQuery, Dataflow, Cloud Storage, Dataproc, and external APIs with rich dependency management and monitoring.
Do NOT use Cloud Composer for simple event-driven function chains (use Google Cloud Workflows) or for running heavy data processing computations directly on the scheduler nodes (delegate compute to BigQuery/Dataflow).
Core Architectural Pillars
Workflows declared purely as Python files defining task order and execution dependencies.
Operators perform actions (e.g. `BigQueryInsertJobOperator`); Sensors poll for conditions (e.g. `GCSObjectExistenceSensor`).
Native UI for triggering runs, inspecting Gantt charts, reading task logs, and managing connection secrets.
Small, Medium, Large presets that dynamically configure scheduler CPU, webserver RAM, and worker autoscaling.