Cloud Composer (Apache Airflow) ➔ Azure Data Factory (ADF)
From Cloud Composer (Apache Airflow Python DAGs) to Azure Data Factory (Visual Serverless Pipelines).
Cloud Composer requires maintaining a GKE cluster running Airflow workers in Python ($250+/month baseline even when idle). Azure Data Factory is 100% serverless ($0 baseline) with a visual drag-and-drop canvas and the Self-Hosted Integration Runtime that acts as an outbound-only secure gateway to your private corporate databases.
1. Architectural Mechanism Comparison
Cloud Composer (Apache Airflow)
Managed Apache Airflow where 100% of pipeline DAGs, operators, and dynamic task graphs are authored in Python code.
- Complete programmatic flexibility using Python code and custom operators.
- Rich Airflow open-source ecosystem, plugins, and custom hooks.
- Version-controlled Git workflows for all DAG definitions.
Azure Data Factory (ADF)
Serverless visual ETL and orchestration service backed by JSON schemas. Connects to 100+ native sources with built-in Self-Hosted IR for hybrid on-prem networks.
- 100% serverless ($0 idle cost vs. Airflow GKE cluster baseline).
- Self-Hosted Integration Runtime (SHIR) securely connects on-prem with outbound-only port 443.
- Mapping Data Flows executes visual transformations on serverless Apache Spark.
2. Interactive Terminology & Concept Bridge
Interactive Concept Bridge: Terminology & Architectural Mapping
Click any concept below to see how your GCP knowledge directly maps into AZURE.
Airflow DAG (Python File)
Directed Acyclic Graph defining tasks and execution dependencies.
ADF Pipeline (JSON Schema)
Visual workflow grouping activities with On-Success/On-Failure links.
Both represent end-to-end data workflows with dependency management.
3. Visual Architecture Pipeline (Azure Data Factory (ADF))
Azure Data Factory (ADF) 3-Stage Architecture: Ingest ➔ Integration Runtime & Spark ➔ Sinks
Click any section below or run the simulation to see how ADF orchestrates hybrid data pipelines.
2. Integration Runtimes (IR) & Mapping Data Flows
The Integration Runtime executes pipeline activities. Azure IR manages serverless cloud movement; Self-Hosted IR securely connects private on-premise networks with outbound-only HTTPS (port 443); Mapping Data Flows compiles visual graphs to Apache Spark.
“Like a secure armored courier that can enter high-security bank vaults (on-prem) without leaving the back doors unlocked.”
- Self-Hosted IR (SHIR): Outbound-only port 443 (zero inbound firewall holes required).
- Azure IR: Managed serverless cloud compute auto-allocating Data Integration Units (DIUs).
- Mapping Data Flows: Drag-and-drop transformations compiled into Spark Scala jobs.
4. Side-by-Side Code, CLI & Terraform Translator
Side-by-Side Code & Syntax Translator
# Cloud Composer (Airflow) CLI Trigger
gcloud composer environments run my-airflow-env \
--location us-central1 \
dags trigger -- DailySalesETL# Azure Data Factory CLI Trigger
az datafactory pipeline create-run \
--factory-name adf-corp-prod \
--name DailySalesETL \
--resource-group rg-analytics5. Paradigm Shift Gotchas: Traps to Avoid in AZURE
The Self-Hosted IR Outbound Rule
Network teams often assume Azure needs inbound access to on-prem databases and deny requests.
Clarify to security teams that SHIR uses outbound-only port 443 to poll Azure for jobs.
6. Test Your Mental Model
Quick Knowledge Check: Test Your AZURE Mental Model
Solidify your cross-cloud understanding with instant feedback.