Cloud Arena
Back to Data & Analytics

Azure Data Factory (ADF)

azure

Cloud-scale serverless data integration and ETL service with 100+ native connectors, visual Mapping Data Flows, and hybrid Self-Hosted IRs.

DATA & ANALYTICS
•Microsoft Azure

Cloud-scale serverless data integration and ETL service with 100+ native connectors, visual Mapping Data Flows, and hybrid Self-Hosted IRs.

Click any section above to jump directly to it
GCP ➔ Azure Bridge
Fast-Track Mental Model Translation

Coming from Google Cloud?

Compare with Google Cloud Data Fusion / Composer

Azure Data Factory combines the visual pipeline design of Cloud Data Fusion with the orchestration capabilities of Cloud Composer (Airflow).

What's Conceptually Identical
  • Visual ETL pipeline designers with rich connectors for enterprise databases.
  • Hybrid integration connecting on-premises data centers to cloud data lakes.
  • Triggering mechanisms based on schedule (cron), event-driven file arrival, or API calls.
Key Architectural Shifts in Azure
  • Compute: ADF executes Mapping Data Flows on managed Spark and offers Self-Hosted IRs for on-prem connectivity; GCP Dataflow executes Apache Beam pipelines.
  • Orchestration: ADF has native visual control flows (If, Switch, ForEach, Until); GCP relies on Python Apache Airflow DAGs in Cloud Composer.
  • Pricing: ADF bills per activity run and DIU-hour; GCP Dataflow bills per vCPU/RAM-hour.
Syntax & Command Translator
Trigger Pipeline Run
GCP:gcloud composer environments run my-env dags trigger -- my_pipeline
Azure:az datafactory pipeline create-run --factory-name adf-prod --name DailySalesETL --resource-group rg-analytics

What is Azure Data Factory (ADF)?

Azure Data Factory is Azure's cloud-native data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale. With 100+ built-in connectors, ADF bridges on-premises enterprise data sources with cloud data lakes using Self-Hosted Integration Runtimes (SHIR) and runs visual scale-out data transformations using Mapping Data Flows on managed Apache Spark clusters.

Decision Guide: When to Choose This Service

When to Use Azure Data Factory (ADF)

Use Azure Data Factory to build automated enterprise ETL/ELT pipelines, extract data from on-premises databases using Self-Hosted Integration Runtimes, execute code-free visual Spark transformations, and orchestrate complex dependency DAGs.

When NOT to Use (Recommended Alternatives)

Do NOT use Azure Data Factory for sub-second real-time stream processing (use Azure Stream Analytics or Event Hubs instead) or simple one-off file copies (use AzCopy).

Core Architectural Pillars

4 Building Blocks

Linked Services (connections), Datasets (schemas), Activities (tasks), Pipelines (DAGs).

Integration Runtimes (IR)

Azure IR (Cloud), Self-Hosted IR (On-premise private VNet bridge), and Azure-SSIS IR.

Copy Activity Tuning

Data Integration Units (DIUs) and Degree of Copy Parallelism for multi-GB/s ingestion.

Mapping Data Flows

Code-free visual Apache Spark transformations with Change Data Capture and SCD Type 2.