Azure Data Factory (ADF) ➔ AWS Glue & AWS Step Functions / Amazon MWAA
From Azure Data Factory (ADF) visual pipelines to AWS Glue, Step Functions & MWAA.
In Azure, ADF does everything in one studio: visual ETL transformations (Mapping Data Flows), pipeline DAG orchestration, and hybrid on-premises connectivity (Self-Hosted IR). In AWS, these responsibilities are modular: AWS Glue provides serverless Spark ETL and the central Data Catalog; AWS Step Functions coordinates multi-step DAG workflows; and Amazon MWAA provides enterprise Apache Airflow Python DAGs.
1. Architectural Mechanism Comparison
Azure Data Factory (ADF)
Unified visual data integration engine: Linked Services (connections), Datasets (schemas), Activities (tasks), Pipelines (DAGs), Self-Hosted IR (on-prem network bridge), and visual Spark Mapping Data Flows.
- 100+ native connectors with visual drag-and-drop pipeline authoring.
- Self-Hosted IR securely bridges on-premises Oracle/SQL Server without opening inbound firewall ports.
- Tumbling Window triggers maintain state, enforce self-dependencies, and automate historical backfills.
AWS Glue & AWS Step Functions / Amazon MWAA
AWS provides a modular data integration ecosystem: AWS Glue for serverless Spark ETL & Data Catalog; AWS Step Functions for visual state machine orchestration; and Amazon MWAA for managed Apache Airflow.
- AWS Glue Studio & PySpark: Serverless Spark jobs billed per Data Processing Unit (DPU) with auto-scaling.
- AWS Glue Data Catalog: Central metastore compatible with Athena, Redshift Spectrum, and EMR.
- AWS Step Functions: Resilient JSON-based DAG state machine orchestration with built-in error handling and retries.
2. Interactive Terminology & Concept Bridge
Interactive Concept Bridge: Terminology & Architectural Mapping
Click any concept below to see how your AZURE knowledge directly maps into AWS.
Linked Service
Connection string with authentication metadata (e.g. ADLS Gen2, Snowflake, Oracle).
AWS Secrets Manager / IAM Role / Glue Connection
IAM Role for AWS resources; Glue Connection with Secrets Manager for external databases.
In AWS, native services authenticate via IAM Roles; external JDBC databases use Glue Connections referencing AWS Secrets Manager.
3. Visual Architecture Pipeline (AWS Glue & AWS Step Functions / Amazon MWAA)
AWS Glue 3-Stage Architecture: Crawlers & Catalog ➔ Serverless Spark DPUs ➔ Curated Sinks
Click any section below or run the simulation to explore AWS Glue serverless Spark ETL and DynamicFrames.
2. Serverless Spark ETL & DynamicFrames
Executes distributed Spark ETL jobs billed in Data Processing Units (DPUs). Glue DynamicFrames resolve semi-structured nested data without failing, while Job Bookmarks prevent re-processing old files.
“Like a flexible assembly line of robots that seamlessly reshape raw materials, even when some parts arrive in unexpected irregular shapes.”
- Serverless DPUs: Auto-scales from 2 to 100+ DPUs (4 vCPUs + 16GB RAM per DPU) in seconds.
- DynamicFrames: Native schema-drift handling using `choice` types to prevent pipeline crashes.
- Job Bookmarks: Tracks processed S3 object state across repeated pipeline runs.
4. Side-by-Side Code, CLI & Terraform Translator
Side-by-Side Code & Syntax Translator
# Azure Data Factory Mapping Data Flow Expression:
# derive(
# is_high_value = iif(amount > 1000.0, true(), false()),
# processed_at = currentDate()
# )
# sink(format: 'parquet')# AWS Glue PySpark Job (glue_etl.py)
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import col, when, current_date
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)
# Read from Glue Data Catalog
datasource = glueContext.create_dynamic_frame.from_catalog(
database="analytics_catalog",
table_name="raw_orders"
)
# Convert to DataFrame & transform
df = datasource.toDF()
clean_df = df.filter(col("amount") > 0) \
.withColumn("is_high_value", when(col("amount") > 1000.0, True).otherwise(False)) \
.withColumn("processed_at", current_date())
# Write back to S3 in Parquet format
clean_df.write.mode("append") \
.partitionBy("processed_at") \
.parquet("s3://my-company-curated-lakehouse-2026/curated_orders/")
job.commit()5. Paradigm Shift Gotchas: Traps to Avoid in AWS
Glue Job Bookmark Amnesia Without `job.commit()`
AWS Glue Job Bookmarks track processed files in S3. If your PySpark script omits `job.commit()`, Glue will never record the state, and every subsequent run will reprocess all historical files from scratch.
Always end your AWS Glue scripts with `job.commit()`, and specify `--job-bookmark-option job-bookmark-enable` in job arguments.
Glue DPU Cold-Start Delays
Running frequent micro-batch jobs on AWS Glue incurs heavy startup latency and a 1-minute minimum billing charge per DPU.
Use AWS Glue Interactive Sessions or AWS Lambda for lightweight micro-tasks, and reserve Glue DPUs for large batch workloads (>15 mins).
6. Test Your Mental Model
Quick Knowledge Check: Test Your AWS Mental Model
Solidify your cross-cloud understanding with instant feedback.