Cloud Arena
Back to Data & Analytics

AWS Glue

aws

Serverless data integration and ETL service featuring automated Crawlers, Glue Data Catalog, and Spark DynamicFrames (DPUs).

DATA & ANALYTICS
•Amazon Web Services

Serverless data integration and ETL service featuring automated Crawlers, Glue Data Catalog, and Spark DynamicFrames (DPUs).

Click any section above to jump directly to it
GCP ➔ AWS Bridge
Fast-Track Mental Model Translation

Coming from Google Cloud?

Compare with Google Cloud Dataproc / Dataplex

AWS Glue combines the serverless Apache Spark capabilities of Dataproc Serverless with the metadata cataloging of Dataplex.

What's Conceptually Identical
  • Execute distributed Apache Spark ETL jobs at scale.
  • Centralized metastore cataloging data lake tables and schemas.
  • Automatic scaling of Spark compute executors.
Key Architectural Shifts in AWS
  • Compute Units: AWS Glue bills in Data Processing Units (DPUs at $0.44/hr); Dataproc Serverless bills in DCUs.
  • DataFrame API: AWS Glue provides custom `DynamicFrames` with schema choice types; Dataproc uses standard PySpark DataFrames.
  • State Management: AWS Glue has native `Job Bookmarks` built into the runtime; GCP requires managing checkpoint offsets manually.

What is AWS Glue?

AWS Glue is a fully managed, serverless data integration service that makes it easy to discover, prepare, combine, and integrate data for analytics, machine learning, and application development. Glue provides automated Crawlers to infer schemas, a centralized Hive-compatible Data Catalog, serverless Apache Spark ETL execution billed in Data Processing Units (DPUs), lightweight Python Shell jobs, streaming ETL, and declarative Data Quality (DQDL) validation.

Decision Guide: When to Choose This Service

When to Use AWS Glue

Use AWS Glue for serverless Apache Spark ETL pipelines, automated schema discovery across Amazon S3 data lakes, centralized metastore management, and data preparation.

When NOT to Use (Recommended Alternatives)

Do NOT use AWS Glue if you need sub-second streaming transformation response times (use Amazon Kinesis Analytics / Flink instead) or full interactive Jupyter cluster orchestration (use Amazon EMR Studio).

Core Architectural Pillars

Data Processing Units (DPUs)

Standard compute unit (1 DPU = 4 vCPUs, 16GB RAM at $0.44/hr) with auto-scaling.

Glue Data Catalog

Centralized Hive-compatible metadata repository for Athena, Redshift, and EMR.

Glue Crawlers

Automated scanners that inspect S3 data, infer schemas, and register partitions.

DynamicFrames

Spark DataFrame extension providing resilient handling of schema drift, choice types, and nested JSON.

Job Bookmarks

Stateful tracking mechanism preventing re-processing of old S3 objects on rerun.

File Grouping

Eliminates the small files problem by grouping tiny S3 files into 128MB partitions in memory.

Glue Flex

34% discounted execution class ($0.29/DPU-hr) for non-urgent batch pipelines and backfills.