intermediate

Data Engineer Path

Master modern cloud data engineering architectures from ingestion to warehouse analytics.

45 mins5 Architectural Steps

Choose Platform:

STEP 1

Real-time Event Ingestion & Buffering

Ingest and buffer millions of incoming clickstream and IoT events per second with low latency.

Google Cloud Platform

Cloud Pub/Sub

Explore GCP Guide

Pub/Sub decouples publishers and subscribers globally across regions with at-least-once delivery and zero cluster provisioning.

Key Concepts:

Topics & SubscriptionsPush vs Pull DeliveryAt-Least-Once GuaranteesDead Letter Topics

CLI Example:

gcloud pubsub topics publish user-events --message='{"event":"click"}'

STEP 2

Stream & Batch Data Processing

Cleanse, filter, transform, and window streaming data in real-time.

Google Cloud Platform

Cloud Dataflow

Explore GCP Guide

Dataflow automatically provisions worker pools, dynamically balances keys (liquid sharding), and supports exactly-once processing.

Key Concepts:

Apache Beam SDKStreaming EngineEvent-Time Windowing & WatermarksAutoscaling Workers

CLI Example:

gcloud dataflow jobs run stream-etl --gcs-location gs://templates/PubSub_to_BigQuery

STEP 3

Enterprise Data Warehousing & Analytics

Store and analyze petabytes of historical structured data with sub-second SQL execution.

Google Cloud Platform

BigQuery

Explore GCP Guide

BigQuery separates compute (Dremel slots) from storage (Colossus Capacitor), enabling instantaneous multi-TB SQL scans without cluster management.

Key Concepts:

Decoupled Storage & ComputeCapacitor Columnar FormatPartitioning & ClusteringBigQuery ML

CLI Example:

bq query --use_legacy_sql=false 'SELECT country, COUNT(*) FROM `project.data.users` GROUP BY 1'

STEP 4

Workflow Orchestration & Scheduling

Schedule and manage complex multi-step data pipelines and dependencies as code.

Google Cloud Platform

Cloud Composer

Explore GCP Guide

Composer 2 manages Airflow schedulers, webservers, and GKE Autopilot workers automatically.

Key Concepts:

Python DAGsAirflow Operators & SensorsGCS DAG SynchronizationAuto-scaling Workers

CLI Example:

gcloud composer environments storage dags import --environment=prod-env --source=dags/daily_dag.py

STEP 5

Serverless SQL & Lakehouse Analytics

Query open data lake files in place with zero cluster provisioning and pay-per-query pricing.

Google Cloud Platform

BigQuery BigLake & Dataproc

Explore GCP Guide

Spin up Spark/Hadoop clusters in under 90 seconds or query open Parquet/Iceberg files directly via BigLake.

Key Concepts:

Ephemeral ClustersBigLake Storage PoliciesGCS Connector (No HDFS)Dataproc Serverless

CLI Example:

gcloud dataproc jobs submit pyspark gs://bucket/job.py --cluster=spark-cluster