All Bridges/AWS to GCP Data Engineering Bridge/Real-Time Streaming & Messaging

AWS ➔ GCP Deep Dive

•Real-Time Streaming & Messaging

Amazon Kinesis Data Streams ➔ Google Cloud Pub/Sub

From Amazon Kinesis fixed shard throughput & KCL leases to Google Cloud Pub/Sub auto-scaling topics.

The 30-Second Mental Model Shift

In AWS Kinesis, streaming capacity is constrained by physical Shards (1 MB/s write each) that require proactive capacity planning, resharding scripts, and DynamoDB lease coordination. In Google Cloud Pub/Sub, shards do not exist: Topics and Subscriptions auto-scale dynamically across Google's global network, and message routing is decoupled from physical partition boundaries.

1. Architectural Mechanism Comparison

AWS (What You Know)

Source

Amazon Kinesis Data Streams

Partition-based streaming organized into discrete Shards (1 MB/s write, 2 MB/s read). Publishers hash partition keys via MD5; consumer applications track leases and checkpoints in Amazon DynamoDB via KCL.

Key Architecture Strengths:

Deterministic hash-based partition routing with strict in-shard message ordering.
Enhanced Fan-Out (EFO) delivers dedicated 2 MB/s HTTP/2 push pipes per consumer.
Tight integration with Kinesis Data Firehose for zero-code S3 Parquet landing.

GCP (How It Works)

Mastery Target

Google Cloud Pub/Sub

Serverless global messaging service. Publishers send messages to Topics; Subscribers pull or receive pushes from Subscriptions. Capacity auto-scales dynamically without managing shards or DynamoDB lease tables.

Why Google Cloud Built It This Way:

Zero shard management: Scales horizontally from zero to millions of messages per second automatically.
Global topic availability: Publish from any region and consume worldwide seamlessly.
Direct BigQuery and Cloud Storage Subscriptions for zero-code streaming data lake ingestion.

2. Interactive Terminology & Concept Bridge

Interactive Concept Bridge: Terminology & Architectural Mapping

Click any concept below to see how your AWS knowledge directly maps into GCP.

Mapping Deep Dive

Key Paradigm Shift

⚠️ Requires shifting your mental model

AWS (What You Know)

Kinesis Shard (1 MB/s in, 2 MB/s out)

Discrete unit of streaming capacity requiring proactive scaling.

GCP (How It Works)

Pub/Sub Global Auto-Partitioning

Dynamic multi-tenant scaling with zero manual shard provisioning.

The Architectural Mental Shortcut:

In Pub/Sub, you never provision, split, or merge shards; the service scales dynamically based on publisher throughput.

3. Visual Architecture Pipeline (Google Cloud Pub/Sub)

Pub/Sub Mental Model: The “Publish & Fan-Out” Pipeline

Click any section below or run the simulation to see how messages travel from Publishers to Topics and Fan-Out to Subscriptions.

1. Publishers

Mobile Apps / IoT Devices

Publish

2. Central Topic

Fan-Out

3. Subscriptions & Workers

Why Decoupling is Powerful:You can connect 10 new subscriber services tomorrow (e.g. Fraud Detection, Slack Alerts) without modifying a single line of your publisher app!

Central Event Bus

Concept Guide

2. The Topic (The Central Bulletin Board)

The named channel where messages land

A Topic is a named channel (e.g. `order-placed`). When a publisher sends a message here, Pub/Sub automatically duplicates and routes it to every subscription attached to this topic.

Real-World Analogy:

Like a YouTube channel. When a creator uploads a video, every subscriber receives a copy in their feed.

Delivery Model

At-Least-Once

Autoscaling

100% Serverless

4. Side-by-Side Code, CLI & Terraform Translator

Side-by-Side Code & Syntax Translator

AWS Syntax

# AWS Kinesis CLI Put Record
aws kinesis put-record \
  --stream-name telemetry-stream \
  --partition-key device_99 \
  --data '{"device_id":"99","temp":24.5}'

GCP Equivalent

# Google Cloud Pub/Sub CLI Publish Message
gcloud pubsub topics publish telemetry-topic \
  --message='{"device_id":"99","temp":24.5}' \
  --attribute=device_id=device_99

Code Translation Notes:Pub/Sub supports message payloads along with key-value attributes used for subscription filter policies.

5. Paradigm Shift Gotchas: Traps to Avoid in GCP

Gotcha #1

high

Message Ordering Is Opt-In in Pub/Sub

The Trap:

An AWS engineer building a financial ledger might assume messages published sequentially will arrive in exact order. In Pub/Sub, parallel workers will receive messages out of order unless message ordering is explicitly configured!

How to Avoid It:

Set `enable_message_ordering = true` on the Subscription and pass a consistent `ordering_key` with every published message.

Gotcha #2

medium

Acknowledgment Deadline Expiration & Duplicate Processing

The Trap:

If a downstream database query hangs for 15 seconds while the `ack_deadline` is 10 seconds, Pub/Sub will redeliver the message to another worker, leading to duplicate processing.

How to Avoid It:

Ensure the subscriber client uses the official SDK (which automatically extends message ack deadlines in the background), and design downstream consumer logic to be idempotent.

Gotcha #3

tip

Zero Shard Throttling vs. AWS ProvisionedThroughputExceeded

The Trap:

Engineers migrating to GCP might spend engineering time writing complex client-side hashing algorithms to balance shards—effort that is completely unnecessary in Pub/Sub.

How to Avoid It:

Rely on Pub/Sub's serverless scaling fabric; focus data engineering efforts on schema validation and processing logic.

6. Test Your Mental Model

Quick Knowledge Check: Test Your GCP Mental Model

Solidify your cross-cloud understanding with instant feedback.

1An engineer needs to guarantee FIFO message ordering per customer in Google Cloud Pub/Sub, similar to Kinesis partition keys. What must be done?

2Which Pub/Sub feature provides direct, zero-code micro-batch streaming into BigQuery, replacing AWS Kinesis Data Firehose?

3What state storage does the Google Cloud Pub/Sub Python SDK require to track consumer progress across parallel workers?

Back to AWS to GCP Data Engineering Bridge Open Full Google Cloud Pub/Sub Guide