Google Cloud Pub/Sub ➔ Amazon Kinesis Data Streams
From Pub/Sub per-message ACK model to Amazon Kinesis Data Streams (Shards & Offsets).
In GCP Pub/Sub, you don't manage partitions, and messages disappear when ACKed. In Amazon Kinesis Data Streams, you manage Shards (or use Kinesis On-Demand mode), messages are immutable logs retained for up to 365 days, and consumer worker applications maintain their own sequence number checkpoints in DynamoDB.
1. Architectural Mechanism Comparison
Google Cloud Pub/Sub
Per-message acknowledgment (ACK) model with dynamic serverless auto-sharding. Messages are deleted once all subscriptions acknowledge.
- Zero partition sizing or capacity planning (fully elastic).
- Individual message level retry, ACK deadlines, and dead-letter queues.
- Global topic endpoints.
Amazon Kinesis Data Streams
Partitioned append-only commit log organized into Shards (1 MB/s ingress / 2 MB/s egress per shard). Events are stored for up to 365 days and consumers track read sequence numbers.
- Sub-100ms ultra-low latency streaming.
- Kinesis Data Firehose: 1-click zero-code continuous streaming ingestion into S3, Redshift, and OpenSearch.
- Enhanced Fan-Out (EFO): Dedicated 2 MB/s read pipe per consumer application.
2. Interactive Terminology & Concept Bridge
Interactive Concept Bridge: Terminology & Architectural Mapping
Click any concept below to see how your GCP knowledge directly maps into AWS.
Pub/Sub Topic
Named streaming channel to which publishers send messages.
Kinesis Data Stream
Ordered streaming log divided into one or more Shards.
Both represent the ingestion resource for streaming publishers.
3. Visual Architecture Pipeline (Amazon Kinesis Data Streams)
Amazon Kinesis 3-Stage Architecture: Producers ➔ Shards & Logs ➔ EFO & Firehose
Click any section below or run the simulation to see how Kinesis handles partitioned real-time streams.
2. Shard Commit Log & Retention Engine
Each Shard operates as an independent, append-only, ordered commit log. Events are stored immutably for up to 365 days. On-Demand mode automatically splits and merges shards based on live traffic volume.
“Like a row of high-speed tape recorders writing continuously, allowing multiple listeners to rewind and replay independently.”
- Sequence Numbers: Monotonically increasing 128-bit integer IDs assigned to every record.
- On-Demand Mode: Auto-scales shard count from 0 to thousands with zero capacity planning.
- Replay Retention: Up to 365 days of historical stream re-processing.
4. Side-by-Side Code, CLI & Terraform Translator
Side-by-Side Code & Syntax Translator
# GCP Pub/Sub Publisher
from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("my-gcp-project", "iot-events")
data = b'{"sensor_id": "sensor_01", "temp": 73.4}'
future = publisher.publish(topic_path, data, ordering_key="sensor_01")
print(f"Published message ID: {future.result()}")# AWS Kinesis Publisher (Boto3)
import boto3, json
kinesis = boto3.client('kinesis', region_name='us-east-1')
payload = json.dumps({"sensor_id": "sensor_01", "temp": 73.4})
response = kinesis.put_record(
StreamName='iot-events',
Data=payload.encode('utf-8'),
PartitionKey='sensor_01' # Guarantees strict FIFO ordering per sensor!
)
print(f"Published Shard ID: {response['ShardId']}, Seq: {response['SequenceNumber']}")5. Paradigm Shift Gotchas: Traps to Avoid in AWS
Kinesis Shard Throughput Limits (Provisioned Mode)
If a sudden traffic spike pushes 5 MB/s into a stream with only 2 provisioned shards, Kinesis will throw `ProvisionedThroughputExceededException` and drop messages!
Enable **Kinesis On-Demand Mode** to automatically scale throughput capacity, or proactively monitor and split shards.
6. Test Your Mental Model
Quick Knowledge Check: Test Your AWS Mental Model
Solidify your cross-cloud understanding with instant feedback.