Amazon Kinesis Data Streams
Massively scalable real-time streaming data service capturing gigabytes of data per second with sub-100ms latency.
Massively scalable real-time streaming data service capturing gigabytes of data per second with sub-100ms latency.
Coming from Google Cloud?
Both are high-throughput real-time streaming ingestion platforms, but Kinesis uses partitioned shard commit logs while Pub/Sub uses dynamic serverless topics with per-message ACKs.
- Ingest hundreds of thousands of events per second with sub-second latency.
- Zero-code automated landing into cloud storage (Kinesis Firehose vs. Pub/Sub GCS Subscriptions).
- Ordering keys / partition keys to enforce ordered message processing per entity.
- Architecture: Kinesis is an append-only partitioned log (like Kafka); Pub/Sub is a dynamic message queue where messages are deleted when ACKed.
- Capacity: Kinesis requires managing Shards (or using On-Demand mode); Pub/Sub scales partitions dynamically behind the scenes.
- Replay: Kinesis retains logs for up to 365 days for re-reading; Pub/Sub retains unacknowledged messages up to 7 days.
What is Amazon Kinesis Data Streams?
Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store streaming data at any scale. Kinesis organizes data into append-only Shards, guaranteeing strict FIFO sequence ordering per partition key. Enhanced Fan-Out (EFO) delivers dedicated 2 MB/sec read throughput per consumer application over HTTP/2.
Decision Guide: When to Choose This Service
Use Amazon Kinesis Data Streams for real-time log ingestion, IoT sensor streams, financial market feeds, and continuous event routing to S3, Lambda, and Flink.
Do NOT use Kinesis for standard asynchronous decoupling of microservice work queues where per-message ACKs and dead-letter queues are needed (use Amazon SQS instead).
Core Architectural Pillars
Base throughput unit (1 MB/s Ingress / 2 MB/s Egress / 1,000 records/sec).
Application keys hashed with MD5 to guarantee strict FIFO ordering per entity.
Dedicated 2 MB/sec read bandwidth per consumer application over HTTP/2.
Zero-code streaming micro-batch ingestion into Amazon S3, Redshift, and Snowflake.