Data & Analytics
Process, analyze, and derive insights from data at any scale with BigQuery, Dataflow, Pub/Sub, and more.
BigQuery
GCP
Serverless, petabyte-scale enterprise data warehouse with built-in ML and BI capabilities.
Architecture & CLI Guide
Explore
Dataflow
GCP
Fully managed, serverless stream and batch data processing pipeline service powered by Apache Beam.
Architecture & CLI Guide
Explore
Pub/Sub
GCP
Globally distributed, asynchronous messaging service providing at-least-once message delivery with ultra-low latency.
Architecture & CLI Guide
Explore
Cloud Composer (Managed Airflow)
GCP
Fully managed Apache Airflow service on Google Cloud for orchestrating complex end-to-end data workflows.
Architecture & CLI Guide
Explore
Dataproc
GCP
Fast, cost-effective managed Apache Spark and Apache Hadoop cluster service on Google Cloud.
Architecture & CLI Guide
Explore
Azure Data Lake Storage Gen2 (ADLS Gen2)
AZURE
Massively scalable, secure data lake built on Azure Blob Storage with Hierarchical Namespace and POSIX ACLs.
Architecture & CLI Guide
Explore
Azure Synapse Analytics
AZURE
Limitless enterprise analytics service bringing together big data analytics, data warehousing, and data integration.
Architecture & CLI Guide
Explore
Azure Data Factory (ADF)
AZURE
Cloud-scale serverless data integration and ETL service with 100+ native connectors, visual Mapping Data Flows, and hybrid Self-Hosted IRs.
Architecture & CLI Guide
Explore
Azure Event Hubs
AZURE
High-throughput real-time data streaming platform and event ingestion service with native Apache Kafka multi-protocol support.
Architecture & CLI Guide
Explore
Azure Databricks
AZURE
Enterprise data intelligence and unified lakehouse platform powered by Apache Spark, Photon, and Unity Catalog.
Architecture & CLI Guide
Explore
Amazon Simple Storage Service (Amazon S3)
AWS
Industry-leading cloud object storage offering 11 Nines durability, S3 Intelligent-Tiering, and seamless data lake integration.
Architecture & CLI Guide
Explore
Amazon Redshift
AWS
Fast, petabyte-scale cloud data warehouse using Massively Parallel Processing (MPP), RA3 managed storage, and Redshift Spectrum.
Architecture & CLI Guide
Explore
Amazon Athena
AWS
Serverless, interactive SQL query service querying Amazon S3 data lakes directly with Presto/Trino ($5.00/TB scanned).
Architecture & CLI Guide
Explore
Amazon Kinesis Data Streams
AWS
Massively scalable real-time streaming data service capturing gigabytes of data per second with sub-100ms latency.
Architecture & CLI Guide
Explore
AWS Glue
AWS
Serverless data integration and ETL service featuring automated Crawlers, Glue Data Catalog, and Spark DynamicFrames (DPUs).
Architecture & CLI Guide
Explore