Cloud Arena
All Cheat Sheets

Data & Analytics Cheat Sheet

Multi-cloud reference for AWS, Azure, and Google Cloud Data & Analytics services

Amazon S3

AWS
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
aws s3 mb s3://my-bucket-name --region us-east-1Create S3 bucket
aws s3 cp local-file.parquet s3://my-bucket/data/Upload file to S3 prefix
aws s3 sync ./local-dir s3://my-bucket/backup/ --deleteSynchronize directory with delete
aws s3 presign s3://my-bucket/report.pdf --expires-in 3600Generate temporary pre-signed URL
aws s3api put-bucket-encryption --bucket my-bucket --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'Enforce default bucket encryption

Architectural Key Facts

  • •Industry-standard object store providing 99.999999999% (11 Nines) data durability.
  • •S3 Intelligent-Tiering automatically moves objects between frequent, infrequent, and archive access tiers with zero retrieval fees.
  • •BucketOwnerEnforced ownership setting disables legacy ACLs and unifies permissions under IAM and Bucket Policies.
  • •Strong read-after-write consistency for PUTs and DELETEs of objects in all AWS regions.
  • •Supports Multipart Uploads for objects >100 MB; mandatory for objects larger than 5 GB.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max single object size5 TB
Max single PUT upload size5 GB (use multipart above 100MB)
Request rate per prefix3,500 PUT/POST/DELETE, 5,500 GET/HEAD req/s
Default bucket limit per account100 (expandable to 1,000)

Amazon Redshift

AWS
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
aws redshift-data execute-statement --cluster-identifier my-cluster --database dev --sql 'SELECT count(*) FROM sales'Execute SQL via Redshift Data API
aws redshift describe-clusters --cluster-identifier my-clusterView cluster status and endpoint
aws redshift-serverless create-workgroup --workgroup-name dev-wg --namespace-name dev-ns --base-capacity 128Provision Serverless workgroup with RPUs
COPY sales FROM 's3://my-bucket/sales.parquet' IAM_ROLE 'arn:aws:iam::123:role/RedshiftS3' FORMAT AS PARQUET;High-throughput parallel bulk COPY from S3

Architectural Key Facts

  • •MPP (Massively Parallel Processing) columnar data warehouse with separate Leader and Compute nodes.
  • •RA3 node architecture decouples compute from Redshift Managed Storage (RMS) on S3.
  • •Distribution Styles (KEY, EVEN, ALL, AUTO) determine how table rows are sliced across compute nodes to prevent skew.
  • •Compound and Interleaved Sort Keys leverage 1 MB zone maps to skip irrelevant data blocks during scans.
  • •Redshift Spectrum enables running SQL queries directly against exabytes of open formats in Amazon S3 without loading.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max tables per cluster9,900 (includes user tables and views)
Max Redshift Serverless RPUs512 RPUs per workgroup
Concurrent queries (WLM)Up to 50 across queues
Max single row size4 MB

Amazon Athena

AWS
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
aws athena start-query-execution --query-string 'SELECT * FROM raw_db.events LIMIT 100' --result-configuration 'OutputLocation=s3://my-query-results/'Execute serverless SQL query
aws athena get-query-execution --query-execution-id QUERY_IDCheck query execution state
aws athena get-query-results --query-execution-id QUERY_IDRetrieve query result rows
MSCK REPAIR TABLE my_table;Sync new S3 partition directories to Glue Catalog

Architectural Key Facts

  • •Serverless interactive query engine based on open-source Trino/Presto and Apache Spark.
  • •Pay-per-query model: $5.00 per TB of data scanned; 10 MB minimum per query.
  • •Utilizes AWS Glue Data Catalog as its central schema and partition metadata repository.
  • •Partition pruning and converting data to columnar Parquet/ORC reduce data scanned by up to 90%.
  • •Athena Workgroups enforce per-query and per-hour data scan cost guardrails.

Production Limits & Quotas

Item / DimensionLimit / Quota
Query execution timeout30 minutes
Concurrent DML queries (default)20 (can be increased via quota)
Max query string length262,144 bytes (256 KB)

Amazon Kinesis Data Streams

AWS
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
aws kinesis create-stream --stream-name clickstream --shard-count 4Create provisioned stream with 4 shards
aws kinesis put-record --stream-name clickstream --partition-key user_123 --data '{"event":"click"}'Put single streaming record
aws kinesis describe-stream-summary --stream-name clickstreamCheck stream health and shard count
aws kinesis register-stream-consumer --stream-arn STREAM_ARN --consumer-name efo-readerRegister Enhanced Fan-Out consumer

Architectural Key Facts

  • •Real-time event streaming service organized into Shards; supports On-Demand and Provisioned capacity modes.
  • •Provisioned Shard baseline capacity: 1 MB/sec (or 1,000 records/sec) write, 2 MB/sec read.
  • •Partition keys are hashed with MD5 to determine shard routing; uniform keys prevent hot sharding.
  • •Enhanced Fan-Out (EFO) provides dedicated 2 MB/sec HTTP/2 push pipes per consumer with sub-70ms latency.
  • •Default retention is 24 hours, extendable up to 365 days for replayability.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max single record payload size1 MB (including partition key)
Standard consumer read transactions5 read transactions/sec per shard (shared)
Max retention period365 days (8,760 hours)
Enhanced Fan-Out consumers per stream20 consumers (default soft limit)

AWS Glue

AWS
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
aws glue start-job-run --job-name etl-cleanse-jobTrigger Glue Spark ETL job
aws glue start-crawler --name s3-telemetry-crawlerStart crawler to infer schema
aws glue create-partition-index --database-name analytics --table-name orders --partition-index '{"Keys":["year","month"],"IndexName":"date_idx"}'Create partition index for fast queries
aws glue get-job-run --job-name etl-cleanse-job --run-id RUN_IDPoll job execution progress

Architectural Key Facts

  • •Serverless data integration service providing managed Apache Spark, Python Shell, and Glue Data Catalog.
  • •DynamicFrames extend PySpark DataFrames with schema evolution and native `choice` resolution.
  • •Job Bookmarks maintain state across scheduled runs to prevent reprocessing previously ingested S3 files.
  • •Glue Flex execution leverages surplus AWS spare compute for non-critical jobs at 34% cost savings.
  • •Glue Data Quality (DQDL) defines declarative assertions directly inside streaming and batch pipelines.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max job execution timeout2,880 minutes (48 hours)
Default concurrent DPU quota50 DPUs (expandable)
Python Shell worker options0.0625 DPU (1 GB RAM) or 1 DPU (16 GB RAM)

Amazon EMR

AWS
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
aws emr create-cluster --name 'SparkCluster' --release-label emr-7.0.0 --applications Name=Spark Name=Hadoop --ec2-attributes KeyName=my-key --instance-type m5.xlarge --instance-count 3 --use-default-rolesLaunch 3-node Spark EMR cluster
aws emr add-job-flow-steps --cluster-id j-XXXX --steps Type=Spark,Name='SparkStep',ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--class,org.example.Main,s3://my-bkt/app.jar]Submit Spark JAR step
aws emr-serverless start-job-run --application-id APP_ID --execution-role-arn ROLE_ARN --job-driver '{"sparkSubmit":{"entryPoint":"s3://my-bkt/job.py"}}'Submit EMR Serverless Spark job
aws emr terminate-clusters --cluster-ids j-XXXXTerminate running cluster

Architectural Key Facts

  • •Cloud big data platform running open-source distributed frameworks: Apache Spark, Hive, Presto, Trino, and HBase.
  • •Cluster node topology: Master Node (YARN ResourceManager), Core Nodes (HDFS + compute), and Task Nodes (spot compute without HDFS).
  • •EMR File System (EMRFS) directly mounts Amazon S3 as an object-backed HDFS replacement with persistent data survival.
  • •EMR Serverless automatically provisions, scales, and manages compute workers without cluster configuration.
  • •EMR on EKS allows running Spark jobs alongside containerized microservices on shared Kubernetes clusters.

Production Limits & Quotas

Item / DimensionLimit / Quota
Steps per cluster limit256 non-completed steps queued
Max active clusters per regionSubject to EC2 instance vCPU quotas
EMR Serverless max worker memoryUp to 120 GB per executor

Azure Data Lake Storage Gen2 (ADLS Gen2)

Azure
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
az storage fs create -n raw-container --account-name mylakeCreate file system container in ADLS Gen2
az storage fs directory create -n telemetry/2026 -f raw-container --account-name mylakeCreate atomic directory in hierarchical namespace
az storage fs file upload -s local.parquet -p telemetry/2026/data.parquet -f raw-container --account-name mylakeUpload Parquet file to ADLS Gen2 path
az storage fs access set --acl 'user::rwx,group::r-x,other::---' -p telemetry -f raw-container --account-name mylakeSet POSIX access control list on directory

Architectural Key Facts

  • •Combines Azure Blob Storage scalability and cost-efficiency with a high-performance Hierarchical Namespace (HNS).
  • •Directory renames and deletes are true atomic O(1) metadata operations rather than copy-and-delete operations.
  • •POSIX-compliant fine-grained Access Control Lists (ACLs) integrate seamlessly with Microsoft Entra ID identities.
  • •Multi-protocol access allows reading/writing files interchangeably via Blob APIs, Azure Data Lake DFS APIs, and NFS 3.0.
  • •Lifecycle management automates transitions between Hot, Cool, Cold, and Archive storage tiers.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max storage account capacity5 PB per storage account
Max request rate per accountUp to 20,000 requests/sec
Max single block blob size4.75 TB (5,000,000 MB)

Azure Synapse Analytics

Azure
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
az synapse workspace create --name myworkspace -g myrg --sql-admin-login-user sqladmin --sql-admin-login-password 'ComplexP@ss1' --location eastus --storage-account mylake --file-system usersProvision Synapse analytics workspace
az synapse sql pool create --name dwpool -g myrg --workspace-name myworkspace --performance-level DW1000cCreate Dedicated SQL Pool (Data Warehouse)
SELECT * FROM OPENROWSET(BULK 'https://mylake.dfs.core.windows.net/raw/*.parquet', FORMAT='PARQUET') AS rows;Query data lake files serverless without loading
az synapse sql pool pause --name dwpool -g myrg --workspace-name myworkspacePause dedicated compute pool to eliminate idle costs

Architectural Key Facts

  • •Unified analytics platform bringing together enterprise data warehousing, big data Spark analytics, and data integration.
  • •Dedicated SQL Pools use Massively Parallel Processing (MPP) distributed across 60 fixed storage locations.
  • •Serverless SQL Pools provide pay-per-query exploration of files in ADLS Gen2 at $5.00 per TB scanned.
  • •Synapse Link enables real-time, hybrid transactional and analytical processing (HTAP) from Azure Cosmos DB.
  • •Built-in integration with Power BI allows instant report publishing directly within Synapse Studio.

Production Limits & Quotas

Item / DimensionLimit / Quota
Dedicated SQL Pool max size240 TB per dedicated pool
Max concurrent queries (DWU)Up to 128 concurrent queries on DW30000c
Serverless SQL query timeout6 hours per query

Azure Data Factory (ADF)

Azure
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
az datafactory factory create --name prod-factory -g myrg --location eastusCreate Azure Data Factory instance
az datafactory pipeline create --factory-name prod-factory -g myrg --name IngestPipeline --pipeline file://pipeline.jsonDeploy pipeline definition
az datafactory pipeline create-run --factory-name prod-factory -g myrg --name IngestPipelineTrigger pipeline execution run
az datafactory integration-runtime self-hosted create --factory-name prod-factory -g myrg --name onprem-irCreate Self-Hosted Integration Runtime for hybrid access

Architectural Key Facts

  • •Fully managed serverless cloud ETL and ELT data integration service with 100+ built-in connectors.
  • •Integration Runtimes (IR): Azure IR for cloud-to-cloud, Self-Hosted IR for on-premises private networks, and Azure-SSIS IR.
  • •Mapping Data Flows allows visually designing code-free data transformation logic executed on managed Apache Spark.
  • •Supports schedule-based, tumbling window, and real-time storage event-based pipeline triggers.
  • •Native CI/CD integration connects directly to Azure DevOps Git or GitHub repositories.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max activities per pipeline40 activities
Concurrent pipeline runs per factory10,000 runs
Data Integration Units (DIU) per copyUp to 256 DIUs per Copy activity

Azure Event Hubs

Azure
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
az eventhubs namespace create --name telemetry-ns -g myrg --location eastus --sku Standard --capacity 4Create Event Hubs namespace with 4 TUs
az eventhubs eventhub create --name clickstream --namespace-name telemetry-ns -g myrg --partition-count 8 --message-retention 7Create Event Hub with 8 partitions and 7-day retention
az eventhubs eventhub consumer-group create --name streaming-analytics --eventhub-name clickstream --namespace-name telemetry-ns -g myrgCreate dedicated consumer group

Architectural Key Facts

  • •Massively scalable big data streaming platform and event ingestion service capable of processing millions of events per second.
  • •Binary compatibility with the Apache Kafka 1.0+ producer and consumer protocols without managing Kafka brokers.
  • •Throughput Units (TUs) provide predictable capacity: 1 TU delivers 1 MB/sec (or 1,000 events/sec) ingress and 2 MB/sec egress.
  • •Event Hubs Capture automatically batches and lands streaming data directly into ADLS Gen2 in Avro or Parquet format.
  • •Consumer Groups provide independent publish-subscribe views for downstream processing without head-of-line blocking.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max event message size1 MB (Standard SKU), 20 MB (Premium/Dedicated SKU)
Partitions per Event HubUp to 32 partitions in Standard; up to 100 in Dedicated
Max retention period90 days (Standard / Premium)

Azure Databricks

Azure
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
az databricks workspace create --name analytics-ws -g myrg --location eastus --sku premiumProvision Premium Azure Databricks workspace
databricks clusters create --json file://cluster-config.jsonCreate automated Spark cluster via Databricks CLI
databricks jobs run-now --job-id 12345Trigger Databricks Workflows DAG run
OPTIMIZE sales_gold ZORDER BY (customer_id, transaction_date);Co-locate columnar Parquet files via Z-Ordering

Architectural Key Facts

  • •First-party Microsoft-optimized Lakehouse platform built on Apache Spark, Delta Lake, and MLflow.
  • •Unity Catalog delivers centralized governance across catalogs, schemas, tables, and volumes via 3-level namespace (`catalog.schema.table`).
  • •Photon vectorized C++ execution engine runs directly on CPU SIMD hardware registers for up to 3x-8x Spark performance.
  • •Delta Lake ACID transaction log (`_delta_log`) guarantees Serializable isolation and deterministic Time Travel (`VERSION AS OF`).
  • •Serverless SQL Warehouses start in sub-5 seconds with automated instant autoscaling and query caching.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max concurrent interactive clusters1,000 clusters per workspace
Max jobs per workspace1,000 jobs (expandable)
Max concurrent task runs per job100 concurrent runs

BigQuery

Google Cloud
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
bq query --use_legacy_sql=false 'SELECT * FROM `project.dataset.table` LIMIT 10'Run standard SQL query
bq mk --dataset project:datasetCreate a new dataset
bq load --autodetect --source_format=CSV dataset.table gs://bucket/file.csvLoad data from GCS to BigQuery
bq extract --destination_format=CSV dataset.table gs://bucket/file.csvExport table to GCS
bq show --format=prettyjson project:dataset.tableView table schema and metadata

Architectural Key Facts

  • •Serverless, highly scalable, and cost-effective multi-cloud data warehouse designed for business agility.
  • •Separates compute and storage, allowing them to scale independently.
  • •Supports standard SQL and enables machine learning capabilities directly using BigQuery ML.
  • •Data is automatically encrypted at rest and in transit by default.
  • •Partitioning and clustering are crucial for optimizing query performance and reducing costs.

Production Limits & Quotas

Item / DimensionLimit / Quota
Query execution time6 hours
Max partitions per partitioned table4,000
Maximum row size100 MB
Concurrent interactive queries100 per project
Storage limitVirtually unlimited

Dataflow

Google Cloud
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
gcloud dataflow jobs run JOB_NAME --gcs-location gs://... --region REGIONRun a Dataflow template
gcloud dataflow jobs list --region REGION --status activeList active Dataflow jobs
gcloud dataflow jobs cancel JOB_ID --region REGIONCancel a running job
gcloud dataflow jobs drain JOB_ID --region REGIONDrain a streaming job to finish processing buffered data

Architectural Key Facts

  • •Fully managed streaming analytics service that minimizes latency, processing time, and cost.
  • •Based on Apache Beam; allows writing pipelines in Java, Python, or Go.
  • •Supports both batch and stream processing with exactly-once processing consistency.
  • •Dynamic work rebalancing minimizes the impact of straggler tasks.
  • •Streaming Engine moves pipeline execution out of the worker VMs and into the Dataflow service backend.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max workers per job (default)1,000 VMs
Max batch job duration30 days
Max graph nodes per pipeline5,000 nodes

Pub/Sub

Google Cloud
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
gcloud pubsub topics create my-topicCreate a topic
gcloud pubsub subscriptions create my-sub --topic=my-topicCreate a subscription
gcloud pubsub topics publish my-topic --message="Hello World"Publish a message
gcloud pubsub subscriptions pull my-sub --auto-ackPull messages from subscription
gcloud pubsub snapshots create my-snapshot --subscription=my-subCreate a snapshot for replay

Architectural Key Facts

  • •Global, asynchronous messaging service that decouples services that produce events from those that process them.
  • •Offers at-least-once message delivery by default; exactly-once delivery available.
  • •Topics act as channels, Subscriptions are attached to Topics.
  • •Supports Push (to HTTPS endpoints) and Pull delivery mechanisms.
  • •Message retention can be up to 31 days (default 7 days).

Production Limits & Quotas

Item / DimensionLimit / Quota
Max message size10 MB
Max message retention31 days
Topic/Subscription name length255 characters
Max push timeout600 seconds

Cloud Composer

Google Cloud
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
gcloud composer environments create env-name --location regionCreate an environment
gcloud composer environments run env-name --location region trigger_dag -- DAG_IDTrigger a DAG manually
gcloud composer environments list --locations regionList environments
gcloud composer environments update env-name --location region --update-pypi-packages-from-file requirements.txtInstall PyPI packages

Architectural Key Facts

  • •Fully managed workflow orchestration service built on Apache Airflow.
  • •Used to author, schedule, and monitor pipelines spanning across clouds and on-premises.
  • •Pipelines are written as Directed Acyclic Graphs (DAGs) in Python.
  • •Composer 2 has autoscaling and eliminates the need to manage scaling configuration manually.
  • •Integrates seamlessly with BigQuery, Dataflow, Dataproc, and other GCP services.

Production Limits & Quotas

Item / DimensionLimit / Quota
DAG execution timeoutConfigurable, no hard limit
PyPI package installation sizeSubject to GKE/Storage limits
Max environments per regionVaries by quota, typically 50+

Dataproc

Google Cloud
Explore Deep Dive Guide

Key Commands & CLI Operations

CommandDescription
gcloud dataproc clusters create my-cluster --region=region --num-workers=2Create a Dataproc cluster
gcloud dataproc jobs submit spark --cluster my-cluster --region region --class org.example.MyJob --jars file.jarSubmit a Spark job
gcloud dataproc clusters delete my-cluster --region regionDelete a cluster
gcloud dataproc clusters update my-cluster --region region --num-workers=5Scale up/down cluster workers

Architectural Key Facts

  • •Fully managed and highly scalable service for running Apache Spark, Apache Flink, Presto, and 30+ open source tools.
  • •Cluster creation takes about 90 seconds on average.
  • •Ideal for migrating existing on-premises Hadoop/Spark workloads to GCP.
  • •Cost-effective: Can use preemptible/Spot VMs for worker nodes.
  • •Supports Ephemeral clusters (deleted after job finishes) to save costs.

Production Limits & Quotas

Item / DimensionLimit / Quota
Max nodes per clusterThousands (depends on Compute quota)
Custom image sizeVaries, standard Compute Engine limits
Job history retention30 days (default) if not saved to GCS