Dataproc
Fast, cost-effective managed Apache Spark and Apache Hadoop cluster service on Google Cloud.
Fast, cost-effective managed Apache Spark and Apache Hadoop cluster service on Google Cloud.
What is Dataproc?
Google Cloud Dataproc is a fully managed service that allows you to spin up Apache Spark, Apache Hadoop, Presto, and Hive clusters in under 90 seconds. Dataproc integrates deeply with Cloud Storage via the Cloud Storage connector (replacing expensive HDFS), allowing you to treat clusters as completely disposable, ephemeral compute engines that terminate immediately once their Spark job completes.
Decision Guide: When to Choose This Service
Use Dataproc for migrating existing Hadoop/Spark/Hive workloads to the cloud, running large-scale distributed PySpark analytics, ML training on Spark MLlib, and running ephemeral Spark jobs.
Do NOT use Dataproc for new greenfield streaming pipelines if you want fully serverless maintenance without tuning Spark executors (use Dataflow instead).
Core Architectural Pillars
Create a cluster, execute a Spark job, and delete the cluster automatically to pay only for the exact duration of the job.
Replaces HDFS with `gs://` buckets, eliminating cluster data loss upon shutdown.
Add cheap Spot/Preemptible VMs to worker pools to cut compute costs by up to 60-80%.
Submit Spark batches directly without provisioning master or worker nodes.
Master Apache Spark on DataPlayArena
While Dataproc manages your cloud cluster infrastructure, master Apache Spark core algorithms, DataFrame APIs, Spark SQL, and execution internals on our sister platform DataPlayArena.