Skip to content

Big Data Cluster Formations

Deploy managed Spark and Hadoop clusters across clouds.

Overview

Big Data Cluster formations provision fully managed Spark and Hadoop clusters through cloud-native services. Each cluster includes a proxy machine for browser access, a head node, and configurable compute nodes (compute nodes on Dataproc).

Available in On-Demand and Spot modes (compute nodes can use Spot for cost savings).

Supported Services

Provider Service
AWS EMR (Elastic MapReduce)
GCP Dataproc
Azure HDInsight
Alibaba Cloud E-MapReduce

Architecture

Browser ──► Proxy Machine ──► Head Node ──► Compute Nodes
            (RosettaHub       (Cluster        (Configurable
             container)        controller)     pool size)
Component Description
Proxy Machine RosettaHub container providing browser access to the cluster
Head Node Cluster controller running the resource manager
Compute Nodes Configurable pool of compute nodes for data processing

Configuration

Setting Description
Cloud Key Credentials for the target cloud account
Cluster Service EMR, Dataproc, HDInsight, or E-MapReduce
Head Node Instance Type Compute resources for the head node
Compute Node Instance Type Compute resources for compute nodes
Compute Node Count Number of compute nodes
Applications Pre-installed tools (Spark, Hive, Presto, Jupyter, Zeppelin, etc.)

Use Cases

Use Case Description
Data engineering ETL pipelines and data transformation at scale
Analytics Interactive SQL queries on large datasets
Machine learning Distributed model training with Spark MLlib
Research computing Large-scale scientific data processing