Deploy managed Spark and Hadoop clusters across clouds.
Overview
Big Data Cluster formations provision fully managed Spark and Hadoop clusters through cloud-native services. Each cluster includes a proxy machine for browser access, a head node, and configurable compute nodes (compute nodes on Dataproc).
Available in On-Demand and Spot modes (compute nodes can use Spot for cost savings).
Supported Services
Provider
Service
AWS
EMR (Elastic MapReduce)
GCP
Dataproc
Azure
HDInsight
Alibaba Cloud
E-MapReduce
Architecture
Browser ──► Proxy Machine ──► Head Node ──► Compute Nodes
(RosettaHub (Cluster (Configurable
container) controller) pool size)
Component
Description
Proxy Machine
RosettaHub container providing browser access to the cluster
Head Node
Cluster controller running the resource manager
Compute Nodes
Configurable pool of compute nodes for data processing
Configuration
Setting
Description
Cloud Key
Credentials for the target cloud account
Cluster Service
EMR, Dataproc, HDInsight, or E-MapReduce
Head Node Instance Type
Compute resources for the head node
Compute Node Instance Type
Compute resources for compute nodes
Compute Node Count
Number of compute nodes
Applications
Pre-installed tools (Spark, Hive, Presto, Jupyter, Zeppelin, etc.)
Use Cases
Use Case
Description
Data engineering
ETL pipelines and data transformation at scale
Analytics
Interactive SQL queries on large datasets
Machine learning
Distributed model training with Spark MLlib
Research computing
Large-scale scientific data processing
Big data clusters with RosettaHub: what it does, how it is governed and how to get started.
Try RosettaHub