What Is Spark Connect? Remote Spark Architecture and Exam Prep

Spark Connect is the client/server decoupled architecture introduced in Apache Spark 3.4. In classic Spark applications, the client code and the Spark driver ran in the same JVM. With Spark Connect, you can attach to a remote SparkSession over the gRPC protocol. Databricks certification exams test the mechanics, benefits, and limitations of this architecture.

Spark Connect Architecture

Spark Connect builds a DataFrame API logical plan on the client side and ships it to the server over gRPC. The server then optimizes the plan, converts it into a physical plan, runs it on the Spark cluster, and streams the results back over gRPC.

┌─────────────────────────┐     gRPC      ┌──────────────────────────────┐
│   クライアント (IDE等)    │ ──────────► │   Spark Connect サーバー       │
│                         │              │                              │
│  PySpark / Scala Client │ ◄────────── │  SparkSession                │
│  DataFrame API          │   結果返却   │    ├─ Catalyst Optimizer     │
│  Spark SQL API          │              │    ├─ Physical Planner       │
│                         │              │    └─ Spark Executors        │
└─────────────────────────┘              └──────────────────────────────┘
        ローカルPC                              リモートクラスタ

In the classic architecture, the client JVM acted as the driver and talked directly to executors over RPC. With Spark Connect, the client JVM is fully decoupled from the driver, so a crash on the client side no longer affects the Spark job.

Benefits of Spark Connect

• Improved stability: OOMs or crashes on the client no longer propagate to the driver, which improves stability for long-running jobs.
• Upgrade flexibility: Client and server Spark versions can be upgraded independently, with backward compatibility guaranteed at the gRPC protocol level.
• Remote development experience: Attach a local IDE to a remote Spark cluster with full code completion, debugging, and step execution.
• Multi-language support: Because the protocol is gRPC-based, clients can be implemented in languages outside the officially supported list.

Spark Connect vs Databricks Connect

Setup

Spark Connect (OSS)

# Spark Connectサーバーの起動
./sbin/start-connect-server.sh \
  --packages org.apache.spark:spark-connect_2.12:3.5.0

# Pythonクライアントからの接続
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .remote("sc://spark-server-host:15002") \
    .getOrCreate()

df = spark.sql("SELECT 1 AS test")
df.show()

Databricks Connect v2

# インストール
pip install databricks-connect==15.4.*

# Pythonクライアントからの接続
from databricks.connect import DatabricksSession

spark = DatabricksSession.builder \
    .host("https://adb-xxxx.azuredatabricks.net") \
    .token("dapi...") \
    .clusterId("0123-456789-abcdefgh") \
    .getOrCreate()

# Unity Catalogのテーブルにアクセス
df = spark.table("my_catalog.my_schema.customers")
df.filter(df.region == "APAC").show()

# DataFrameをローカルのPandasに変換
pdf = df.toPandas()

Limitations (Frequently Tested)

Because Spark Connect only ships gRPC logical plans between client and server, the following operations are not supported.

How This Topic Shows Up on the Exam

• Architecture understanding: "Which protocol do the Spark Connect client and server use to communicate?" → gRPC
• Identifying limitations: "Which of the following operations cannot run on Spark Connect?" → Direct RDD operations
• Differences from Databricks Connect: "What is the internal protocol of Databricks Connect v2?" → Spark Connect (gRPC)
• Selecting the right benefit: "What is the benefit of the Spark Connect client decoupling?" → A client crash no longer affects the running job

Sample Question

Frequently Asked Questions