Databricks Job Clusters Complete Guide: When to Use vs All-Purpose Clusters

Databricks offers two cluster types: All-Purpose Clusters (interactive) and Job Clusters (job-execution only). The rule of thumb is All-Purpose for development and debugging, Job Clusters for production batch jobs. Job Clusters have a lower unit cost and terminate automatically after the job finishes, making them very cost-efficient.

All-Purpose vs Job Cluster

The standard workflow: develop interactively in notebooks on an All-Purpose Cluster, and once the code is stable, switch to a Job Cluster and schedule it on Workflows (the job scheduler).

Cost Differences

Job Clusters have a substantially lower DBU rate than All-Purpose Clusters. With the same instance type and runtime, simply switching to a Job Cluster can cut DBU cost by more than half.

コスト比較の概念例:

All-Purpose Cluster:
  i3.xlarge × 4ノード × 2時間 = 8ノード時間
  DBU単価: 高い（Interactive Compute）

Job Cluster:
  i3.xlarge × 4ノード × 2時間 = 8ノード時間
  DBU単価: 安い（Jobs Compute）

→ 同じ処理でも Job Cluster の方が DBU コストが低い
→ さらに自動終了でアイドル課金もゼロ

Running production jobs on an All-Purpose Cluster is the worst cost pattern there is. Expect exam questions where the correct answer in any cost-optimization scenario is to migrate to Job Clusters.

Configuring Job Clusters in Workflows

Job Clusters are configured inside the workflow (job) definition. You can either assign a dedicated Job Cluster to each task or define a shared cluster that several tasks reuse.

ワークフロー定義例:
Job: daily_etl
  ├── Task 1: bronze_load（Job Cluster: i3.xlarge × 2ノード）
  ├── Task 2: silver_transform（Job Cluster: i3.2xlarge × 4ノード）
  └── Task 3: gold_aggregate（Job Cluster: i3.xlarge × 2ノード）

各タスクのJob Clusterはタスク完了後に自動終了
→ Task 1 完了 → Cluster 1 終了
→ Task 2 開始 → Cluster 2 起動
→ ...

A key benefit of Job Clusters is that you can size each task independently — heavy tasks get a large cluster, light tasks get a small one — for fine-grained optimization.

Using an Existing Cluster (Anti-Pattern)

When you run a job you can also choose to reuse an existing All-Purpose Cluster, but it is discouraged for the following reasons.

• DBU rate stays at the higher Interactive Compute pricing
• If the cluster is shared with other users, you hit resource contention
• The cluster does not auto-terminate when the job finishes
• You cannot size the cluster per job

Outside of one-off testing during development, you should always use Job Clusters.

Best Practices

• Always run production jobs on a Job Cluster
• Combine Job Clusters with Instance Pools to cut startup time
• Use Cluster Policies to restrict allowed instance types and node counts
• Right-size the cluster for each task
• Use the latest LTS Runtime version
• Once development is done, migrate from All-Purpose to Job Cluster

Key Exam Points

• The use-case split between All-Purpose and Job Cluster
• Job Clusters have a lower DBU rate
• Job Clusters terminate automatically once the job ends
• In a cost-optimization context, the answer is Job Cluster + Pool + Policy combined
• Why reusing an existing cluster is an anti-pattern

Practice Question

Frequently Asked Questions