Databricks Job Clusters Complete Guide: When to Use vs All-Purpose Clusters

Databricks offers two cluster types: All-Purpose Clusters (interactive) and Job Clusters (job-execution only). The rule of thumb is All-Purpose for development and debugging, Job Clusters for production batch jobs. Job Clusters have a lower unit cost and terminate automatically after the job finishes, making them very cost-efficient.

All-Purpose vs Job Cluster

The standard workflow: develop interactively in notebooks on an All-Purpose Cluster, and once the code is stable, switch to a Job Cluster and schedule it on Workflows (the job scheduler).

Cost Differences

Job Clusters have a substantially lower DBU rate than All-Purpose Clusters. With the same instance type and runtime, simply switching to a Job Cluster can cut DBU cost by more than half.

Cost comparison, conceptually:

All-Purpose Cluster:
  i3.xlarge x 4 nodes x 2 hours = 8 node-hours
  DBU rate: high (Interactive Compute)

Job Cluster:
  i3.xlarge x 4 nodes x 2 hours = 8 node-hours
  DBU rate: low (Jobs Compute)

-> the same work costs fewer DBUs on a Job Cluster
-> and auto-termination removes idle charges entirely

Running production jobs on an All-Purpose Cluster is the worst cost pattern there is. Expect exam questions where the correct answer in any cost-optimization scenario is to migrate to Job Clusters.

Configuring Job Clusters in Workflows

Job Clusters are configured inside the workflow (job) definition. You can either assign a dedicated Job Cluster to each task or define a shared cluster that several tasks reuse.

Example workflow definition:
Job: daily_etl
  ├── Task 1: bronze_load (Job Cluster: i3.xlarge x 2 nodes)
  ├── Task 2: silver_transform (Job Cluster: i3.2xlarge x 4 nodes)
  └── Task 3: gold_aggregate (Job Cluster: i3.xlarge x 2 nodes)

Each task's Job Cluster terminates automatically when the task finishes
-> Task 1 completes -> Cluster 1 terminates
-> Task 2 starts    -> Cluster 2 launches
→ ...

A key benefit of Job Clusters is that you can size each task independently — heavy tasks get a large cluster, light tasks get a small one — for fine-grained optimization.

Using an Existing Cluster (Anti-Pattern)

When you run a job you can also choose to reuse an existing All-Purpose Cluster, but it is discouraged for the following reasons.

• DBU rate stays at the higher Interactive Compute pricing
• If the cluster is shared with other users, you hit resource contention
• The cluster does not auto-terminate when the job finishes
• You cannot size the cluster per job

Outside of one-off testing during development, you should always use Job Clusters.

Best Practices

• Always run production jobs on a Job Cluster
• Combine Job Clusters with Instance Pools to cut startup time
• Use Cluster Policies to restrict allowed instance types and node counts
• Right-size the cluster for each task
• Use the latest LTS Runtime version
• Once development is done, migrate from All-Purpose to Job Cluster

Key Exam Points

• The use-case split between All-Purpose and Job Cluster
• Job Clusters have a lower DBU rate
• Job Clusters terminate automatically once the job ends
• In a cost-optimization context, the answer is Job Cluster + Pool + Policy combined
• Why reusing an existing cluster is an anti-pattern

Practice Question

Frequently Asked Questions