Databricks offers three types of compute resources, each built around a different design philosophy. Defaulting to "just run everything on an All-Purpose Cluster" is cost-inefficient, and "trying to write Python on a SQL Warehouse" is a classic beginner pitfall. This article lays out the essential differences between the three compute types and the criteria for choosing the right one for each workload.
| Aspect | Job Cluster | All-Purpose Cluster | SQL Warehouse |
|---|---|---|---|
| Primary use case | Production ETL and scheduled jobs | Interactive development, exploration, ML experiments | SQL analytics, BI integration, dashboards |
| DBU rate (rough) | Low (Jobs Compute pricing) | High (All-Purpose Compute pricing) | Middle (SQL Compute pricing) |
| Lifecycle | Auto-created when the job starts, auto-deleted when it ends | Started manually or via API, stopped manually or via Auto Terminate | Started manually, stopped automatically by Auto Stop |
| User sharing | Not allowed (job-dedicated) | Allowed (multiple users concurrently) | Allowed (handles queries from many users) |
| Supported languages | Python / Scala / R / SQL / Spark Submit | Python / Scala / R / SQL | SQL only |
| Photon | Optional (can be enabled) | Optional (can be enabled) | Enabled by default |
| Unity Catalog support | Supported (Shared / Single User Mode) | Supported (Shared / Single User Mode) | Fully supported (always UC-integrated) |
| Init Script | Global / Cluster-scoped (Single User mode) | Global / Cluster-scoped (Single User mode) | Not supported |
| Scaling | Autoscaling (node count) | Autoscaling (node count) | Auto scale-out by cluster count |
| Serverless variant | Serverless Jobs | Serverless Notebooks | Serverless SQL Warehouse |
Pick the right compute by following the flow below, based on workload characteristics.
A Job Cluster is tightly coupled to job execution. When you define a Databricks Job and specify cluster spec in the new_cluster field, a new cluster is created at the start of every Run and is automatically deleted when the Run finishes.
All-Purpose Cluster is optimized for interactive development. Multiple notebooks can be attached and executed simultaneously, making it well-suited for debugging and data exploration.
SQL Warehouse is SQL-dedicated compute that runs queries from the Databricks SQL UI or from BI tools (Tableau, Power BI, etc.).
| Pattern | Fix | Expected effect |
|---|---|---|
| Developers running production batches on All-Purpose | Migrate to Job Cluster | Lower DBU rate + reduced idle cost |
| Leaving All-Purpose Clusters running idle | Set Auto Terminate to 30-60 minutes | Reduces idle DBU |
| Large clusters with fixed node count | Enable Autoscaling (min=1, max=N) | Auto-reduce node count during low load |
| Running SQL analytics and Python dev on the same cluster | Split: SQL → SQL Warehouse, Python → All-Purpose | Use the best-fit compute for each workload |
| Dev clusters using GPU instances | Restrict instance types to CPU via Cluster Policy | Prevent unnecessary use of expensive instances |
| Job Cluster startup takes too long | Pre-warm instances with Instance Pool | Shorter startup (minutes → tens of seconds) |
Data Engineer Associate
問題 1
A team runs the following three workloads on Databricks: (1) a PySpark ETL pipeline that runs daily at 6:00 AM, (2) interactive ML experiments by 3 data scientists, and (3) executive dashboards (20 SQL reports). Which compute configuration handles all workloads appropriately while minimizing cost?
正解: B
B is correct because it assigns the best-fit compute to each workload. ETL goes on Job Cluster for low DBU rate + auto termination, ML experiments use All-Purpose Cluster for interactive development, and dashboards use SQL Warehouse to leverage fast SQL + caching + concurrent-execution scaling. A is inefficient due to the high DBU rate, C cannot run PySpark on SQL Warehouse, and D cannot do interactive development on a Job Cluster.
Should I avoid using an All-Purpose Cluster for production ETL?
It works technically, but it is discouraged for two reasons. First, the DBU rate is higher than Job Cluster (All-Purpose: ~0.40 DBU/h vs Job: ~0.15 DBU/h, varies by region), so running daily batches on an All-Purpose Cluster more than doubles the cost. Second, All-Purpose Clusters are designed to stay running and incur idle costs, while Job Clusters spin up only when the job runs and terminate immediately after, eliminating waste.
Both SQL Warehouse and All-Purpose Cluster can run queries — how do I choose between them?
Use SQL Warehouse for SQL-centric workloads (dashboards, reports, ad-hoc analytics), and All-Purpose Cluster for interactive Python/Scala/R development and ML experiments. SQL Warehouse enables the Photon query engine by default and delivers overwhelmingly better SQL performance thanks to Result Cache, Disk Cache, and IWM.
Where does Serverless Compute fit among Job Cluster, All-Purpose Cluster, and SQL Warehouse?
Serverless is offered as an option for all three compute types. Serverless SQL Warehouse is for SQL, Serverless Notebooks for interactive Python development, and Serverless Jobs for job execution. All of them eliminate infrastructure management and start quickly, but they impose restrictions on custom libraries and Init Scripts.
Practice with certification-focused question sets
無料で問題を解いてみるNicheeLab Editorial Team
NicheeLab editorial team focused on data engineering and cloud certification learning. Content is structured around practical study needs and official exam domains.
Databricks Certifications: All 7 Exams, Difficulty & Study Plan (2026)
Complete guide to all 7 Databricks certifications — Data Eng...
Databricks Exam Difficulty Ranking: All 7 Certs Compared (2026)
Every Databricks certification ranked by difficulty, with st...
Databricks Study Guide: Fastest Pass Route & Time Estimates (2026)
How to pass Databricks certifications efficiently. Official ...
Databricks Data Engineer Associate: Complete Guide (2026)
Domain-by-domain breakdown of the Databricks Certified Data ...
Databricks Data Engineer Professional: Complete Guide (2026)
Tactics for the Databricks Certified Data Engineer Professio...