Databricks Cluster Policies: Cost Control, Permissions & Standardization

As a Databricks environment grows, letting anyone spin up any cluster they want becomes a real risk — both for cost control and operational quality. A Cluster Policy lets administrators define and restrict the parameters available at cluster creation time. You can govern instance types, node counts, autoscaling ranges, and tags while keeping the flexibility users actually need.

This article walks through how Cluster Policies work, how to write a definition, the three practical angles of cost control, permissions, and standardization, and the points that show up on the Databricks certification exams.

What Is a Cluster Policy?

A Cluster Policy is a JSON-defined template of cluster settings. A workspace admin creates the policy and assigns it to users or groups — "use this policy to create your clusters." Users can only configure clusters within the bounds the policy allows; anything out of range is greyed out in the UI or returns an error from the API.

Here are the main parameters a Cluster Policy can control:

Basic Structure of a Policy Definition

A Policy definition is JSON. For each parameter, you specify the type of constraint (fixed / range / allowlist / regex, etc.). Here is a skeleton of a pattern commonly used in production:

{
  "spark_version": {
    "type": "regex",
    "pattern": "14\\.3\\.x-scala.*",
    "defaultValue": "14.3.x-scala2.12"
  },
  "node_type_id": {
    "type": "allowlist",
    "values": ["i3.xlarge", "i3.2xlarge", "m5.xlarge"],
    "defaultValue": "i3.xlarge"
  },
  "num_workers": {
    "type": "range",
    "minValue": 1,
    "maxValue": 8,
    "defaultValue": 2
  },
  "autoscale.min_workers": {
    "type": "range",
    "minValue": 1,
    "maxValue": 4
  },
  "autoscale.max_workers": {
    "type": "range",
    "minValue": 2,
    "maxValue": 8
  },
  "custom_tags.CostCenter": {
    "type": "fixed",
    "value": "engineering"
  },
  "autotermination_minutes": {
    "type": "range",
    "minValue": 10,
    "maxValue": 120,
    "defaultValue": 30
  }
}

This definition restricts Runtime to the 14.3.x line, limits node types to 3 choices, allows 1-8 workers, auto-termination of 10-120 minutes (default 30), and pins the CostCenter tag to "engineering". Users can configure clusters anywhere inside that range.

Using Policies for Cost Control

The biggest motivation for adopting Cluster Policies is cost control. Once a team grows past about 10 people, accidents like "someone spun up a huge cluster and left it running" or "a GPU instance was used for a quick test and never terminated" become much more common.

Three Policy settings have the biggest impact on cost control:

• Cap the worker node count (num_workers.maxValue)
• Force a minimum auto-termination time (autotermination_minutes.minValue)
• Exclude expensive instance types (drop GPU SKUs from node_type_id.allowlist)

Combining these lets you enforce constraints like "max 8 nodes, auto-terminate after 30 minutes, no GPUs" at the Policy level, structurally eliminating the risk of users leaving giant clusters running.

Permissions and Policy Assignment

Creating a Policy on its own does nothing — you have to assign it to users or groups so they know "which Policy they can use." Assignment goes through Databricks access control (ACLs).

• CAN_USE: can create clusters with that Policy
• CAN_MANAGE: can edit the Policy definition itself

In practice you assign role-based Policies — "a cost-limited Policy for the dev group," "a GPU-enabled Policy for the ML team," "no limits for admins." Anyone with CAN_USE on the Unrestricted Policy can effectively create whatever they want, so you need to be very selective about who gets it.

Standardizing Cluster Configuration

Policies aren't just for cost — they also govern the environment. Encoding rules like "every team uses the same Runtime version," "always use LTS," "enable Photon," or "force specific Spark settings" into a Policy reduces problems like "it doesn't work because of environment drift" or "the bug doesn't reproduce because of a version mismatch."

Mandatory tagging also runs through Policies. Organizations that do cost allocation (chargeback) can require CostCenter or Project tags so they can accurately track "which team spent how much" after the fact. The real win is that Policy prevents untagged clusters from being created in the first place.

Applying to Job Clusters vs. All-Purpose Clusters

Cluster Policies apply to Job Clusters (used for job runs) as well as All-Purpose Clusters (interactive). When you specify a Policy in the job definition, the cluster the job launches follows that Policy's constraints.

In practice Job Clusters often have the larger cost impact (overnight batches running on a lot of nodes, for example), so it's common to define a separate Policy for jobs and split it like "production batches: max 16 nodes," "dev batches: max 4 nodes."

What the Exam Actually Tests

Databricks exams almost never ask you to write Policy JSON. Instead they test "which feature solves what problem" — questions like the following:

• "Restrict the size of clusters users can create" → Cluster Policy
• "Reduce cluster startup time" → Instance Pool (not a Policy)
• "Require tags for cost allocation" → Cluster Policy + custom_tags
• "Allow only specific Runtime versions" → Cluster Policy + spark_version
• "Prevent non-admins from creating GPU clusters" → Cluster Policy + node_type_id allowlist

Exams love to put Instance Pool, Cluster Policy, and Permissions in the same answer set to trip you up — make sure you can clearly distinguish each one's purpose.

Check with a Sample Question

Frequently Asked Questions