Terraform for Databricks: Automating Workspaces with IaC

The Databricks Terraform Provider lets you declaratively manage Databricks resources — workspaces, clusters, jobs, Unity Catalog, networking, and more — using HashiCorp Terraform (HCL). Infrastructure as Code (IaC) eliminates manual operations and gives you reproducibility and auditability for every environment.

This article walks through the basic configuration of the Terraform Provider, a catalog of key resources, HCL examples, state management best practices, and how to split responsibilities between Terraform and DABs.

Databricks Terraform Provider Overview

The Databricks Terraform Provider is an officially maintained HashiCorp Terraform plugin. It is published on the Terraform Registry and is declared inside an HCL file as shown below.

terraform {
  required_providers {
    databricks = {
      source  = "databricks/databricks"
      version = "~> 1.40"
    }
  }
}

provider "databricks" {
  host  = var.databricks_host
  token = var.databricks_token
}

You can authenticate with a Personal Access Token (PAT), a service principal, an Azure Managed Identity, or an AWS IAM role. The standard for production is to use service principals and run from a CI/CD pipeline rather than relying on PATs.

Key Resources at a Glance

A Basic HCL Configuration Example

The example below is a baseline configuration that creates a Unity Catalog catalog and schema and defines a cluster policy.

resource "databricks_catalog" "analytics" {
  name    = "analytics"
  comment = "分析チーム用カタログ"
}

resource "databricks_schema" "bronze" {
  catalog_name = databricks_catalog.analytics.name
  name         = "bronze"
  comment      = "RAWデータ格納用スキーマ"
}

resource "databricks_grants" "bronze_grants" {
  schema = databricks_schema.bronze.id
  grant {
    principal  = "data-engineers"
    privileges = ["USE_SCHEMA", "CREATE_TABLE", "SELECT"]
  }
  grant {
    principal  = "data-analysts"
    privileges = ["USE_SCHEMA", "SELECT"]
  }
}

resource "databricks_cluster_policy" "standard" {
  name = "standard-policy"
  definition = jsonencode({
    "spark_version" : {
      "type" : "regex",
      "pattern" : "14\\.3\\.x-scala.*"
    },
    "num_workers" : {
      "type" : "range",
      "minValue" : 1,
      "maxValue" : 8,
      "defaultValue" : 2
    },
    "autotermination_minutes" : {
      "type" : "range",
      "minValue" : 10,
      "maxValue" : 60,
      "defaultValue" : 20
    },
    "custom_tags.CostCenter" : {
      "type" : "fixed",
      "value" : "analytics-team"
    }
  })
}

resource "databricks_secret_scope" "app_secrets" {
  name = "app-secrets"
}

State Management Best Practices

The Terraform state file (terraform.tfstate) records the current state of every managed resource. For team development, follow these best practices.

# AWSの場合のバックエンド設定
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "databricks/prod/terraform.tfstate"
    region         = "ap-northeast-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

Managing Environments (dev / staging / prod)

There are two main approaches for managing per-environment differences in Terraform.

• Terraform Workspaces: reuse the same HCL files and switch environments with terraform workspace select dev. The state file is separated per workspace
• Directory separation + tfvars: split into directories like environments/dev/ and environments/prod/, and override variables with a tfvars file per environment. This is the recommended approach for large-scale configurations

DABs vs Terraform: Comparison Table

What the Exam Asks About

• "How do I manage workspace configuration reproducibly as code?" → Terraform (Databricks Provider)
• "How do I define cluster policies and Secret Scopes as code?" → Terraform
• "How do I deploy jobs and DLT pipelines across environments?" → DABs (Terraform also works, but DABs is more Databricks-native)
• "When should I use Terraform vs DABs?" → Terraform for the infrastructure base, DABs for application assets

Check Your Understanding

Frequently Asked Questions