Databricks Asset Bundles: Asset Distribution and Per-Environment Management

Databricks Asset Bundles (DABs) provide a way to define Databricks assets — jobs, DLT pipelines, notebooks, ML models, and more — as code, and deploy them consistently across multiple environments (dev / staging / prod). You describe asset configuration in a databricks.yml file, and use the Databricks CLI bundle commands to validate, deploy, run, and destroy them.

This article walks through the core concepts of DABs, the databricks.yml structure, per-environment targets, the main CLI commands, and how to choose between DABs, Terraform, and Repos.

What are Databricks Asset Bundles?

DABs are the tool that brings "Infrastructure as Code (IaC) + Application as Code" to Databricks. Historically, Databricks development meant creating jobs and pipelines manually in the UI, and copying them by hand when moving between environments (dev → staging → prod). DABs let you define those same assets in YAML, manage them in Git, and deploy them automatically through CI/CD.

DABs can manage the following kinds of assets:

• Jobs (Workflows / Lakeflow Jobs)
• DLT pipelines (Delta Live Tables pipeline definitions)
• MLflow experiments and models
• Notebooks and Python scripts
• Dashboards (Databricks SQL Dashboards)
• Schemas and volumes (UC objects)

Anatomy of a databricks.yml

The databricks.yml placed at the project root is built from the following main sections:

bundle:
  name: my-etl-pipeline

variables:
  catalog_name:
    description: "Target catalog"
    default: "dev_catalog"
  warehouse_id:
    description: "SQL Warehouse ID"

workspace:
  root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}

resources:
  jobs:
    daily_etl:
      name: "daily-etl-${bundle.target}"
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"
        timezone_id: "Asia/Tokyo"
      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ./src/ingest.py
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            num_workers: 4
            node_type_id: "i3.xlarge"
        - task_key: transform
          depends_on:
            - task_key: ingest
          notebook_task:
            notebook_path: ./src/transform.py
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            num_workers: 8

  pipelines:
    dlt_pipeline:
      name: "dlt-pipeline-${bundle.target}"
      target: "${var.catalog_name}.bronze"
      libraries:
        - notebook:
            path: ./src/dlt_definitions.py

targets:
  dev:
    mode: development
    default: true
    variables:
      catalog_name: "dev_catalog"
  staging:
    variables:
      catalog_name: "staging_catalog"
  prod:
    mode: production
    variables:
      catalog_name: "prod_catalog"
    run_as:
      service_principal_name: "prod-deployer-sp"

Designing targets (environments)

The targets section is how you deploy the same asset definitions to different environments. Each target can override variables, permissions, and the run-as identity (run_as).

In development mode, resource names are automatically prefixed with [dev username], so multiple developers can deploy bundles to the same workspace without name collisions. In production mode, run_as is required, and using a service principal is the recommended best practice.

Key CLI commands

Environment variables and secrets management

Declare variables inside databricks.yml. Override their defaults per target, or inject values at CLI runtime via the --var flag.

# Inject variables at CLI runtime
databricks bundle deploy -t prod --var="catalog_name=prod_catalog"

# Example in a CI/CD pipeline (GitHub Actions)
- name: Deploy to prod
  run: |
    databricks bundle deploy -t prod \
      --var="catalog_name=${{ secrets.PROD_CATALOG }}" \
      --var="warehouse_id=${{ secrets.PROD_WAREHOUSE_ID }}"

Best practice: do not write secrets (passwords, API keys, etc.) into databricks.yml. Store them in a Databricks Secret Scope and read them inside notebooks via dbutils.secrets.get.

DABs vs Terraform vs Repos comparison

What the exam asks about

DABs show up on the Data Engineer Professional and administration-focused exams.

• "How do I deploy jobs and DLT pipelines consistently across dev/staging/prod?" → DABs (configure targets)
• "How do I swap the catalog name per environment in databricks.yml?" → Override variables in each target
• "How do I pin the job run identity for production deploys?" → Specify a service principal via run_as
• "When do I use Terraform vs DABs?" → Terraform for infrastructure, DABs for application assets

Check your understanding

Frequently Asked Questions