Databricks CI/CD Pipeline Guide: DABs + GitHub Actions

Running Databricks in production requires automated CI/CD deployment of notebooks, jobs, and pipelines. Since 2024, Databricks has positioned Declarative Automation Bundles as the recommended deployment method, and the Data Engineer Professional exam also covers Automation Bundles configuration, GitHub Actions integration, and environment separation. This article walks through how to build a Databricks CI/CD pipeline with Automation Bundles at the center.

End-to-End CI/CD Overview

A Databricks CI/CD pipeline is typically structured around the following flow:

┌──────────┐    PR/Merge     ┌─────────────┐   databricks   ┌──────────────┐
│ Git Repo │ ─────────────► │ CI/CD Tool  │ ─── CLI ────► │  Databricks  │
│ (GitHub) │                │ (GH Actions)│   bundle       │  Workspace   │
│          │ ◄───────────── │             │   deploy       │              │
│ - src/   │   test results  │ - validate  │               │ - dev        │
│ - tests/ │                │ - pytest    │               │ - staging    │
│ - bundle │                │ - deploy    │               │ - prod       │
└──────────┘                └─────────────┘               └──────────────┘

• Development phase: edit and test notebooks on a branch using Databricks Git Folders.
• CI phase: GitHub Actions runs automated tests (lint, pytest, bundle validate) on PR merge.
• CD phase: once tests pass, databricks bundle deploy ships the changes to each environment.

Declarative Automation Bundles

Automation Bundles lets you define Databricks resources (jobs, Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) pipelines, ML models, etc.) declaratively in a databricks.yml file and deploy them with a single CLI command. You define a target per environment and switch between dev, staging, and prod by overriding variables.

Structure of databricks.yml

bundle:
  name: sales-etl-pipeline

workspace:
  host: https://adb-xxxx.azuredatabricks.net

variables:
  catalog:
    default: dev_catalog
  warehouse_id:
    default: abc123def456

resources:
  jobs:
    daily_etl:
      name: "daily-sales-etl-${bundle.target}"
      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "Asia/Tokyo"
      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ./src/01_ingest.py
          existing_cluster_id: ${var.cluster_id}
        - task_key: transform
          depends_on:
            - task_key: ingest
          notebook_task:
            notebook_path: ./src/02_transform.py
          existing_cluster_id: ${var.cluster_id}
        - task_key: publish
          depends_on:
            - task_key: transform
          notebook_task:
            notebook_path: ./src/03_publish.py
          existing_cluster_id: ${var.cluster_id}

  pipelines:
    dlt_pipeline:
      name: "sales-dlt-${bundle.target}"
      target: "${var.catalog}.sales"
      libraries:
        - notebook:
            path: ./src/dlt_definitions.py

targets:
  dev:
    mode: development
    default: true
    variables:
      catalog: dev_catalog
      cluster_id: "0123-dev-cluster"

  staging:
    mode: development
    variables:
      catalog: staging_catalog
      cluster_id: "0456-staging-cluster"

  prod:
    mode: production
    variables:
      catalog: prod_catalog
      cluster_id: "0789-prod-cluster"
    run_as:
      service_principal_name: "sp-production-deployer"

Key Automation Bundles commands

GitHub Actions Integration

Here is an example GitHub Actions workflow that drives a Automation Bundles-based CI/CD pipeline:

# .github/workflows/databricks-cicd.yml
name: Databricks CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
  DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}

jobs:
  validate-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          pip install databricks-cli pytest databricks-connect==15.4.*
          pip install -r requirements.txt

      - name: Validate bundle
        run: databricks bundle validate -t staging

      - name: Run unit tests
        run: pytest tests/unit/ -v --junitxml=test-results.xml

      - name: Run integration tests (staging)
        if: github.event_name == 'pull_request'
        run: |
          databricks bundle deploy -t staging
          databricks bundle run -t staging integration_test_job
          databricks bundle destroy -t staging

  deploy-production:
    needs: validate-and-test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Install Databricks CLI
        run: pip install databricks-cli

      - name: Deploy to production
        run: databricks bundle deploy -t prod

Test Automation

Unit tests (pytest)

# tests/unit/test_transform.py
from pyspark.sql import SparkSession
from src.transforms import clean_sales_data

def test_clean_sales_data(spark: SparkSession):
    """Verify that rows containing null values are excluded"""
    input_df = spark.createDataFrame([
        (1, "ProductA", 100.0),
        (2, None, 200.0),
        (3, "ProductC", None),
    ], ["id", "product_name", "amount"])

    result = clean_sales_data(input_df)

    assert result.count() == 1
    assert result.first()["product_name"] == "ProductA"


# conftest.py
import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")
def spark():
    return SparkSession.builder \
        .master("local[2]") \
        .appName("unit-tests") \
        .getOrCreate()

Notebook tests (dbutils.notebook.run)

# Test runner notebook
test_notebooks = [
    "/Repos/main/project/tests/test_ingest",
    "/Repos/main/project/tests/test_transform",
]

results = []
for nb in test_notebooks:
    try:
        result = dbutils.notebook.run(nb, timeout_seconds=600)
        results.append({"notebook": nb, "status": "PASS", "result": result})
    except Exception as e:
        results.append({"notebook": nb, "status": "FAIL", "error": str(e)})

failed = [r for r in results if r["status"] == "FAIL"]
if failed:
    raise Exception(f"{len(failed)} test(s) failed: {failed}")

Environment Separation (dev / staging / prod)

Setting mode: production in Automation Bundles drops the resource-name prefix and runs jobs as the service principal specified in run_as. With mode: development, resources get a [dev your-username] prefix and schedules are disabled.

Databricks Git Folders (Git Integration)

Databricks Git Folders lets you clone Git repositories, switch branches, pull, and commit — all from within the workspace. Developers edit notebooks in Repos, push changes to Git, and open pull requests from there.

• Supported Git providers: GitHub, Azure DevOps, GitLab, Bitbucket
• Supported file types: native support for .py, .sql, .r, .scala, and .ipynb
• Limits: 10 GB repository size limit, with a cap on the number of file changes per commit

Terraform vs Automation Bundles Comparison

Sample Question

Frequently Asked Questions