Databricks Jobs API: Job Automation, CI/CD, and Monitoring

Databricks jobs can be managed by hand in the UI, but understanding the Jobs API is essential for production operations and CI/CD pipelines. This article walks through Jobs API 2.1's endpoint layout, request examples, job-definition JSON structure, authentication methods, and execution monitoring with the Runs API in a single, structured guide.

Jobs API 2.1 Endpoint Reference

The main endpoints used in Jobs API 2.1 are listed below. The base URL for every endpoint is https://<workspace-url>/api/2.1/jobs/.

reset replaces the entire job definition (unspecified fields revert to defaults), whereasupdate only changes the fields you specify. CI/CD workflows manage the full definition in code, so reset is used heavily there.

curl Request Examples

Below are curl examples that create and then run a job using a PAT.

# Create job
curl -X POST \
  https://adb-1234567890.12.azuredatabricks.net/api/2.1/jobs/create \
  -H "Authorization: Bearer dapi_xxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "daily_etl_pipeline",
    "tasks": [{
      "task_key": "ingest",
      "notebook_task": {
        "notebook_path": "/Repos/prod/etl/ingest"
      },
      "new_cluster": {
        "spark_version": "14.3.x-scala2.12",
        "node_type_id": "i3.xlarge",
        "num_workers": 4
      }
    }]
  }'

# Run job (run-now)
curl -X POST \
  https://adb-1234567890.12.azuredatabricks.net/api/2.1/jobs/run-now \
  -H "Authorization: Bearer dapi_xxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"job_id": 12345}'

Python Request Example

The same flow using the requests library. In production, add proper error handling and retry logic.

import requests

WORKSPACE_URL = "https://adb-1234567890.12.azuredatabricks.net"
TOKEN = "dapi_xxxxxxxxxxxx"
HEADERS = {"Authorization": f"Bearer {TOKEN}"}

# Create job
job_config = {
    "name": "daily_etl_pipeline",
    "tasks": [{
        "task_key": "ingest",
        "notebook_task": {"notebook_path": "/Repos/prod/etl/ingest"},
        "new_cluster": {
            "spark_version": "14.3.x-scala2.12",
            "node_type_id": "i3.xlarge",
            "num_workers": 4
        }
    }]
}
resp = requests.post(
    f"{WORKSPACE_URL}/api/2.1/jobs/create",
    headers=HEADERS, json=job_config
)
job_id = resp.json()["job_id"]

# Run now
run_resp = requests.post(
    f"{WORKSPACE_URL}/api/2.1/jobs/run-now",
    headers=HEADERS, json={"job_id": job_id}
)
run_id = run_resp.json()["run_id"]

# Poll for run result
import time
while True:
    status = requests.get(
        f"{WORKSPACE_URL}/api/2.1/jobs/runs/get",
        headers=HEADERS, params={"run_id": run_id}
    ).json()
    life_cycle = status["state"]["life_cycle_state"]
    if life_cycle in ("TERMINATED", "INTERNAL_ERROR", "SKIPPED"):
        break
    time.sleep(30)

Job Definition JSON Structure (Multi-Task)

The headline feature of Jobs API 2.1 is multi-task jobs. You can define multiple tasks inside a single job and declare dependencies between them with depends_on.

{
  "name": "daily_pipeline",
  "schedule": {
    "quartz_cron_expression": "0 0 6 * * ?",
    "timezone_id": "Asia/Tokyo"
  },
  "tasks": [
    {
      "task_key": "extract",
      "notebook_task": {"notebook_path": "/Repos/prod/etl/extract"},
      "new_cluster": { ... }
    },
    {
      "task_key": "transform",
      "depends_on": [{"task_key": "extract"}],
      "notebook_task": {"notebook_path": "/Repos/prod/etl/transform"},
      "new_cluster": { ... }
    },
    {
      "task_key": "load",
      "depends_on": [{"task_key": "transform"}],
      "notebook_task": {"notebook_path": "/Repos/prod/etl/load"},
      "new_cluster": { ... }
    }
  ],
  "email_notifications": {
    "on_failure": ["[email protected]"]
  },
  "max_concurrent_runs": 1
}

Setting max_concurrent_runs to 1 prevents the next run from starting until the previous one completes. Setting it to 1 is standard for production jobs to avoid collisions between scheduled triggers and the run-now API.

Passing Parameters

You can pass parameters dynamically when triggering a job with the run-now API. For notebook tasks, values flow through base_parameters (defaults defined in the job) and notebook_params (overrides supplied at run-now time).

# Override parameters at run-now
curl -X POST .../api/2.1/jobs/run-now \
  -d '{
    "job_id": 12345,
    "notebook_params": {
      "target_date": "2026-03-27",
      "mode": "full_refresh"
    }
  }'

Inside the notebook, read parameters with dbutils.widgets.get("target_date"). For Python tasks use python_params, and for SparkSubmit tasks use spark_submit_params.

Monitoring Runs with the Runs API

Use the Runs API to inspect job execution results. The most important fields are listed below.

Comparing API Authentication Methods

What the Exam Tests

• "reset vs. update" → reset overwrites the entire definition; update changes only specific fields
• "run-now vs. submit" → run-now runs an existing job; submit (runs/submit) runs an ephemeral job definition immediately
• "Purpose of max_concurrent_runs" → caps concurrent runs of the same job to prevent resource contention and data inconsistency
• "Best auth for CI/CD pipelines" → OAuth M2M (Service Principal)

Check with a Sample Question

Frequently Asked Questions