Databricks Jobs Webhooks & Notifications: Failure Alerts and Ops Automation

When a Databricks job fails, how quickly you detect and respond to it determines your operational quality. Situations like "I came in this morning and all the overnight batches had failed" or "we noticed the incident hours late" are usually caused by inadequate notification setups.

This article covers the notification channels available in Databricks Jobs (Email/Slack/Teams/PagerDuty/Webhook), how to configure webhooks with implementation examples, an end-to-end automation flow from failure detection to ticket creation, and how to monitor job runs using System Tables.

Notification Channel Comparison

Databricks Jobs lets you configure notifications for events such as job start, success, failure, timeout, and retry. Here is a comparison of the available notification channels.

In practice, it is common to combine multiple channels. For example, you might configure "successes go to email (for records)" and "failures go to Slack + PagerDuty (for immediate response)".

Types of Notification Events

Job notifications can be configured individually for each of the following events.

How to Configure Job Notifications

Job notification settings can be made via the UI, REST API, or DABs YAML. Here is an example using the REST API.

# ジョブ作成時に通知を設定
curl -X POST \
  "https://<workspace-url>/api/2.1/jobs/create" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nightly_etl_pipeline",
    "tasks": [...],
    "email_notifications": {
      "on_start": [],
      "on_success": ["[email protected]"],
      "on_failure": ["[email protected]", "[email protected]"],
      "no_alert_for_skipped_runs": true
    },
    "webhook_notifications": {
      "on_failure": [
        {
          "id": "<notification-destination-id>"
        }
      ]
    },
    "notification_settings": {
      "no_alert_for_skipped_runs": true,
      "no_alert_for_canceled_runs": false
    }
  }'

Configuring Notification Destinations

Notifications to Webhook, Slack, Microsoft Teams, and PagerDuty work by pre-registering them as Notification Destinations and referencing them from the job by ID.

# Slack Notification Destinationの作成
curl -X POST \
  "https://<workspace-url>/api/2.0/notification-destinations" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "Data Platform Slack Channel",
    "config": {
      "slack": {
        "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXX"
      }
    }
  }'

# PagerDuty Notification Destinationの作成
curl -X POST \
  "https://<workspace-url>/api/2.0/notification-destinations" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "Data Platform PagerDuty",
    "config": {
      "pagerduty": {
        "integration_key": "xxxx-xxxx-xxxx-xxxx"
      }
    }
  }'

Custom Notification Flows with Generic Webhooks

To notify systems other than Slack/Teams/PagerDuty, use a generic webhook. Databricks sends a POST request and the receiver processes the payload to perform any action you define.

# 汎用Webhook Notification Destinationの作成
curl -X POST \
  "https://<workspace-url>/api/2.0/notification-destinations" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "Custom Ticket System Webhook",
    "config": {
      "generic_webhook": {
        "url": "https://automation.example.com/databricks-webhook",
        "username": "databricks",
        "password": "secure-password"
      }
    }
  }'

The webhook payload includes information such as the job ID, job name, run ID, run status, and start/end times. The receiver can parse this information to create tickets, update dashboards, trigger downstream jobs, and more.

Automation Flow: Failure Detection → Ticket Creation

Here is the ideal automation flow when a production ETL job fails.

1. Job failure: Databricks Job terminates with an error
2. Webhook notification: Webhook fires on the on_failure event
3. Processing via relay service: Azure Logic Apps / AWS Lambda / n8n receives the webhook
4. Ticket creation: Automatically creates a ticket by calling the Jira / ServiceNow / GitHub Issue API
5. Slack notification: Simultaneously sends a Slack channel notification including the ticket link
6. PagerDuty: For high-severity jobs, immediately alert the on-call engineer via PagerDuty

With this setup, a ticket is created and the on-call engineer is alerted within seconds of a job failure. Manual incident detection is eliminated entirely, so overnight batch failures can be addressed immediately.

Job Monitoring with System Tables

Databricks System Tables store job run history and audit logs as Delta tables. Combined with webhook notifications, they enable more sophisticated monitoring and analysis.

-- 過去7日間で失敗したジョブの一覧
SELECT
  job_id,
  run_id,
  result_state,
  start_time,
  end_time,
  TIMESTAMPDIFF(MINUTE, start_time, end_time) AS duration_minutes
FROM system.lakeflow.job_run_timeline
WHERE result_state = 'FAILED'
  AND start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
ORDER BY start_time DESC

-- ジョブごとの失敗率（過去30日）
SELECT
  job_id,
  COUNT(*) AS total_runs,
  SUM(CASE WHEN result_state = 'FAILED' THEN 1 ELSE 0 END) AS failed_runs,
  ROUND(
    SUM(CASE WHEN result_state = 'FAILED' THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
    1
  ) AS failure_rate_pct
FROM system.lakeflow.job_run_timeline
WHERE start_time > CURRENT_TIMESTAMP() - INTERVAL 30 DAYS
GROUP BY job_id
HAVING failed_runs > 0
ORDER BY failure_rate_pct DESC

-- 実行時間の異常検知（平均の2倍以上かかったジョブ）
WITH stats AS (
  SELECT
    job_id,
    AVG(TIMESTAMPDIFF(MINUTE, start_time, end_time)) AS avg_duration,
    STDDEV(TIMESTAMPDIFF(MINUTE, start_time, end_time)) AS stddev_duration
  FROM system.lakeflow.job_run_timeline
  WHERE result_state = 'SUCCESS'
    AND start_time > CURRENT_TIMESTAMP() - INTERVAL 30 DAYS
  GROUP BY job_id
)
SELECT
  t.job_id,
  t.run_id,
  TIMESTAMPDIFF(MINUTE, t.start_time, t.end_time) AS duration_minutes,
  s.avg_duration AS avg_minutes,
  s.avg_duration + 2 * s.stddev_duration AS threshold_minutes
FROM system.lakeflow.job_run_timeline t
JOIN stats s ON t.job_id = s.job_id
WHERE t.start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
  AND TIMESTAMPDIFF(MINUTE, t.start_time, t.end_time) > s.avg_duration + 2 * s.stddev_duration
ORDER BY t.start_time DESC

By combining System Tables queries with Databricks SQL Alerts, you can build proactive monitoring such as "notify when the failure rate exceeds a threshold" or "detect jobs whose runtime is abnormally long".

Notification Settings in DABs

When you define jobs with Databricks Asset Bundles (DABs), you can declaratively specify notification settings inside the YAML.

# databricks.yml
resources:
  jobs:
    nightly_etl:
      name: "nightly_etl_pipeline"
      email_notifications:
        on_failure:
          - "[email protected]"
        on_success:
          - "[email protected]"
      webhook_notifications:
        on_failure:
          - id: "${var.slack_notification_destination_id}"
      notification_settings:
        no_alert_for_skipped_runs: true
      tasks:
        - task_key: "ingest"
          notebook_task:
            notebook_path: "./notebooks/ingest.py"

When managed through DABs, notification settings become subject to code review and version control, making it possible to track "who changed the notification settings and when".

Key Points for the Exam

• "How do you immediately notify the team on job failure?" → Webhook notifications (Slack/Teams) or PagerDuty
• "How do you analyze job run history with SQL?" → System Tables (system.lakeflow.job_run_timeline)
• "What is on_duration_warning_threshold_exceeded used for?" → Detecting when a job's runtime exceeds a threshold
• "How do you share notification destinations across multiple jobs?" → Register them as Notification Destinations and reference them by ID

Check Your Understanding

Frequently Asked Questions