Databricks OAuth Machine-to-Machine Implementation Guide

For automated access to Databricks (CI/CD pipelines, Airflow jobs, external applications), it is a program rather than a human user that authenticates. Personal Access Tokens (PAT) were the traditional choice, but since 2024 Databricks has positioned OAuth Machine-to-Machine (M2M) as the recommended method.

This article walks through how OAuth M2M works (Client Credentials Flow), concrete steps from creating a service principal to registering the OAuth app and fetching tokens, a comparison with PAT, and what the certification exam tests.

Why OAuth M2M Matters

PATs have the following operational issues:

• PATs are tied to individual users, so tokens get orphaned when someone leaves or changes roles
• Lifetime management is manual, so expirations easily cause pipeline outages
• If a PAT leaks, the attacker gains the full set of permissions of the issuing user
• Auditing PAT usage is hard — it is unclear which PAT is used by which pipeline

OAuth M2M structurally solves these problems by combining a service principal (a non-human identity entity) with the OAuth 2.0 Client Credentials Flow.

OAuth 2.0 Client Credentials Flow

OAuth M2M uses the OAuth 2.0 Client Credentials Grant. With no user interaction at all, the client (program) directly obtains a token using its Client ID and Client Secret.

1. The client sends a token request: a POST request containing the Client ID and Client Secret is sent to the Databricks token endpoint.
2. Databricks authenticates the client: it validates the Client ID and Secret and checks the service principal's permissions.
3. An access token is issued: a short-lived Bearer Token (1 hour by default) is returned.
4. The token is used to call the API: attach the token in the Authorization header and call Databricks APIs.

Unlike the Authorization Code Flow, there are no browser redirects or user consent screens. Because it is entirely a machine-to-machine exchange, it is called "Machine-to-Machine".

Setup Steps

There are 3 steps to configure OAuth M2M authentication.

Step 1: Create a Service Principal

Create a service principal from the Account Console or via the databricks service-principals create CLI command.

# Create a service principal via the Databricks CLI
databricks service-principals create \
  --application-id "<application-id>" \
  --display-name "ci-cd-pipeline-sp"

# Or via the Account SCIM API
curl -X POST \
  "https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/scim/v2/ServicePrincipals" \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "displayName": "ci-cd-pipeline-sp",
    "entitlements": [
      { "value": "workspace-access" }
    ]
  }'

Step 2: Register the OAuth Application (Client Secret)

Generate an OAuth Secret for the service principal.

# Generate an OAuth Secret for the service principal
curl -X POST \
  "https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/servicePrincipals/<sp-id>/credentials/secrets" \
  -H "Authorization: Bearer <admin-token>"

# Example response
{
  "id": "secret-id-xxxx",
  "secret": "dose...xxxx",   // store this value securely
  "status": "ACTIVE",
  "create_time": "2026-03-26T10:00:00.000Z"
}

Note: the Client Secret is returned only in the response at creation time. It cannot be retrieved later, so always store it in a secret manager (AWS Secrets Manager, Azure Key Vault, etc.).

Step 3: Acquire a Token

Use the Client ID and Client Secret to obtain an access token.

# Acquire an access token
curl -X POST \
  "https://<workspace-url>/oidc/v1/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials" \
  -d "client_id=<service-principal-client-id>" \
  -d "client_secret=<client-secret>" \
  -d "scope=all-apis"

# Example response
{
  "access_token": "eyJhbGciOiJSUzI1NiI...",
  "token_type": "Bearer",
  "expires_in": 3600
}

Attach the obtained access token in the Authorization header to call Databricks APIs.

# Use the token to list jobs
curl -X GET \
  "https://<workspace-url>/api/2.1/jobs/list" \
  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiI..."

PAT vs OAuth M2M Comparison

Usage Patterns in CI/CD Pipelines

Here is a typical pattern for integrating OAuth M2M into a CI/CD pipeline.

# GitHub Actions example
name: Deploy Databricks Jobs
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Databricks CLI
        run: pip install databricks-cli

      - name: Get OAuth Token
        id: token
        run: |
          TOKEN=$(curl -s -X POST \
            "${{ secrets.DATABRICKS_HOST }}/oidc/v1/token" \
            -d "grant_type=client_credentials" \
            -d "client_id=${{ secrets.DATABRICKS_CLIENT_ID }}" \
            -d "client_secret=${{ secrets.DATABRICKS_CLIENT_SECRET }}" \
            -d "scope=all-apis" | jq -r '.access_token')
          echo "::add-mask::$TOKEN"
          echo "token=$TOKEN" >> $GITHUB_OUTPUT

      - name: Deploy with Databricks CLI
        run: databricks bundle deploy --target production
        env:
          DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
          DATABRICKS_TOKEN: ${{ steps.token.outputs.token }}

The key point is to store the Client ID and Client Secret in GitHub Secrets and never hard-code them in source.

Security Considerations

• Rotate the Client Secret: periodically generate a new Secret and revoke the old one. Define an organizational policy for Secret lifetimes.
• Principle of least privilege: grant the service principal only the minimum permissions the pipeline needs. Prefer read-only (e.g., CAN_VIEW) over CAN_MANAGE_RUN whenever possible.
• Store the Secret safely: use a secret manager such as AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault — never embed the Secret directly in environment variables or source code.
• Review audit logs: service principal actions are recorded in the Databricks audit log. Set up alerts to detect suspicious access patterns.

What the Exam Tests

• "Recommended authentication for CI/CD pipelines" → OAuth M2M (service principal + Client Credentials)
• "Security issues with PAT" → long-lived, user-bound, and broadly scoped
• "OAuth 2.0 flow used by OAuth M2M" → Client Credentials Grant
• "What is a service principal" → a non-human identity entity (for applications and pipelines)

Check Your Understanding

Frequently Asked Questions