Databricks Workload Identity Federation Guide

When a Databricks cluster needs to access an S3 bucket or ADLS Gen2 storage in a cloud environment, the traditional approach was to "hold a secret" — for example, an access key or an Instance Profile. Workload Identity Federation (WIF) uses the cloud provider's identity mechanisms to authenticate without holding any secret (keyless).

This article walks through the WIF concept, a comparison of how it is implemented on AWS, Azure, and GCP, the background of Instance Profile deprecation, and how WIF integrates with Unity Catalog.

Why Keyless Authentication Matters

Traditional cloud access methods each carry their own security risks.

WIF structurally eliminates these risks by removing static secrets and authenticating with short-lived, identity-based credentials from the cloud provider.

WIF Implementation Compared Across Three Clouds

WIF Setup Flow on AWS

On AWS, you configure a Trust Policy on an IAM Role so it trusts the Databricks OIDC provider.

1. Create an IAM Role: Create an IAM Role that grants S3 bucket access permissions (GetObject/PutObject/DeleteObject/ListBucket)
2. Add Databricks to the Trust Policy: Configure a condition that trusts the OIDC provider ARN of your Databricks account
3. Create a Storage Credential in Unity Catalog: Create the Storage Credential by specifying the IAM Role ARN
4. Create an External Location: Create the External Location referencing the Storage Credential and pointing to an S3 path

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-xxxx"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<databricks-account-id>"
        }
      }
    }
  ]
}

WIF Setup Flow on Azure

On Azure, you use an Access Connector together with a Managed Identity.

1. Create a Databricks Access Connector: Create the Access Connector resource via the Azure Portal or Terraform (a System-Assigned Managed Identity is provisioned automatically)
2. Assign RBAC on the storage account: Assign the "Storage Blob Data Contributor" role to the Access Connector's Managed Identity
3. Create a Storage Credential in Unity Catalog: Create the Storage Credential by specifying the Access Connector ID
4. Create an External Location: Specify the ADLS path using the abfss:// scheme

# Terraform例: Access Connector
resource "azurerm_databricks_access_connector" "unity" {
  name                = "unity-catalog-connector"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location

  identity {
    type = "SystemAssigned"
  }
}

# ストレージアカウントへのRBAC割り当て
resource "azurerm_role_assignment" "storage_contributor" {
  scope                = azurerm_storage_account.datalake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.unity.identity[0].principal_id
}

WIF Setup Flow on GCP

On GCP, you configure a Workload Identity Pool and Provider to allow token exchange from Databricks.

1. Create a service account: Create a GCP service account with access permissions on the target GCS bucket
2. Create a Workload Identity Pool: Configure a Pool that trusts the Databricks OIDC provider
3. Allow service account impersonation: Allow Pool members to obtain tokens for the service account
4. Create a Storage Credential in Unity Catalog: Specify the service account's email

Background on Instance Profile Deprecation

Instance Profiles, long used on AWS, are being deprecated for the following reasons.

• Coarse granularity: Instance Profiles apply the same IAM Role to the entire cluster, so fine-grained controls such as "can access table A but not table B" are not possible.
• Not integrated with Unity Catalog: Instance Profiles are a cluster-level setting and operate independently of Unity Catalog's data governance (GRANT/REVOKE), which leads to duplicated permission management.
• Risk on shared clusters: If you set an Instance Profile on a shared cluster, every user can access cloud resources with that IAM Role's permissions.
• Operational complexity: Configuring and managing Instance Profiles per cluster carries a high operational burden.

With the Unity Catalog Storage Credential + External Location approach, data access permissions are integrated into the Unity Catalog permission model, enabling fine-grained control per user or group.

Integration with Unity Catalog

WIF is closely tied to the following Unity Catalog concepts.

This setup integrates cloud storage access control into the Unity Catalog permission model and eliminates Instance Profile-style "shared cluster-wide permissions for everyone."

Key Points Tested on the Exam

• "How do you access S3 without a secret?" → Workload Identity Federation (IAM Role + Trust Policy)
• "What is the recommended replacement for Instance Profiles?" → Unity Catalog Storage Credential (WIF-based)
• "What is the recommended Azure setup to access ADLS from Databricks?" → Access Connector + Managed Identity
• "How do Storage Credential and External Location relate?" → Storage Credential handles cloud authentication; External Location binds it to a path

Check Your Understanding

Frequently Asked Questions