Databricks Identity Federation Deep Dive: SSO, SCIM, and Organizational Integration

When running Databricks across an entire organization, the "create users directly in Databricks" model breaks down. The standard design is to manage users and groups centrally in your IdP (Entra ID, Okta, Google Workspace, etc.) and federate those identities into Databricks.

This article walks through SSO authentication via SAML 2.0/OIDC, automated identity provisioning via SCIM, the relationship between Account Console and Workspace, and identity management best practices in the Unity Catalog era.

How SSO Works: SAML 2.0 and OIDC

SSO (Single Sign-On) lets a user authenticate once at the IdP and have that session carry over into Databricks. Databricks supports two SSO protocols.

SAML 2.0 has a long track record but is XML-based and somewhat complex to configure. OIDC is a lightweight, modern, JWT-based protocol, and Databricks recommends OIDC for new setups. You do not need to forcibly migrate existing SAML configurations.

SSO Authentication Flow

The SAML 2.0 SSO authentication flow consists of the following steps.

1. The user accesses the Databricks workspace URL
2. Databricks generates a SAML AuthnRequest and redirects to the IdP
3. The user authenticates at the IdP (password + MFA, etc.)
4. The IdP generates a SAML Response and redirects back to Databricks
5. Databricks validates the assertion and creates a user session

With OIDC (Authorization Code Flow), an ID Token (JWT) is used in place of the SAML assertion. From the user's perspective, the experience is identical: authenticate at the IdP login screen and you are automatically logged in to Databricks.

Automated Provisioning with SCIM

SCIM (System for Cross-domain Identity Management) is a protocol that automatically syncs user and group changes from the IdP to Databricks. While SSO unifies authentication (login), SCIM unifies the identity lifecycle (creation, updates, deactivation, deletion).

If you use SSO alone without SCIM, deactivating a user in the IdP leaves the Databricks user record in place. If their PAT is still valid, they can keep accessing the API — a real risk. For production, configuring both SSO and SCIM is mandatory.

Account Console vs Workspace: The Two-Tier Identity Model

Databricks has two levels — Account and Workspace — and identity management mirrors that two-tier structure.

The recommended Unity Catalog-era configuration is to sync with the IdP via Account-level SCIM and manage users and groups centrally at the account. Each workspace then assigns account-level users and groups to itself. Workspace-level SCIM remains supported as a legacy configuration, but Account-level SCIM is the recommended approach for new deployments.

End-to-End Identity Federation Flow

Here is the typical enterprise Identity Federation setup flow.

1. Register Databricks as an application in the IdP: create a SAML/OIDC application in Entra ID or Okta and configure the Databricks ACS URL / Redirect URI
2. Configure SSO in Databricks: in the Account Console, register the IdP metadata (Entity ID, Login URL, certificate)
3. Configure SCIM provisioning: enable the SCIM connector on the IdP side and set the Databricks SCIM API endpoint and Bearer token
4. Configure group mapping: set up the IdP's groups (data-engineers, ml-team, etc.) to sync into Databricks
5. Assign users to workspaces: from the Account Console, assign users and groups to workspaces
6. Validate: log in via SSO with a test user and confirm that SCIM sync creates the user correctly

Group Management Best Practices

Group management under Identity Federation is critical because it ties directly into Unity Catalog permission management.

• IdP groups = Databricks groups: define groups such as data-engineers, data-scientists, and platform-admins in the IdP and sync them to Databricks via SCIM
• Keep Databricks-local groups to a minimum: local groups that aren't synced by SCIM scatter management across systems — limit them to special cases like workspace admins
• Watch out for nested group limitations: some SCIM providers have restrictions on nested group sync. A flat group structure is recommended
• Grant Unity Catalog permissions to groups, not individuals: assign permissions via GRANT statements to groups, and let membership changes happen entirely on the IdP side

Configuration Notes by IdP

Key Exam Points

• "What happens on the Databricks side when an IdP user is deactivated?" → If SCIM is configured, it auto-updates to active=false
• "SSO is configured, but users are not being auto-created in Databricks" → You need SSO JIT provisioning or SCIM
• "How to centrally manage users across multiple workspaces" → Account-level SCIM
• "Which authentication protocols does SSO support?" → SAML 2.0 or OIDC
• "Benefit of group-based access control" → Membership changes are handled entirely on the IdP side, eliminating per-user permission management

Check with a Sample Question

Frequently Asked Questions