Databricks Secret Scopes: Secure Credential Management

Database passwords, API keys, storage account connection strings — are you still hard-coding these credentials into notebook cells or job configurations?Secret Scope is the Databricks-native secret management feature that separates credentials from code and lets you retrieve them safely with dbutils.secrets.get. Retrieved values are automatically replaced with [REDACTED] in notebook output, which reduces the risk of leaks during screen sharing or notebook export.

This article compares the two Secret Scope backends (Databricks-backed and Key Vault-backed), walks through CLI/API management, shows Python usage patterns, covers ACL design, and highlights the points tested on the Data Engineer Associate and Professional exams.

How Secret Scopes Work

A Secret Scope is a logical container for secrets. Each Scope holds multiple keys (secrets), and ACLs are configured per Scope. The structure looks like this:

Secret Scope: "production-secrets"
  ├── key: "db-password"       → value: "P@ssw0rd123"
  ├── key: "api-key"           → value: "sk-abc123..."
  └── key: "storage-account"   → value: "DefaultEndpoints..."

Secret Scope: "development-secrets"
  ├── key: "db-password"       → value: "dev-password"
  └── key: "api-key"           → value: "sk-dev..."

# Use from a notebook
password = dbutils.secrets.get(scope="production-secrets", key="db-password")
# → The value is bound to the variable, but print(password) shows [REDACTED]

Because each secret is uniquely identified by (Scope name, Key name), a common pattern is to create one Scope per environment (production / development / staging) and reuse the same key names across them. You can switch environments just by changing the Scope name, without touching the code.

Backend Comparison

Creating and Managing Scopes via CLI / API

Here are the Databricks CLI commands for managing Secret Scopes and secrets.

Operating on a Databricks-backed Scope

# Create a Scope
databricks secrets create-scope --scope production-secrets

# Add a secret (value is entered at the prompt)
databricks secrets put-secret --scope production-secrets --key db-password

# Add a secret (pass the value directly as a string)
databricks secrets put-secret --scope production-secrets \
  --key api-key --string-value "sk-abc123..."

# List Scopes
databricks secrets list-scopes

# List keys in a Scope (values are not shown)
databricks secrets list-secrets --scope production-secrets

# Delete a secret
databricks secrets delete-secret --scope production-secrets --key api-key

# Delete a Scope (also deletes all secrets under it)
databricks secrets delete-scope --scope production-secrets

Creating a Key Vault-backed Scope (Azure)

# Create an Azure Key Vault-backed Scope
databricks secrets create-scope --scope kv-production \
  --scope-backend-type AZURE_KEYVAULT \
  --resource-id "/subscriptions/<sub_id>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault_name>" \
  --dns-name "https://<vault_name>.vault.azure.net/"

# For Key Vault-backed Scopes, add/update/delete happens on the Key Vault side.
# From Databricks, only READ via dbutils.secrets.get is allowed.

Using the Secrets REST API

# List Scopes via the REST API
curl -X GET "https://<workspace_url>/api/2.0/secrets/scopes/list" \
  -H "Authorization: Bearer <token>"

# Add a secret via the REST API
curl -X POST "https://<workspace_url>/api/2.0/secrets/put" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "scope": "production-secrets",
    "key": "db-password",
    "string_value": "P@ssw0rd123"
  }'

Python Usage Patterns

Here are common implementation patterns for using secrets inside notebooks and jobs.

# Pattern 1: JDBC database connection
jdbc_url = "jdbc:postgresql://prod-db.example.com:5432/analytics"
username = dbutils.secrets.get(scope="production-secrets", key="db-username")
password = dbutils.secrets.get(scope="production-secrets", key="db-password")

df = (spark.read
  .format("jdbc")
  .option("url", jdbc_url)
  .option("dbtable", "public.orders")
  .option("user", username)
  .option("password", password)
  .load())

# Pattern 2: External API request
import requests

api_key = dbutils.secrets.get(scope="production-secrets", key="api-key")
response = requests.get(
    "https://api.example.com/data",
    headers={"Authorization": f"Bearer {api_key}"}
)

# Pattern 3: Azure Storage connection (via Spark Config)
storage_key = dbutils.secrets.get(scope="production-secrets", key="storage-key")
spark.conf.set(
    "fs.azure.account.key.mystorageaccount.dfs.core.windows.net",
    storage_key
)
df = spark.read.parquet("abfss://[email protected]/data/")

How Secret Redaction Behaves

Databricks automatically replaces values fetched via dbutils.secrets.get with [REDACTED] whenever they would appear in a notebook output cell. This applies not only in the notebook UI but also in job output logs and the result of the display() function.

# Running the code below produces...
password = dbutils.secrets.get(scope="production-secrets", key="db-password")
print(password)
# → Notebook output: [REDACTED]

print(f"Password is: {password}")
# → Notebook output: Password is: [REDACTED]

# But the value still works as a normal variable
# JDBC connections and API calls can use it without issues

Secret Redaction is purely display-side masking — the secret value still exists in memory. It does not fully protect against malicious code that, for example, prints the secret one character at a time. That is why it is critical to combine it with Secret Scope ACLs to control who can access which Scope.

ACLs (Access Control Lists)

ACLs can be configured per Secret Scope (Premium plan or above). There are three permission levels.

# Configure ACLs (Databricks CLI)
databricks secrets put-acl --scope production-secrets \
  --principal data-engineers --permission READ

databricks secrets put-acl --scope production-secrets \
  --principal cicd-sp --permission WRITE

# List ACLs
databricks secrets list-acls --scope production-secrets

# Delete an ACL
databricks secrets delete-acl --scope production-secrets \
  --principal data-engineers

What the Exam Tests

Secret Scopes show up on both the Data Engineer Associate and Professional exams. Make sure you are comfortable with these recurring patterns:

• "What is displayed when you print() a secret in a notebook" → [REDACTED] is shown (Secret Redaction)
• "The API for safely retrieving a secret" → dbutils.secrets.get(scope=..., key=...)
• "The difference between Databricks-backed and Key Vault-backed Scopes" → where they are stored and who owns secret operations differs
• "The Secret Scope ACL permission levels" → MANAGE / READ / WRITE (three levels)
• "What to use instead of hard-coding passwords in code" → Secret Scope
• "The return value of dbutils.secrets.list()" → a list of key names (no values; metadata only)

Practice Question

Frequently Asked Questions