Databricks Init Scripts: Cluster Initialization and Dependency Distribution

There are many scenarios where you want to run custom initialization logic when a Databricks cluster starts up. Installing OS libraries, deploying security agents, setting environment variables, launching monitoring tools — init scripts are an essential mechanism for customizing the cluster environment. This article organizes everything from init script types, storage locations, execution order, and constraints to troubleshooting.

Types of Init Scripts

Init scripts are divided into the following three types based on their scope and configuration method.

Use Cluster-scoped Init Scripts when you want to apply a custom environment to a specific cluster, and use Global Init Scripts for logic you want to run uniformly across all clusters, such as security agents.

Storage Location Options

In Unity Catalog environments, placing scripts in Volumes is the best practice. When you place scripts in Volumes, access is managed by the Unity Catalog permission model, and you get audit logs showing who changed the script and when.

Execution Order

When multiple init scripts exist at cluster startup, they execute in the following order.

1. Global Init Scripts (in the order registered in Admin Settings)
2. Cluster-scoped Init Scripts (in the order specified in the cluster settings)

Because Global Init Scripts run first, a common design pattern is to handle baseline configuration shared across all clusters (security, monitoring) in Global scripts, and to handle cluster-specific customization (such as installing particular libraries) in Cluster-scoped scripts.

Within each script, shell commands run from top to bottom. If a non-zero exit code such as exit 1 is returned partway through, the cluster startup fails.

Init Script Example

#!/bin/bash
# Set environment variables
echo "export ENVIRONMENT=production" >> /etc/environment

# Install OS-level packages
apt-get update -y && apt-get install -y libgeos-dev

# Install additional Python libraries
/databricks/python/bin/pip install geopy==2.4.1

# Launch monitoring agent
if [ -f /opt/monitoring/agent.sh ]; then
  /opt/monitoring/agent.sh start
fi

It is important to write scripts to be idempotent (producing the same result no matter how many times they run). Init scripts re-execute every time a cluster restarts or resizes, so make sure that reinstalling already-installed packages causes no problems.

Constraints on Shared Mode / Unity Catalog Shared Clusters

On Shared Access Mode clusters (clusters shared by multiple users), there are important constraints on init scripts.

• Cluster-scoped Init Scripts cannot be used (for security reasons)
• Only Global Init Scripts can run
• Some filesystem operations within scripts are restricted
• Because users cannot change cluster settings, administrators must prepare the required environment via Global Init Scripts

On Single User Access Mode clusters, Cluster-scoped Init Scripts can be used. This distinction is a frequent exam topic.

How to Check Logs

The results of init script execution are saved to logs on the cluster's driver node.

Relationship with Serverless Compute

Init scripts cannot run on Serverless Compute (SQL Warehouse Serverless / Serverless Notebooks). Because the Serverless environment is fully managed by Databricks, OS-level customization by users is not permitted.

To add libraries in a Serverless environment, use the following approaches.

• Use %pip install within a notebook
• Use libraries managed via Workspace Libraries
• Place a wheel file in Unity Catalog Volumes and reference it with %pip install /Volumes/...

Troubleshooting

Cluster will not start

• Check for INIT_SCRIPT_FAILURE in the Event Log
• Check stderr in the driver logs to identify the cause (apt-get failure, insufficient permissions, network error, etc.)
• Add set -ex at the top of the script to enable debug output

Script stalls partway through

• An interactive prompt from apt-get or similar may be waiting for input → confirm that the -y flag is set
• Network timeout → add --timeout or --retry options

Restarts take a long time every time

• Downloading large packages → stage packages in Volumes ahead of time and switch to local installation
• Unnecessary execution → guarantee idempotency with a conditional (if [ ! -f /opt/already_installed ]; then ...)

Key Exam Points

• "To install a custom library on all nodes of a Shared Access Mode cluster" → Global Init Script
• "To place an init script under Unity Catalog management" → Place it in Volumes
• "How does an init script failure affect the cluster?" → The cluster startup fails
• "To add OS packages on Serverless Compute" → Not possible. Only Python packages via %pip install

Check Your Understanding

Frequently Asked Questions