Databricks Notebooks: Collaboration, Execution Model, and Best Practices

Databricks notebooks are an interactive development environment that supports four languages: Python, SQL, Scala, and R. You can switch languages within a single notebook using magic commands, and real-time collaborative editing is fully supported.

This article systematically covers every magic command, the collaboration and permission model, execution contexts (interactive vs. job), parameterization with widgets, notebook chaining, and Git integration via Repos.

Multi-language Support and Default Language

When you create a notebook, you pick one default language (Python / SQL / Scala / R). Each cell runs in the default language, but you can switch languages per cell by putting a magic command (%python, %sql, etc.) at the top of the cell.

%python
# Python cell: create a DataFrame and register it as a TempView
df = spark.range(100).toDF("id")
df.createOrReplaceTempView("numbers")

%sql
-- SQL cell: reference the TempView created in Python
SELECT id, id * 10 AS multiplied FROM numbers WHERE id < 10;

%scala
// Scala cell: shares the same SparkSession
val df = spark.sql("SELECT * FROM numbers LIMIT 5")
df.show()

The SparkSession is shared across languages, so a TempView created in Python can be referenced from SQL or Scala. However, Python and Scala variables cannot be shared directly — you exchange data through TempViews or Delta tables.

Magic Commands Reference

%sh runs on the driver node's OS, not on worker nodes. Its output also cannot be passed directly to Spark processing, so use it for file inspection and debugging.

Collaboration and Access Permissions

Databricks notebooks support real-time collaborative editing. Multiple users can edit cells simultaneously, and changes are reflected to others instantly. Comments and @mentions enable in-line review discussions.

Notebooks and folders have four tiers of access permissions:

• Grant only CAN RUN on job notebooks to prevent accidental edits
• Retrieve secrets via dbutils.secrets.get(); never hard-code passwords or tokens in notebooks
• For large changes, create a branch in Repos and merge after PR review

Execution Context: Interactive vs. Job

Notebooks have two execution modes — choose the right one for the job at hand.

# Interactive execution
# Notebook -> attached to an All-Purpose Cluster -> run cells interactively
#
# Job execution
# Workflows -> spin up a Job Cluster -> run the notebook top to bottom -> terminate
#
# +-------------+    attach    +--------------------+
# |  Notebook   |------------->| All-Purpose Cluster|  (interactive)
# | (dev/explore)|             | (shared, stateful) |
# +-------------+              +--------------------+
#
# +-------------+    task      +--------------------+
# |  Workflow   |------------->|   Job Cluster      |  (job)
# | (scheduled) |   create     | (dedicated, fresh) |
# +-------------+              +--------------------+

In interactive execution, results from earlier cells (variables, temp views, etc.) carry over to later cells. Job execution starts from a clean environment every time, so designs that do not depend on external state (idempotent writes, explicit parameter passing) are critical.

Parameterization with Widgets (dbutils.widgets)

To reuse the same notebook across dates, environments, or filter conditions, parameterize it with widgets. Widgets integrate with Workflows task parameters (Base parameters), so values can be injected at job run time.

%python
# Create widgets (4 types)
dbutils.widgets.text("proc_date", "2026-01-01", "Processing date")
dbutils.widgets.dropdown("env", "dev", ["dev", "staging", "prod"], "Environment")
dbutils.widgets.combobox("region", "APAC", ["APAC", "EMEA", "NA"], "Region")
dbutils.widgets.multiselect("status", "active", ["active", "inactive", "pending"], "Status")

# Read widget values
proc_date = dbutils.widgets.get("proc_date")
env = dbutils.widgets.get("env")

# Use the values in a query
df = spark.sql(f"""
    SELECT * FROM {env}_catalog.orders.daily
    WHERE order_date = '{proc_date}'
""")
df.display()

# Return a result when the notebook ends (parent job receives it)
dbutils.notebook.exit(f"OK: {proc_date}, rows={df.count()}")

There are four widget types: text (free entry), dropdown (single choice), combobox (choice plus free entry), and multiselect (multiple choice). When Workflows runs the job, values are auto-injected as long as Base parameter keys match widget names.

%run vs. dbutils.notebook.run: Chaining Notebooks

There are two ways to call one notebook from another, and they differ critically in whether they share state.

%python
# Option 1: %run - inline load into the same session (shared state)
# %run ./includes/common_setup
# -> all variables and functions defined in common_setup are available
# -> state can leak, so be careful in production

# Option 2: dbutils.notebook.run - isolated execution (recommended for production)
result = dbutils.notebook.run(
    "./jobs/etl_daily",
    timeout_seconds=3600,
    arguments={"proc_date": "2026-03-27", "env": "prod"}
)
print(f"Child notebook result: {result}")
# -> e.g., "OK: 2026-03-27, rows=15234"

# Run multiple notebooks in parallel (with ThreadPoolExecutor)
from concurrent.futures import ThreadPoolExecutor

notebooks = ["./jobs/etl_orders", "./jobs/etl_customers", "./jobs/etl_products"]
with ThreadPoolExecutor(max_workers=3) as executor:
    futures = [
        executor.submit(dbutils.notebook.run, nb, 3600, {"proc_date": "2026-03-27"})
        for nb in notebooks
    ]
    results = [f.result() for f in futures]

Git Integration (Repos): Manage Notebooks as Code

Databricks Repos integrates with Git providers (GitHub, GitLab, Bitbucket, Azure DevOps) and lets you manage notebooks and Python modules at the branch level.

• Branching: feature -> PR -> main. Code review and approval happen at the PR
• Saving notebooks in Source format (.py) makes diff review easier
• You can import .py files under a Repo, so package shared logic and keep notebooks thin
• Workflows' Git Source feature pins production runs to a specific branch/tag/commit for reproducibility

%python
# Import a Python module from a Repo
# /Repos/[email protected]/my-project/mypkg/transform.py
from mypkg.transform import run_etl, validate_output

# Keep the notebook focused on orchestration
result_df = run_etl(spark, source="s3://landing/orders/", date="2026-03-27")
validation = validate_output(result_df, expected_min_rows=1000)

if not validation["passed"]:
    dbutils.notebook.exit(f"VALIDATION_FAILED: {validation['reason']}")

result_df.write.format("delta").mode("overwrite").saveAsTable("gold.daily_orders")

Ensuring Reproducibility: Environment, Dependencies, and State

• Pin the Databricks Runtime version (LTS recommended) and declare it in the job definition
• Pin dependencies with %pip install package==version
• Design without depending on session state (temp views, caches). Keep the flow pure: input -> transform -> output
• Use Delta Lake's idempotent writes (MERGE / overwrite with replaceWhere) so identical input always yields identical output
• When using randomness, fix the seed (random.seed / np.random.seed)

What the Exam Tests

• %run runs in the same session (shared state); dbutils.notebook.run runs in isolation (return value is a string)
• %sql runs on the attached cluster, not on a SQL Warehouse
• The four permission levels and what each can do (CAN VIEW / CAN RUN / CAN EDIT / CAN MANAGE)
• How widgets connect to Workflows Base parameters
• %sh runs only on the driver node (it does not run on worker nodes)

Check with a Practice Question

Frequently Asked Questions