Databricks Workspace Files: Code Layout and Dependency Management

In Databricks you often need to place Python modules, SQL files, configuration files, requirements.txt, and other assets in the workspace in addition to notebooks. Workspace Files is the feature that lets you put files of any type in the same file tree as your notebooks, enabling relative imports and dependency management.

This article walks through what Workspace Files is, how it compares to Repos, how relative imports work, how to manage packages with requirements.txt, and the dependency management patterns most commonly used in production.

What Are Workspace Files?

Workspace Files is the umbrella term for non-notebook files placed under the /Workspace/ path of a Databricks workspace. Since 2023, Databricks lets you upload, create, and edit Python files (.py), text files (.txt/.csv/.json), YAML, and configuration files directly from the workspace file browser.

Before Workspace Files, managing non-notebook files on Databricks required either uploading them to DBFS or calling other notebooks with the %run magic command. With Workspace Files, you can use the same directory structure and import style as any standard Python project.

Repos vs Workspace Files

The common pattern is to manage core team code in Repos, while keeping personal experimentation scripts and temporary configuration files in Workspace Files.

How Relative Imports Work

In Workspace Files (including Repos), Python files placed in the same directory as a notebook can be imported with standard import statements. This is because the current directory is automatically added to sys.path when the notebook runs.

Example Directory Structure

/Workspace/Users/[email protected]/my-project/
├── main_notebook.py        # メインのノートブック
├── utils.py                # ユーティリティ関数
├── config.yaml             # 設定ファイル
├── requirements.txt        # 依存パッケージ定義
└── transformations/
    ├── __init__.py          # パッケージ化に必要
    ├── clean.py             # データクレンジング関数
    └── aggregate.py         # 集約処理関数

Importing from a Notebook

# main_notebook.py

# 同じディレクトリのモジュールをインポート
from utils import validate_schema, log_metrics

# サブディレクトリのパッケージをインポート
from transformations.clean import remove_duplicates
from transformations.aggregate import daily_summary

# 設定ファイルの読み込み
import yaml
with open("config.yaml", "r") as f:
    config = yaml.safe_load(f)

# 処理の実行
df = spark.read.format("delta").load(config["input_path"])
df_clean = remove_duplicates(df)
df_summary = daily_summary(df_clean)
df_summary.write.format("delta").mode("overwrite").save(config["output_path"])

It is critical that __init__.py exists in the subdirectory. Without it, Python will not recognize the directory as a package, and from transformations.clean import ... will fail.

Package Management with requirements.txt

Placing a requirements.txt inside Workspace Files and installing from it with %pip in the notebook is a simple pattern for dependency management.

# requirements.txt
pandas==2.2.0
requests==2.31.0
pyyaml==6.0.1
great-expectations==0.18.0

# ノートブックの先頭セルで実行
%pip install -r ./requirements.txt

%pip performs a notebook-scoped package install and does not affect the cluster-wide environment. However, running a %pip cell restarts the Python process, so you must run it at the very top of the notebook.

Comparison of Dependency Management Patterns

Managing Workspace Files via the API

Workspace Files can be managed not only via browser upload but also via the REST API and the Databricks CLI. This is convenient for automated deployment in CI/CD pipelines.

# Databricks CLIでファイルをアップロード
databricks workspace import \
  --path /Workspace/Shared/etl-project/utils.py \
  --file ./src/utils.py \
  --format SOURCE \
  --language PYTHON

# REST APIでファイル一覧を取得
curl -X GET \
  "https://<workspace-url>/api/2.0/workspace/list" \
  -H "Authorization: Bearer <token>" \
  -d '{"path": "/Workspace/Shared/etl-project"}'

# REST APIでファイルをエクスポート
curl -X GET \
  "https://<workspace-url>/api/2.0/workspace/export" \
  -H "Authorization: Bearer <token>" \
  -d '{"path": "/Workspace/Shared/etl-project/utils.py", "format": "SOURCE"}'

Best Practices

• Structure directories by project: group notebooks, modules, and configuration files into a single directory so that relative paths work naturally.
• Put shared files under /Workspace/Shared/: place utility modules used across the team in the shared folder rather than a personal folder, and make access control explicit.
• Do not put large data files in Workspace Files: store CSV, Parquet, and other data files in cloud storage (S3/ADLS/GCS) or Unity Catalog Volumes.
• Use Repos for real version control: Workspace Files has no version control, so manage core team code in Repos and keep Workspace Files for auxiliary purposes only.

Key Exam Points

• "How do you import an external Python module from a notebook?" → Place a .py file in Workspace Files / Repos and import it
• "How do you manage dependencies per notebook?" → %pip install -r requirements.txt
• "What is the difference between Workspace Files and DBFS?" → Workspace Files is under /Workspace/ for code; DBFS is under /dbfs/ for data
• "How do you import a module from a subdirectory?" → Add an __init__.py to turn it into a package

Check Your Understanding

Frequently Asked Questions