Unity Catalog Lineage: Data Lineage and Impact Analysis at Table and Column Granularity

As data pipelines grow more complex, you need a system that can immediately answer questions like "if I change this table, what downstream assets are affected?" or "which source columns produced this number in the gold table?" Unity Catalog Lineage automatically records table-to-table and column-to-column dependencies for Spark jobs running on Databricks, and lets you explore them through both the UI and APIs.

This article walks through how lineage works, how to visualize it in the UI, the Information Schema views, programmatic access via the REST API, the details of column-level lineage, the practical impact-analysis workflow, and the points the Databricks certification exams focus on.

What Is Data Lineage?

Data lineage is the recorded trail of where data came from, how it was transformed, and where it was stored. Unity Catalog Lineage automatically captures the relationship between tables a Spark job reads from (upstream) and writes to (downstream). You do not need to add special annotations to your ETL code.

• Table-level: the dependency that table A was read and then written into table B
• Column-level: the transformation path showing that the price and quantity columns in table A produced the revenue column in table B
• Notebook/job: which notebook or job actually ran the transformation

Having these three layers accumulate automatically streamlines both data governance and incident investigation.

Lineage Visualization in the Data Explorer UI

The most intuitive way to inspect lineage is the Data Explorer (Catalog Explorer) UI. On a table's detail page, opening the Lineage tab renders the upstream tables (sources read from) and downstream tables (sinks written to) as a directed graph.

• Clicking a node navigates to that table's detail page
• Expanding column-level lineage highlights the relationships between individual columns
• Notebook/job links jump straight to the code that ran the transformation

The UI is also well-suited for sharing with non-technical stakeholders — it is a great way to explain to business users where the numbers in a given report come from.

Information Schema LINEAGE Views

When you want to retrieve lineage information programmatically, the dedicated views in the Information Schema are the most convenient. Unity Catalog's Information Schema lives under system.information_schema.

-- テーブルレベルのリネージ: どのテーブルがどのテーブルに依存しているか
SELECT
  source_table_full_name,
  target_table_full_name,
  event_type,         -- READ / WRITE
  created_by,         -- 実行ユーザー
  event_time
FROM system.access.table_lineage
WHERE target_table_full_name = 'prod.gold.daily_revenue'
ORDER BY event_time DESC
LIMIT 50;

-- カラムレベルのリネージ: 特定カラムの算出元を逆引き
SELECT
  source_table_full_name,
  source_column_name,
  target_table_full_name,
  target_column_name,
  event_time
FROM system.access.column_lineage
WHERE target_table_full_name = 'prod.gold.daily_revenue'
  AND target_column_name = 'total_amount'
ORDER BY event_time DESC;

system.access.table_lineage exposes table-level dependencies, while system.access.column_lineage exposes column-level dependencies. Access to these views requires metastore admin or account admin privileges.

Retrieving Lineage via the REST API

The REST API is the best fit for integrating with CI/CD pipelines or external governance tools. Use the Unity Catalog API's /lineage-tracking endpoints.

# テーブルの上流リネージを取得
GET /api/2.1/unity-catalog/lineage-tracking/table-lineage
  ?table_name=prod.gold.daily_revenue
  &direction=UPSTREAM

# レスポンス例（簡略化）
{
  "upstreams": [
    {
      "tableInfo": {
        "name": "prod.silver.orders",
        "catalog_name": "prod",
        "schema_name": "silver"
      },
      "notebookInfos": [
        {
          "notebook_id": "12345",
          "workspace_id": "67890"
        }
      ]
    }
  ]
}

# カラムレベルの上流リネージを取得
GET /api/2.1/unity-catalog/lineage-tracking/column-lineage
  ?table_name=prod.gold.daily_revenue
  &column_name=total_amount

On the REST API, the direction parameter toggles between UPSTREAM (upstream) and DOWNSTREAM (downstream). Use DOWNSTREAM for impact analysis and UPSTREAM for root-cause investigation.

Column-Level Lineage in Detail

Column-level lineage tracks at a finer granularity than table-level. Databricks analyzes the logical plan Spark executes (via the Catalyst Optimizer) to derive the transformation path from input columns to output columns.

Because UDF internals cannot be analyzed from Spark's logical plan, column lineage is recorded only at the level of "UDF input columns → UDF output columns." Note that you cannot tell which columns the UDF actually used internally.

The Impact-Analysis Workflow

Impact analysis is the practice of enumerating, ahead of a change, which downstream tables, dashboards, and jobs will be affected when you modify a table or column. The lineage-driven workflow looks like this:

1. Identify the target table or column being changed
2. Fetch downstream (DOWNSTREAM) lineage via the Data Explorer UI or REST API
3. Build a list of affected tables, views, and dashboards
4. Notify the owners of each downstream object
5. After the change, verify that lineage has updated correctly

-- 影響分析: silver.ordersを変更した場合の下流テーブルを洗い出し
SELECT DISTINCT
  target_table_full_name,
  target_column_name
FROM system.access.column_lineage
WHERE source_table_full_name = 'prod.silver.orders'
  AND source_column_name = 'amount'
ORDER BY target_table_full_name;

-- 結果例:
-- prod.gold.daily_revenue     | total_amount
-- prod.gold.customer_summary  | lifetime_value
-- prod.gold.monthly_report    | gross_sales

From this result, you can see that retyping or renaming the silver.orders.amount column requires re-validating three downstream Gold tables.

Lineage Limitations and Caveats

• Writes that bypass Databricks (for example, Airflow → S3 direct) are not recorded in lineage
• Lineage data is retained for one year; older data is deleted automatically
• Queries that come in via JDBC/ODBC are table-level only (column-level lineage is not supported)
• DLT pipeline lineage is shown separately in the DLT UI (as of 2026, integration with the Information Schema is still in progress)
• Streaming-job lineage is recorded per micro-batch

What the Exam Tests

• Tracking granularity: both table-level and column-level are supported
• Three retrieval paths: UI, Information Schema, and REST API
• Direction of analysis: UPSTREAM (root cause) vs. DOWNSTREAM (impact scope)
• Lineage is captured automatically — no extra configuration in ETL code
• Limitation: processing outside Databricks is not included in lineage
• Know the view names: system.access.table_lineage and system.access.column_lineage

Check Your Understanding

Frequently Asked Questions