Databricks Declarative Pipelines (Delta Live Tables) vs Workflows: Design Decisions

When building data pipelines on Databricks, one of the most common design questions is whether to use Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT), Workflows (Jobs), or a combination of both. DLT is declarative: you describe what you want, and the framework manages data quality and dependencies for you. Workflows is imperative: you describe how things should run, composing arbitrary tasks into a DAG for flexible orchestration.

This article compares the two across four dimensions — execution model, error handling, data quality, and cost — then walks through the combined pattern and a decision flow for picking between them.

DLT vs Workflows: Side-by-Side Comparison

DLT Basics: Declarative Table Definitions

In DLT you simply declare "this table comes from this source with these transformations," and the framework handles dependency resolution, incremental processing, and schema evolution.

import dlt
from pyspark.sql.functions import col, current_timestamp

@dlt.table(comment="Bronzeレイヤー：生データ取り込み")
def bronze_orders():
    return (
        spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/mnt/landing/orders/")
    )

@dlt.table(comment="Silverレイヤー：クレンジング済み")
@dlt.expect_or_drop("valid_amount", "amount > 0")
def silver_orders():
    return (
        dlt.read_stream("bronze_orders")
        .select("order_id", "customer_id", "amount", "region",
                current_timestamp().alias("processed_at"))
        .filter(col("order_id").isNotNull())
    )

@dlt.table(comment="Goldレイヤー：集約")
def gold_revenue_by_region():
    return (
        dlt.read("silver_orders")
        .groupBy("region")
        .agg({"amount": "sum"})
        .withColumnRenamed("sum(amount)", "total_revenue")
    )

Use dlt.read_stream() for streaming reads and dlt.read() for batch reads. Dependencies between tables are resolved automatically from what each read/read_stream references, so you never have to specify execution order.

Workflows Basics: A Multi-Task DAG

Workflows defines task dependencies as a DAG (directed acyclic graph). Each task can be a notebook, Python script, JAR, DLT pipeline, dbt project, and more, and you can assign different compute resources to each one.

# Workflows のDAG構造の概念
#
# [extract_task] ──→ [dlt_pipeline_task] ──→ [notify_task]
#       │                                         ↑
#       └──→ [validate_source_task] ──────────────┘
#
# - extract_task: 外部APIからデータ取得（Pythonノートブック）
# - dlt_pipeline_task: DLTパイプラインでETL（タスクタイプ: Pipeline）
# - validate_source_task: ソースデータの疎通確認
# - notify_task: Slack通知（成功/失敗を条件分岐で制御）

The strength of Workflows is the flexibility to set retries, timeouts, and conditional branches (if/else/for_each) per task. It can also unify non-transformation work like ML model training, external API calls, and notifications under one orchestrator.

The Combined Pattern: Workflows + DLT

The most common production pattern is to embed a DLT pipeline as a single task inside a Workflows DAG. Let DLT handle the ETL transformations and quality controls, and let Workflows orchestrate everything around them.

In this setup the DLT pipeline's success or failure propagates as the Workflows task status, so you can combine it with downstream conditional branches and retry settings for sophisticated error handling.

Decision Flow

Choose between DLT and Workflows based on the nature of your pipeline and your team's needs.

パイプラインの中心はテーブル変換（ETL/ELT）か？
├─ YES → データ品質管理（Expectations）が必要か？
│    ├─ YES → DLTを使う（Workflowsと併用も検討）
│    └─ NO  → シンプルならWorkflows単体、複雑ならDLTを検討
└─ NO  → MLトレーニング・外部API・通知など多様なタスクか？
     └─ YES → Workflowsを使う（ETL部分があればDLTを併用）

• Table transformations + quality controls needed → DLT (standalone or combined with Workflows)
• Orchestrating diverse tasks → Workflows
• Integrating ETL + ML + notifications → Workflows + a DLT pipeline task
• Scheduling a single notebook → Workflows alone is enough

Cost Differences

DLT bills at a DLT-specific DBU rate, which is higher than standard Jobs Compute DBUs. However, DLT also delivers cluster optimization via Enhanced Autoscaling and automates incremental processing, so the right comparison is total cost — including the development and operational cost of building the same features by hand.

What the Exam Tests

• The concrete difference between DLT (declarative) and Workflows (imperative)
• DLT provides data quality controls via built-in Expectations; Workflows has no built-in quality features
• Workflows can run a DLT pipeline as a task (the combined pattern)
• DLT resolves dependencies automatically, while Workflows defines them explicitly via the DAG
• The DBU rate difference (DLT > Jobs Compute)

Check Your Understanding

Frequently Asked Questions