Databricks Declarative Pipelines (Delta Live Tables) vs Workflows: Design Decisions

When building data pipelines on Databricks, one of the most common design questions is whether to use Lakeflow Spark Declarative Pipelines (formerly Lakeflow Spark Declarative Pipelines / Lakeflow Spark Declarative Pipelines), Workflows (Jobs), or a combination of both. Lakeflow Spark Declarative Pipelines is declarative: you describe what you want, and the framework manages data quality and dependencies for you. Workflows is imperative: you describe how things should run, composing arbitrary tasks into a DAG for flexible orchestration.

This article compares the two across four dimensions — execution model, error handling, data quality, and cost — then walks through the combined pattern and a decision flow for picking between them.

Lakeflow Spark Declarative Pipelines vs Workflows: Side-by-Side Comparison

Lakeflow Spark Declarative Pipelines Basics: Declarative Table Definitions

In Lakeflow Spark Declarative Pipelines you simply declare "this table comes from this source with these transformations," and the framework handles dependency resolution, incremental processing, and schema evolution.

import dlt
from pyspark.sql.functions import col, current_timestamp

@dlt.table(comment="Bronze layer: raw ingestion")
def bronze_orders():
    return (
        spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/mnt/landing/orders/")
    )

@dlt.table(comment="Silver layer: cleansed")
@dlt.expect_or_drop("valid_amount", "amount > 0")
def silver_orders():
    return (
        dlt.read_stream("bronze_orders")
        .select("order_id", "customer_id", "amount", "region",
                current_timestamp().alias("processed_at"))
        .filter(col("order_id").isNotNull())
    )

@dlt.table(comment="Gold layer: aggregated")
def gold_revenue_by_region():
    return (
        dlt.read("silver_orders")
        .groupBy("region")
        .agg({"amount": "sum"})
        .withColumnRenamed("sum(amount)", "total_revenue")
    )

Use dlt.read_stream() for streaming reads and dlt.read() for batch reads. Dependencies between tables are resolved automatically from what each read/read_stream references, so you never have to specify execution order.

Workflows Basics: A Multi-Task DAG

Workflows defines task dependencies as a DAG (directed acyclic graph). Each task can be a notebook, Python script, JAR, Lakeflow Spark Declarative Pipelines pipeline, dbt project, and more, and you can assign different compute resources to each one.

# Conceptual DAG structure for Workflows
#
# [extract_task] ──→ [dlt_pipeline_task] ──→ [notify_task]
#       │                                         ↑
#       └──→ [validate_source_task] ──────────────┘
#
# - extract_task: pull data from an external API (Python notebook)
# - dlt_pipeline_task: ETL in a DLT pipeline (task type: Pipeline)
# - validate_source_task: check connectivity to the source data
# - notify_task: Slack notification (success/failure routed by a condition)

The strength of Workflows is the flexibility to set retries, timeouts, and conditional branches (if/else/for_each) per task. It can also unify non-transformation work like ML model training, external API calls, and notifications under one orchestrator.

The Combined Pattern: Workflows + Lakeflow Spark Declarative Pipelines

The most common production pattern is to embed a Lakeflow Spark Declarative Pipelines pipeline as a single task inside a Workflows DAG. Let Lakeflow Spark Declarative Pipelines handle the ETL transformations and quality controls, and let Workflows orchestrate everything around them.

In this setup the Lakeflow Spark Declarative Pipelines pipeline's success or failure propagates as the Workflows task status, so you can combine it with downstream conditional branches and retry settings for sophisticated error handling.

Decision Flow

Choose between Lakeflow Spark Declarative Pipelines and Workflows based on the nature of your pipeline and your team's needs.

Is the pipeline centred on table transformation (ETL/ELT)?
├─ YES -> do you need data quality management (Expectations)?
│    ├─ YES -> use DLT (consider pairing it with Workflows)
│    └─ NO  -> Workflows alone if simple, consider DLT if complex
└─ NO  -> a mix of tasks such as ML training, external APIs, notifications?
     └─ YES -> use Workflows (add DLT if there is an ETL part)

• Table transformations + quality controls needed → Lakeflow Spark Declarative Pipelines (standalone or combined with Workflows)
• Orchestrating diverse tasks → Workflows
• Integrating ETL + ML + notifications → Workflows + a Lakeflow Spark Declarative Pipelines pipeline task
• Scheduling a single notebook → Workflows alone is enough

Cost Differences

Lakeflow Spark Declarative Pipelines bills at a Lakeflow Spark Declarative Pipelines-specific DBU rate, which is higher than standard Jobs Compute DBUs. However, Lakeflow Spark Declarative Pipelines also delivers cluster optimization via Enhanced Autoscaling and automates incremental processing, so the right comparison is total cost — including the development and operational cost of building the same features by hand.

What the Exam Tests

• The concrete difference between Lakeflow Spark Declarative Pipelines (declarative) and Workflows (imperative)
• Lakeflow Spark Declarative Pipelines provides data quality controls via built-in Expectations; Workflows has no built-in quality features
• Workflows can run a Lakeflow Spark Declarative Pipelines pipeline as a task (the combined pattern)
• Lakeflow Spark Declarative Pipelines resolves dependencies automatically, while Workflows defines them explicitly via the DAG
• The DBU rate difference (Lakeflow Spark Declarative Pipelines > Jobs Compute)

Check Your Understanding

Frequently Asked Questions