Databricks

What is Lakeflow Jobs? Complete Databricks Workflow Orchestration Guide

2026-03-26
Updated: 2026-10-13
NicheeLab Editorial Team
Table of contents
  1. Multi-Task Job Structure (DAG)
  2. Task Types at a Glance
  3. Scheduling
  4. Retries and Error Handling
  5. Choosing Between Job Cluster and All-Purpose Cluster
  6. Parameters and Task Value Hand-off
  7. Notification Settings
  8. Monitoring (Run History and System Tables)
  9. What the Exam Tests
  10. Check Your Understanding
  11. Frequently Asked Questions

Lakeflow Jobs is the orchestration layer on Databricks for scheduling and monitoring data pipelines and ML workflows. The product formerly known as "Databricks Jobs" (the "Workflows" item in the left navigation) was renamed "Lakeflow Jobs" on June 11, 2025. You compose multiple tasks into a DAG (directed acyclic graph) and manage dependencies, retries, and parameter passing in one place.

This article walks through multi-task job structure, scheduling, error handling, and monitoring from both a practical and exam-prep perspective.

Multi-Task Job Structure (DAG)

The base unit of Lakeflow Jobs is the "job", and one job consists of one or more "tasks". Defining dependencies between tasks forms a DAG, and a task starts automatically once its upstream dependencies complete.

[Ingest] --> [Transform_Bronze] --> [Transform_Silver] --> [Load_Gold]
                    |                                          |
                    +--> [Data_Quality_Check] ---------------->+
                                                               |
                                                          [Notify]

In the DAG above, Transform_Bronze and Data_Quality_Check start in parallel once Ingest completes, while Transform_Silver and Load_Gold run sequentially. The Notify task only runs after both Load_Gold and Data_Quality_Check finish. By combining serial, parallel, and join patterns, you can express complex pipelines.

You define dependencies either by selecting upstream tasks in the Depends on field of the UI or by listing the upstream task_key values under depends_on in a JSON/YAML definition. Cyclic dependencies fail validation.

Task Types at a Glance

You can mix different task types within a single job. For example, one job can run ETL in a Notebook, aggregate the result in a SQL task, and then run a streaming Lakeflow Spark Declarative Pipelines pipeline — all end-to-end.

Task TypeWhat It RunsTypical Use Case
NotebookRuns a Notebook in the workspaceETL, data transformation, ML training
Python scriptRuns a .py file from workspace files, a volume, cloud storage, or GitPackaged Python code
Python wheelInstalls and runs a .whl packageDistributable code produced by CI/CD
SQLRuns a SQL query or SQL fileAggregation and reporting table refresh
DashboardRefreshes results on a published dashboardRefreshing reports after aggregation
PipelineRuns a Lakeflow Spark Declarative Pipelines (formerly DLT) pipelineDeclarative ETL, streaming ingestion
dbtRuns tasks from a dbt Core projectBuilding and testing dbt models
JARRuns a Spark application JARBatch processing in Scala/Java
If/Else conditionBranches the DAG based on a conditionDynamic workflow branching
For EachLoops a task over an input listRepeating the same processing across multiple tables

Scheduling

Lakeflow Jobs can be triggered automatically on a schedule, on table updates, on file arrival, on Unity Catalog model updates (Beta), or continuously. You can also run a job manually (Run now) or from external orchestration tools. The DEA exam guide names three trigger types: scheduled, file arrival, and table update. This section covers those three plus continuous runs.

CRON Schedule

A standard time-based schedule. You can specify the timezone explicitly, for example 0 0 2 * * ? (daily at 2 AM) or 0 0 */6 * * ? (every 6 hours). It uses the Quartz CRON format (6 fields including a seconds field), so be aware that the field count differs from the standard Linux 5-field CRON.

File Arrival Trigger

Watches a Unity Catalog external location or volume (or a subpath of one) on S3 / ADLS / GCS and launches the job when new files arrive. Databricks checks for new files every minute on a best-effort basis; enabling file events on the external location lets it use change notifications from the cloud provider and detect files more efficiently. It is ideal for batch ingestion pipelines that follow a "process when a file lands" pattern, where files arrive at irregular times.

Table Update Trigger

Launches the job when monitored tables are updated, including updates, merges, and deletes. You can monitor one or more Unity Catalog managed tables (Delta or Iceberg), external Delta tables, materialized views, streaming tables, or views that depend on them, and choose whether a run starts when any table or all tables are updated. The advanced options "Minimum time between triggers" and "Wait after last change" let you combine closely spaced updates into one run. The job's Run as principal needs SELECT on the monitored tables plus USE CATALOG and USE SCHEMA on their parents.

Continuous Run

A mode that always keeps exactly one run of the job active: when a run finishes, successfully or not, the next run starts after a delay of less than 60 seconds. Use it for always-on streaming workloads. Retry policies cannot be set on a continuous job; instead, the whole job is retried automatically with exponential backoff, and failed tasks inside a run are retried according to the Task retry mode (On failure by default).

Retries and Error Handling

Retry policies are configured per task. In most configurations tasks are not retried by default, but serverless jobs retry failed tasks through auto-optimization by default, and continuous jobs use the exponential backoff described above. The last row of the table is not a retry setting; it is the job-level limit on concurrent runs.

LevelParameterBehavior
Task-levelmax_retriesMaximum number of times that single task is re-run when it fails (default 0; -1 retries indefinitely)
Task-levelmin_retry_interval_millisMinimum interval between the start of the failed run and the retry (milliseconds)
Task-levelretry_on_timeoutWhether timeouts are also retried
Job-levelmax_concurrent_runsMaximum concurrent runs of the same job (default 1)

When a retry policy is configured on a task, a failed task is first retried in isolation. If the task still fails after max_retries, it is marked as a final "failure", and downstream tasks that use the default Run if condition (All succeeded) are marked "Upstream failed" and do not run. The overall job result becomes "failure", but tasks on branches that do not depend on the failed task continue to run normally.

The "Repair Run" feature lets you re-run only the failed task and its downstream tasks. Already-successful tasks are not re-executed, which is very useful for partial recovery in large pipelines.

Choosing Between Job Cluster and All-Purpose Cluster

Lakeflow Jobs tasks can run on serverless compute for jobs (the default for tasks that support it), a classic Job Cluster (a job-dedicated cluster), a SQL warehouse for SQL tasks, or pipeline compute. An All-Purpose Cluster (an interactive cluster) can also be selected, but Databricks does not recommend it for production jobs. The table compares the two classic options.

AspectJob ClusterAll-Purpose Cluster
LifecycleAuto-created at job start, auto-terminated on completionManually started and stopped (auto-termination is configurable)
DBU rateLower than All-Purpose (check each cloud's pricing page for the rates)Higher than a Job Cluster
Startup latencyCluster spins up on each run (a few minutes)Runs immediately if the cluster is already up
When to useProduction batch and scheduled runsDevelopment test runs and interactive debugging

For production pipelines, you should generally use jobs compute (serverless or a Job Cluster). A Job Cluster has a lower DBU rate than All-Purpose compute and terminates automatically after the run, so there is no risk of leaving resources idle. During development, it is common to test manually on an already-running All-Purpose Cluster and then switch to jobs compute when deploying to production.

Parameters and Task Value Hand-off

Lakeflow Jobs offers two mechanisms: "Job Parameters" passed into the job from outside, and "Task Values" passed between tasks at runtime.

Job Parameters

You define parameter keys and default values (fixed values or dynamic value references) at job definition time, and you can override them when you trigger a run with different parameters from the UI or the API, or when you repair a run. Built-in dynamic references such as {{job.start_time.iso_date}} and {{job.run_id}} are available. Inside a Notebook, fetch them with dbutils.widgets.get("param_name").

Task Values (dbutils.jobs.taskValues)

A mechanism for passing the result of one task to downstream tasks.

# Set values in Task A
dbutils.jobs.taskValues.set(key="row_count", value=df.count())
dbutils.jobs.taskValues.set(key="target_date", value="2026-03-27")

# Read values in Task B (depends on Task A)
row_count = dbutils.jobs.taskValues.get(
    taskKey="task_a", key="row_count"
)
target_date = dbutils.jobs.taskValues.get(
    taskKey="task_a", key="target_date"
)

Values must be representable as JSON (strings, numbers, booleans, lists, and so on), and the JSON representation cannot exceed 48 KiB. Large DataFrames or binary data cannot be passed this way; in those cases, write to a Delta Table or temporary file and share only the path via a taskValue.

Notification Settings

You can configure notifications for job events such as start, success, failure, a run exceeding its duration threshold, and a streaming backlog exceeding its threshold. You can also mute notifications for skipped or canceled runs. Destinations other than email must first be set up by an administrator as system notification destinations.

  • Email: send to any email address; multiple recipients supported
  • Webhook: send to any HTTPS endpoint to integrate with custom systems
  • Slack integration: register a Slack Webhook as a Databricks destination to post into channels
  • Microsoft Teams: register a webhook URL as the destination
  • PagerDuty: register a PagerDuty integration key as the destination

Notifications can be configured at both the job and task levels. For example, you can route "overall job failure to the on-call team" while sending "specific task failures to the data quality team", splitting granularity so the right people get the right signal without alert fatigue.

Monitoring (Run History and System Tables)

The Lakeflow Jobs UI lets you inspect each job's run history, with success/failure status, execution time, and per-task status displayed together. Run history is retained for 60 days.

For deeper analytics and longer retention, use the Unity Catalog System Tables. The system.lakeflow schema provides the following tables.

TableContents
system.lakeflow.jobsJob definition metadata (name, creator, trigger configuration, run-as identity, etc.)
system.lakeflow.job_tasksPer-task definition (task key, upstream dependencies, etc.)
system.lakeflow.job_run_timelinePer-run timeline (start/end times, result status)
system.lakeflow.job_task_run_timelinePer-task run timeline (duration and status of each task)

Aggregating these tables in SQL surfaces insights like "the jobs with the highest failure rate over the past 30 days", "tasks whose average runtime is trending upward", and "cluster utilization spikes during specific time windows". Combined with Databricks SQL dashboards, you can build operations-team-facing job health monitoring.

What the Exam Tests

On the Data Engineer Associate exam, Lakeflow Jobs has its own section, "Working with Lakeflow Jobs" (16%). Based on the guide's objectives, make sure you can answer the following.

  • Which to use in production: jobs compute or an All-Purpose Cluster → jobs compute, serverless or a Job Cluster (cost and auto-termination)
  • How to re-run only the failed task after a job failure → Repair Run
  • How to pass a previous task's computed result to the next task → dbutils.jobs.taskValues.set() / get()
  • How to launch a job when a file arrives → File Arrival Trigger
  • How to launch a job when source tables are updated → Table Update Trigger
  • Field count of the CRON expression → 6 fields in Quartz format (including seconds)
  • If task dependencies contain a cycle → Validation error (no longer a DAG, so cannot be created)

Also keep the roles of a Lakeflow Spark Declarative Pipelines pipeline, a Lakeflow Jobs job, and a standalone Notebook run distinct. A pipeline declares the relationships between datasets and transformations, while a job defines the order and trigger conditions of tasks procedurally; a pipeline can run as one task inside a job.

Check Your Understanding

Data Engineer Associate – Working with Lakeflow Jobs

Question 1

A data engineer operates a production multi-task job. In last night's run, only the third of five tasks (Transform_Silver) failed due to a transient network issue. Tasks 1 and 2 succeeded, and tasks 4 and 5 did not run (Upstream failed). To efficiently re-run only the failed task and its downstream tasks without re-running the already-successful tasks, which feature should be used?

  1. Re-run the entire job from the beginning
  2. Use Repair Run to re-run only the failed task and its downstream tasks
  3. Manually run the failed task's Notebook on an All-Purpose Cluster
  4. Delete the job, recreate it with the same definition, and run it

Correct answer: B

Repair Run re-runs only the failed task and its downstream tasks. Already-successful tasks (Ingest, Transform_Bronze) are not re-executed, saving compute cost and runtime. A full job re-run is inefficient because it re-executes all tasks, including ones that already succeeded. A manual Notebook run loses DAG dependency management and does not trigger downstream tasks automatically.

Frequently Asked Questions

What is the difference between Lakeflow Jobs and Databricks Jobs (Workflows)?

There is no functional difference. On June 11, 2025, the product known as Databricks Jobs was renamed Lakeflow Jobs, and no migration is required. DLT (Delta Live Tables) was renamed on the same day; the DEA exam guide calls it Lakeflow Spark Declarative Pipelines. The Workflows item in the left navigation was folded into Jobs & Pipelines on June 18, 2025. The REST API (for example /api/2.2/jobs/) and the CLI jobs commands still use the jobs name. The current exam guide says Lakeflow Jobs, but older study materials and blog posts may say Databricks Jobs or Workflows; they all refer to the same feature.

When should I use dbutils.jobs.taskValues vs. Job Parameters?

Job parameters are key-value pairs set on the job (static values or dynamic value references). You can override them when you trigger a run, and they are pushed down automatically to tasks that take key-value parameters. By contrast, dbutils.jobs.taskValues.set() / get() are for runtime data hand-off between tasks. For example, when an upstream task computes a row count or partition key that a downstream task needs, use taskValues. Job parameters suit values that are known when the run starts, such as the run date, passed as fixed values or dynamic value references (such as {{job.start_time.iso_date}}).

How heavily is Lakeflow Jobs tested on the Data Engineer Associate exam?

In the Data Engineer Associate Exam Guide (exam version as of May 4, 2026), Lakeflow Jobs has its own section, 'Working with Lakeflow Jobs', which accounts for 16% of the exam. The guide's objectives cover control flow with retries and conditional tasks (branching and looping), common tasks (notebook, SQL query, dashboard, and pipeline tasks) and their dependencies in the DAG, the scheduled, file arrival, and table update trigger types, and choosing between time-based and data-driven triggers. Choosing compute for jobs falls under the compute-selection objective of Section 1 (Databricks Intelligence Platform).

Check what you learned with practice questions

Practice with certification-focused question sets

Try free questions
Author

NicheeLab Editorial Team

NicheeLab editorial team focused on data engineering and cloud certification learning. Content is structured around practical study needs and official exam domains.


Related articles
Databricks

Databricks Certifications: All 8 Exams, Difficulty & Study Plan (2026)

Complete guide to all 8 Databricks certifications — Data Eng...

Databricks

Databricks Exam Difficulty Ranking: 7 Certs Compared (2026)

Seven Databricks certifications ranked by difficulty, with s...

Databricks

Databricks Study Guide: Fastest Pass Route & Time Estimates (2026)

How to pass Databricks certifications efficiently. Official ...

Databricks

Databricks Data Engineer Associate: Complete Guide (2026)

Domain-by-domain breakdown of the Databricks Certified Data ...

Databricks

Databricks Data Engineer Professional: Complete Guide (2026)

Tactics for the Databricks Certified Data Engineer Professio...

Browse all Databricks articles (110)
© 2026 NicheeLab All rights reserved.