Liquid Clustering Complete Guide: Delta Lake's Next-Gen Clustering

As Delta Lake tables grow, controlling how much data each query scans becomes the dominant performance factor. Traditionally, teams used partitioning or Z-ORDER to shape data layout, but changing partition keys required rebuilding the table, and Z-ORDER re-sorted the entire table every time it ran.

Liquid Clustering is Delta Lake's next-generation data layout optimization, introduced in DBR 13.3 LTS (GA in late 2023). With three defining features — no partitions, dynamic clustering key changes, and incremental re-clustering — it fundamentally removes the constraints of the older approaches.

Three Defining Features of Liquid Clustering

Liquid Clustering fundamentally differs from older approaches in these three ways.

• No partitions required: You no longer need PARTITIONED BY to split physical directories. You also do not need to worry about partition column cardinality (the number of distinct values), and data skipping works well even with high-cardinality columns such as user_id or timestamp.
• Instant key changes: A single ALTER TABLE changes the clustering keys. No table rebuild and no full data copy — only a metadata update. The next OPTIMIZE incrementally re-clusters the table with the new keys.
• Incremental optimization: OPTIMIZE only processes data that has not yet been clustered (files added by recent INSERTs). Unlike Z-ORDER, which re-sorts the entire table every time, the work scales with new data volume, so even very large tables finish quickly.

CLUSTER BY Syntax

Liquid Clustering is configured with the CLUSTER BY clause. You can specify it at table creation time or add it to an existing table afterwards.

Specifying at table creation time

-- Create a new table with Liquid Clustering
CREATE TABLE catalog.schema.sales (
  sale_id     BIGINT,
  customer_id BIGINT,
  region      STRING,
  sale_date   DATE,
  amount      DECIMAL(18,2)
)
CLUSTER BY (region, sale_date);

-- Create a Liquid Clustering table with CTAS
CREATE TABLE catalog.schema.sales_optimized
CLUSTER BY (customer_id, sale_date)
AS SELECT * FROM catalog.schema.raw_sales;

Changing clustering keys

-- Change the clustering keys (metadata-only update, completes immediately)
ALTER TABLE catalog.schema.sales
CLUSTER BY (customer_id, region);

-- Turn clustering off
ALTER TABLE catalog.schema.sales
CLUSTER BY NONE;

CLUSTER BY NONE disables clustering. The existing data layout is preserved, but subsequent OPTIMIZE runs will no longer perform clustering.

Z-ORDER vs. Partitioning vs. Liquid Clustering

Here is a side-by-side comparison of the three data layout optimization techniques. Liquid Clustering combines the strengths of both predecessors while eliminating their weaknesses.

Databricks officially recommends Liquid Clustering for all new tables. Partitioning and Z-ORDER remain available primarily for backward compatibility with existing tables.

Internals: Hilbert Curve and Incremental Re-clustering

Internally, Liquid Clustering projects multi-dimensional data onto one dimension using a Hilbert Curve, a space-filling curve. Compared with the Z-curve (Morton curve) used by Z-ORDER, the Hilbert Curve keeps spatially adjacent data points closer together along the 1D projection, which improves data skipping precision.

┌──────────────────────────────────────────────────────┐
│  Z-Curve (Morton order)         Hilbert Curve         │
│                                                       │
│  0 ─ 1   4 ─ 5                0 ─ 1   14─ 15        │
│  │   │   │   │                │   │    │   │         │
│  3 ─ 2   7 ─ 6                3 ─ 2   13─ 12        │
│                                    │        │         │
│  12─ 13  8 ─ 9                4 ─ 5   10─ 11        │
│  │   │   │   │                │   │    │   │         │
│  15─ 14  11─ 10               7 ─ 6    9 ─ 8        │
│                                                       │
│  -> spatially close data can  -> spatially close data │
│    end up far apart in 1-D       stays close in 1-D    │
└──────────────────────────────────────────────────────┘

Advantages of the Hilbert Curve:
  - better spatial locality than the Z-Curve
  - WHERE region = 'APAC' AND sale_date = '2026-03-01'
    improves skipping accuracy for multi-dimensional filters like this
  - narrower min/max statistics ranges,
    so a higher share of files can be excluded

How incremental re-clustering works

Liquid Clustering attaches metadata to each data file indicating whether it is "clustered" or "unclustered". When OPTIMIZE runs, only the unclustered files (those added by recent INSERT, UPDATE, or MERGE) are eligible for clustering work.

-- How incremental re-clustering works

-- Step 1: create the table
CREATE TABLE events CLUSTER BY (user_id, event_date) ...;

-- Step 2: add data (new files land unclustered)
INSERT INTO events SELECT * FROM raw_events;
-- -> new files have is_clustered = false

-- Step 3: run OPTIMIZE
OPTIMIZE events;
-- -> only files with is_clustered = false are read
-- -> they are re-arranged along the Hilbert Curve and written as new files
-- -> the new files have is_clustered = true
-- -> the old files are invalidated in the transaction log

-- Step 4: re-cluster only the newly added data
INSERT INTO events SELECT * FROM new_events;
OPTIMIZE events;
-- -> only the files added by the INSERT in Step 4 are touched
-- -> files already clustered in Step 3 are skipped

Relationship with OPTIMIZE

On tables with Liquid Clustering configured, the OPTIMIZE command behaves differently from a regular Delta table.

-- The correct OPTIMIZE for a Liquid Clustering table
OPTIMIZE catalog.schema.sales;
-- -> processes only the unclustered data, incrementally

-- An example that errors
OPTIMIZE catalog.schema.sales ZORDER BY (region);
-- -> Error: ZORDER BY cannot be used on a table with CLUSTER BY

Interaction with auto-OPTIMIZE

When Databricks Predictive Optimization is enabled, OPTIMIZE is automatically scheduled for Unity Catalog managed tables. On Liquid Clustering tables, this auto-OPTIMIZE also runs incremental re-clustering at the same time. You no longer need to run OPTIMIZE by hand, dramatically reducing operational overhead.

-- Enable Predictive Optimization (per Unity Catalog schema)
ALTER SCHEMA catalog.schema
SET DBPROPERTIES ('delta.enablePredictiveOptimization' = 'true');

-- Per-table setting
ALTER TABLE catalog.schema.sales
SET TBLPROPERTIES ('delta.enablePredictiveOptimization' = 'true');

-- How a Liquid Clustering table behaves once it is enabled:
-- 1. OPTIMIZE is scheduled automatically
-- 2. incremental re-clustering runs automatically
-- 3. VACUUM is scheduled automatically as well
-- -> no need to run maintenance commands by hand

Migration Patterns from Existing Tables

The migration steps from partitioned tables or Z-ORDER tables to Liquid Clustering differ depending on the table type.

Pattern 1: Migrating from a non-partitioned table that uses Z-ORDER

-- Step 1: enable Liquid Clustering
ALTER TABLE catalog.schema.events
CLUSTER BY (user_id, event_date);

-- Step 2: run incremental re-clustering
OPTIMIZE catalog.schema.events;

-- From here on OPTIMIZE ZORDER BY is unnecessary (and errors if used)
-- Subsequent OPTIMIZE runs cluster along the Hilbert Curve automatically

Pattern 2: Migrating from a partitioned table

You cannot enable Liquid Clustering directly on a partitioned table. You need to create a new table using CTAS (CREATE TABLE AS SELECT) or DEEP CLONE.

-- Option A: create a new table with CTAS
CREATE TABLE catalog.schema.events_v2
CLUSTER BY (region, event_date)
AS SELECT * FROM catalog.schema.events_partitioned;

-- Option B: DEEP CLONE + ALTER TABLE
CREATE TABLE catalog.schema.events_v2
DEEP CLONE catalog.schema.events_partitioned;

ALTER TABLE catalog.schema.events_v2
CLUSTER BY (region, event_date);

OPTIMIZE catalog.schema.events_v2;

-- Verify after migrating
SELECT COUNT(*) FROM catalog.schema.events_v2;
DESCRIBE DETAIL catalog.schema.events_v2;
-- Confirm that clusteringColumns shows [region, event_date]

Pattern 3: Changing clustering keys

-- Change the keys as the analysis pattern changes
-- Before: most filters were on region and sale_date
-- After:  filters on customer_id and product_category became more common

ALTER TABLE catalog.schema.sales
CLUSTER BY (customer_id, product_category);

-- The next OPTIMIZE starts incremental re-clustering on the new keys
OPTIMIZE catalog.schema.sales;

Constraints and Requirements

• DBR 13.3 LTS or later is required. Older runtimes do not recognize the CLUSTER BY syntax.
• Photon recommended: Clustering work is optimized on Photon Engine. It also runs without Photon, but OPTIMIZE will be noticeably slower.
• Cannot be combined with partitioning: You cannot configure PARTITIONED BY and CLUSTER BY on the same table. Migrating from a partitioned table requires CTAS or DEEP CLONE.
• Cannot be combined with Z-ORDER: Running OPTIMIZE ... ZORDER BY on a table that has CLUSTER BY configured raises an error.
• Number of clustering key columns: Up to 4 columns are allowed. Adding more columns dilutes the clustering effect, so 2-3 of your most frequently filtered columns is the recommended sweet spot.
• Supported column types: Numeric, date, timestamp, and string types are supported. Struct (STRUCT) and map (MAP) types cannot be used as clustering keys.
• Delta Lake only: Cannot be used with Parquet, CSV, or JSON tables. A Delta format table (managed or external Delta table) is required.

What to Watch for on the Exam

Liquid Clustering is an important topic on both the Data Engineer Associate (DEA) and Spark Developer exams. The differences versus Z-ORDER come up especially often, and you are typically asked to decide which one to use.

• The decisive difference vs. Z-ORDER: Z-ORDER re-sorts the full dataset every time. Liquid Clustering processes only unclustered data incrementally. When asked which is more efficient for periodic OPTIMIZE on large tables, the answer is Liquid Clustering.
• Co-existence constraint with partitioning: PARTITIONED BY and CLUSTER BY cannot be configured on the same table. Migrating from a partitioned table requires CTAS. Questions test this constraint directly.
• Changing keys with ALTER TABLE: You can change keys with ALTER TABLE t CLUSTER BY (new_col), with no need to rebuild the table. This shows up as 'the lowest-cost response when query patterns change'.
• CLUSTER BY NONE: The syntax for disabling clustering. The existing data layout is preserved, but subsequent OPTIMIZE runs no longer perform clustering.
• Predictive Optimization: Auto-OPTIMIZE is available for Unity Catalog managed tables. Combined with Liquid Clustering, you get fully automated data layout optimization.
• Why Hilbert Curve wins: Hilbert Curve has stronger spatial locality than Z-ORDER's Z-curve (Morton curve), improving skipping precision on multi-dimensional filters. Internalize this as the 'why' behind Liquid Clustering's advantage.

Sample Question

Frequently Asked Questions