Delta Lake Schema Evolution Complete Guide: mergeSchema & overwriteSchema

By default, Delta Lake performs schema enforcement (schema validation) and rejects writes whose data does not match the table definition. In real-world operations, however, columns get added or types change as source systems evolve. Schema evolution is the mechanism that lets a table flexibly keep up with those changes.

This article walks through the default behavior of schema enforcement, the difference between mergeSchema and overwriteSchema, Auto Loader's schema evolution modes, schema evolution in MERGE INTO, Column Mapping, and the constraints on partition columns.

Schema Enforcement (Default Behavior)

When you write to a Delta Lake table, if the schema of the incoming data does not match the table schema, the write is rejected with an error by default. That is schema enforcement.

• Incoming data has a column not in the table → error
• Column types do not match (e.g. STRING vs INT) → error
• Incoming data is missing a table column → filled with NULL (no error)

-- テーブル定義
CREATE TABLE events (event_id BIGINT, event_type STRING, amount DECIMAL(10,2));

-- 新しいカラム "region" を含むデータを書き込もうとするとエラー
INSERT INTO events
SELECT 1 AS event_id, 'purchase' AS event_type, 99.99 AS amount, 'APAC' AS region;
-- AnalysisException: cannot resolve 'region' given input columns

mergeSchema vs overwriteSchema Comparison

mergeSchema: Allowing Column Additions

# DataFrameの書き込み時にmergeSchemaを有効化
(df.write
  .format("delta")
  .mode("append")
  .option("mergeSchema", "true")
  .saveAsTable("events")
)

# SQLでセッション単位で有効化
SET spark.databricks.delta.schema.autoMerge.enabled = true;

# テーブルプロパティで永続的に有効化
ALTER TABLE events
SET TBLPROPERTIES ('delta.autoOptimize.autoCompact' = 'true');

Understanding exactly which operations mergeSchema does and does not allow is important on the exam.

overwriteSchema: Full Schema Replacement

# mode("overwrite") + overwriteSchema でスキーマを完全置換
(df.write
  .format("delta")
  .mode("overwrite")
  .option("overwriteSchema", "true")
  .saveAsTable("events")
)

overwriteSchema is a destructive operation. Because it can break compatibility with the table's existing data, use it carefully in production. You can use Time Travel to roll back to the previous schema, but data written under the new schema cannot be read under the old one.

Schema Evolution with Auto Loader

Auto Loader has its own schema inference and evolution features. cloudFiles.schemaEvolutionMode controls the behavior.

# Auto Loader + スキーマエボリューション
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
  .option("cloudFiles.schemaLocation", "/schema/events")
  .load("/landing/events/")
  .writeStream
  .option("checkpointLocation", "/checkpoint/events")
  .option("mergeSchema", "true")
  .table("bronze.events")
)

Schema Evolution in MERGE INTO

Schema evolution also works in MERGE INTO. When the source data has a new column and the Spark setting is enabled, the column is automatically added to the target table.

-- セッションレベルで有効化
SET spark.databricks.delta.schema.autoMerge.enabled = true;

-- ソースに "loyalty_tier" という新カラムがある場合
-- ターゲットに自動的に loyalty_tier カラムが追加される
MERGE INTO silver.customers t
USING staging.updates s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Schema evolution only kicks in when you use UPDATE SET * and INSERT *. If you specify columns individually with UPDATE SET t.col = s.col, new columns will not be added.

Column Mapping (Renaming and Dropping Columns)

From Delta Lake 2.0 onward, the Column Mapping feature makes it possible to rename and drop columns using ALTER TABLE.

-- Column Mappingの有効化
ALTER TABLE events SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.minReaderVersion' = '2',
  'delta.minWriterVersion' = '5'
);

-- カラム名の変更
ALTER TABLE events RENAME COLUMN event_type TO event_category;

-- カラムの削除
ALTER TABLE events DROP COLUMN temp_column;

Constraints on Partition Columns

Partition columns have special constraints under schema evolution.

• Adding a partition column: you cannot add a new partition column to an existing table
• Dropping a partition column: not allowed even with Column Mapping enabled
• Changing the type of a partition column: not allowed even with overwriteSchema
• Renaming a partition column: allowed when Column Mapping is enabled

If you need to fundamentally change the partition structure, you have to rebuild the table (CTAS). This is also one of the motivations for migrating to Liquid Clustering.

Key Exam Points

• Schema enforcement: enabled by default; schema mismatches reject the write
• mergeSchema: allows only new columns and safe type widenings; cannot drop columns
• overwriteSchema: full schema replacement; a destructive operation
• Differences between Auto Loader schema evolution modes (addNewColumns / failOnNewColumns / rescue / none)
• How to enable schema evolution in MERGE INTO (autoMerge setting + SET * syntax)
• Column Mapping: required for renaming and dropping columns (delta.columnMapping.mode = 'name')
• Constraints on changing partition columns

Check Your Understanding

Frequently Asked Questions