Kafka Schema Evolution in Practice: A Guide to Preserving Backward Compatibility

Preserving backward compatibility means keeping messages written with old Writer schemas readable by the new Reader schema. The BACKWARD setting on Confluent Schema Registry (especially the TRANSITIVE variant) is the linchpin.

This article is aligned with the CCDAK exam scope and focuses on precise terminology and concrete techniques you can apply on the job — safely adding fields, renaming them, and phasing them out incrementally.

Terminology and Compatibility Levels (Frequent CCDAK Topics)

Backward compatibility guarantees that a new Reader schema (typically on the new consumer side) can read data written with old Writer schemas. BACKWARD on Confluent Schema Registry expresses this property, and adding TRANSITIVE extends the guarantee to every version in the full history.

Conversely, when you want to ship a new Producer without breaking existing consumers, you need forward compatibility (FORWARD). FULL requires satisfying BACKWARD and FORWARD at the same time — it is the safest mode to maintain, but it narrows the set of allowed changes. In practice, you pick a level based on rollout order: are you updating consumers first, or producers first?

Based on Avro's reader/writer resolution rules, defaults on added fields, numeric type promotion, and aliases all help preserve compatibility. Protobuf and JSON Schema follow similar principles, but the details differ (for example, in Protobuf the field number is what matters).

• BACKWARD (TRANSITIVE recommended): new Reader can read data from old Writers
• FORWARD (TRANSITIVE recommended): old Reader can read data from new Writers
• FULL: satisfies both at once (the most restrictive set of allowed changes)
• Picking the right mode based on rollout order is the standard practice

Backward compatibility (BACKWARD_TRANSITIVE) flow

Setting the compatibility level (example: BACKWARD_TRANSITIVE)

curl -s -X PUT http://localhost:8081/config/t-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD_TRANSITIVE"}'

# グローバル既定を変える場合（推奨はSubject単位での設定）
# curl -s -X PUT http://localhost:8081/config -H "Content-Type: application/vnd.schemaregistry.v1+json" -d '{"compatibility": "BACKWARD_TRANSITIVE"}'

Core Patterns for Preserving Backward Compatibility

Adding a field is the safest and most common change. In Avro, always assign a default to the new field. Even when the old Writer omits the field, the new Reader can fall back to the default and still read the record.

Numeric type promotion (int → long/float/double, long → float/double, float → double) is valid for backward compatibility on the Reader side under Avro's resolution rules. Adding a new enum symbol is also backward compatible, but removing or reordering symbols requires care.

Renaming a field is breaking by default, but Avro's aliases let you map data written with the old name onto the field with the new name. In Protobuf, the field number is what matters more than the name, so renames are safe as long as the number stays the same.

• Added fields require a default in Avro. Missing defaults very often trigger compatibility violations.
• Type promotion is only safe on the Reader side (in the backward-compatibility context).
• For enums, additions are fine; deletions and replacements are effectively forbidden.
• Absorb renames with aliases (Avro) or by keeping the same field number (Protobuf).

Avro schema example: adding fields and renaming with aliases

// 旧スキーマ（Sv0）
{
  "type": "record",
  "name": "Order",
  "namespace": "com.example",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "amount", "type": "int"},
    {"name": "status", "type": {"type":"enum","name":"Status","symbols":["NEW","PAID"]}}
  ]
}

// 新スキーマ（Sr1, BACKWARD互換）：フィールド追加＋改名
{
  "type": "record",
  "name": "Order",
  "namespace": "com.example",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "amount", "type": "long"},               // Reader側の型拡張
    {"name": "state", "type": "string", "aliases": ["status"], "default": "NEW"},
    {"name": "coupon", "type": ["null","string"], "default": null} // 新規追加
  ]
}

# 互換性チェック（最新と比較）
curl -s -X POST http://localhost:8081/compatibility/subjects/t-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{ \"type\": \"record\", \"name\": \"Order\", ... }"}'

Staged Migration: Making Breaking Changes Safely

Apparently breaking changes — removing a required field, narrowing a type, removing an enum symbol, or changing a Protobuf field number — should never be done in one shot. Decompose them into stages: first run the old and new shapes side by side in a backward-compatible form, then switch consumers, then producers, and finally remove what is no longer needed.

During migration, pin the Registry compatibility to BACKWARD_TRANSITIVE and only switch briefly to a FORWARD variant if you genuinely need a producer-first phase. Keep in mind that FULL narrows the set of allowed changes even further because it must satisfy both directions simultaneously.

• Step 1: Add the new field (with a default) and keep the old field around (mark it deprecated, etc.)
• Step 2: Switch consumers to read the new field, falling back to the old field if needed
• Step 3: Migrate producers to write only the new field (add a dual-write period if it makes the transition safer)
• Step 4: Once every consumer has been updated, drop the old field

Registration flow guarded by compatibility checks (pseudocode)

# 1) 互換性は原則 BACKWARD_TRANSITIVE
curl -s -X PUT http://localhost:8081/config/t-value -H "Content-Type: application/vnd.schemaregistry.v1+json" -d '{"compatibility":"BACKWARD_TRANSITIVE"}'

# 2) 登録前に必ずドライラン確認
curl -s -X POST http://localhost:8081/compatibility/subjects/t-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" -d '{"schema": "...new schema..."}'

# 3) 問題なければ登録
curl -s -X POST http://localhost:8081/subjects/t-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" -d '{"schema": "...new schema..."}'

Choosing a Subject Naming Strategy and Its Impact

Compatibility is evaluated per subject. TopicNameStrategy (for example, topic-name-value) is the common choice and lets you manage compatibility independently per topic. If you want to share a record across multiple topics, RecordNameStrategy shares the compatibility history of identically named records — but watch out for unexpected collisions.

When you mix in multi-topic publishing or Kafka Connect/ksqlDB, confirm up front which subject will be registered (value vs. key, topic-driven vs. record-driven) and align your compatibility level and migration plan accordingly.

• TopicNameStrategy: intuitive operationally, isolated per topic
• RecordNameStrategy: great for schema reuse, but wider blast radius when collisions happen
• TopicRecordNameStrategy: a middle ground — reduces collision risk while keeping reuse

Configuring the subject name strategy on the Kafka Avro Serializer (example)

props.put("value.subject.name.strategy", "io.confluent.kafka.serializers.subject.TopicNameStrategy");
// 例: RecordNameStrategy を使う場合
// props.put("value.subject.name.strategy", "io.confluent.kafka.serializers.subject.RecordNameStrategy");

Production Workflow: Checklist and CI/CD Gates

Gate schema changes with the same rigor as code reviews. Run a dry-run check up front against the Registry's compatibility API, and automate a regression test that reads old-version messages with the new schema using fixture data. Comparing normalized schemas (absorbing field ordering and whitespace differences) keeps reviews focused on real changes.

In production, set the compatibility level explicitly on every subject. When you need to relax it as an exception, file a ticket that captures the time window, impact, and rollback procedure. Keeping an audit map from schema IDs to Git commits is also valuable.

• Require compatibility-API dry runs in CI
• Replay-test against old data (reader-side regression check)
• Manage compatibility levels explicitly per subject
• Maintain a mapping from schema IDs to release tags

Simple CI dry-run example (Bash)

SUBJECT="t-value"
NEWSCHEMA_FILE="schema_new.avsc"
BODY=$(jq -Rs '{schema: .}' < "$NEWSCHEMA_FILE")

curl -s -X POST "http://localhost:8081/compatibility/subjects/${SUBJECT}/versions/latest" \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d "$BODY" | jq .

Consumer Implementation: Caveats and Operations

Backward compatibility lives or dies on consumer robustness. Define defaults on the new Reader schema, ignore unknown fields, and branch on nullability explicitly. Log-compacted topics may deliver tombstones (value=null), so make sure your code is null-safe whether you use Avro, JSON Schema, or Protobuf.

With GenericRecord, do existence checks and apply defaults yourself; with SpecificRecord, you rely on the defaults baked into the generated code. On errors, prefer routing messages to a dead-letter or retry queue over silently skipping them — that way schema and data mismatches stay observable.

• Never blur the distinction between null and unset
• Always carry defaults on the schema itself
• Quarantine exceptions in a DLQ to surface root causes
• Have a replay plan ready

Defensive read with Avro GenericRecord (Java, sketch)

GenericRecord r = consumerRecord.value();
Long amount = null;
if (r.getSchema().getField("amount") != null) {
  Object v = r.get("amount");
  if (v instanceof Integer) amount = ((Integer) v).longValue();
  else if (v instanceof Long) amount = (Long) v;
}
String state = (String) (r.get("state") != null ? r.get("state") : "NEW");
// coupon は optional
String coupon = r.get("coupon") == null ? null : r.get("coupon").toString();

Check Your Understanding

Frequently Asked Questions