Kafka Cluster Linking: Offset-Preserving Replication and Migration

Cluster Linking is an official Confluent feature in which brokers replicate directly to each other, copying partitions, records, and offsets one-to-one. This minimizes consumer rebalancing and offset translation during migrations and DR scenarios.

This article covers the design considerations for offset-preserving replication and migration, a phased cutover procedure, monitoring and verification, and the differentiators most commonly tested on the CCAAK exam.

Cluster Linking Fundamentals and Offset Preservation

Cluster Linking is server-side replication in which the destination cluster fetches data from the source via a link. Unlike MirrorMaker 2, there is no external connector — a per-topic "mirror" is created on the destination that preserves the source topic's partition layout, ordering, and the exact offset of every record.

Because offsets are preserved, consumers can resume from the same numeric offset after a cluster switch. The offset translation and consistency wait times that were previously required are largely eliminated, simplifying DR, region migrations, and blue-green deployments.

• Links are one-way. The destination holds a read-only mirror topic, and only the source cluster accepts writes.
• The mirror topic matches the source in partition count and numbering, and preserves the offset of every record exactly.
• Promotion converts the mirror topic into a regular topic, after which the destination can accept produces.
• Plan under the assumption that consumer group commit information (_consumer_offsets) is not transferred automatically — this is the safe default.

Offset-preservation overview (same partition, same offset)

Minimal link creation (executed on the destination cluster)

# 宛先: リンク作成（プロパティファイルで安全に資格情報を保持）\n# link-src.properties 例\n# bootstrap.servers=src-broker:9092\n# security.protocol=SASL_SSL\n# sasl.mechanism=PLAIN\n# sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"...\" password=\"...\";\n\n# 宛先でリンクを作成\n$ kafka-cluster-links \\\n  --bootstrap-server dest-broker:9092 \\\n  --create --link src-to-dest \\\n  --config-file link-src.properties\n\n# ミラートピック作成（orders を元からミラー）\n$ kafka-mirrors \\\n  --bootstrap-server dest-broker:9092 \\\n  --create --mirror-topic orders --link src-to-dest

How Offset Preservation Behaves and What It Requires

With Cluster Linking, each mirror-topic partition fetches from the matching source partition and preserves every record's offset, timestamp, and headers. Consumers therefore need no offset translation during a cluster switch, and seek semantics remain consistent.

However, consumers may not resume cleanly unless the following conditions are met. In particular, the destination cannot accept writes while the topic is mirrored, so writes must remain directed at a single cluster until full cutover is complete.

• Matching partition count and partition IDs (the mirror topic inherits the source partition layout).
• Consistent key distribution and ordering (the same key always lands on the same partition).
• Consumer group commit information must be migrated to the destination in advance or aligned manually (see the procedure below).
• Design under the assumption that transaction boundaries do not span clusters.

Inspecting mirror state (lag and health)

# リンク/ミラーの記述\n$ kafka-cluster-links --bootstrap-server dest:9092 --describe --link src-to-dest\n$ kafka-mirrors --bootstrap-server dest:9092 --describe --mirror-topic orders\n\n# 代表的な遅延メトリクス（JMX名は環境によりプレフィックス差異あり）\n# kafka.server: type=ReplicaFetcherManager,name=MaxLag,clientId=*,link=src-to-dest\n# kafka.server: type=MirrorTopicMetrics,name=TimeLag,topic=orders

Choosing Between Migration, DR, and Blue-Green Use Cases

Migration: Keep producing to the existing cluster (source), let the destination mirror catch up, promote when caught up, and switch writes over. Thanks to offset preservation, consumers can keep reading from the same offsets.

DR: Mirror one-way in steady state. On failure, promote the mirror on the destination and switch producers and consumers over. After recovery, create a new link in the reverse direction (avoid simultaneous two-way writes to prevent split-brain).

• Blue/green: Create a mirror on the green side, promote it, and shift traffic gradually.
• Phased validation: Mirror and promote a subset of topics for a rolling migration.
• Compliance / data residency: For cross-region replication, manage encryption and authorization independently (distribute ACLs/RBAC separately).

Minimal sequence for promotion and write switchover

# 1) 生産を短時間停止（ソースの最終レコードまで追いつかせる）\n# 2) 宛先でミラーを昇格（以後、通常トピック）\n$ kafka-mirrors --bootstrap-server dest:9092 --promote --mirror-topic orders\n# 3) プロデューサのbootstrap.serversを宛先に変更して再開\n# 4) コンシューマは同じgroup.id/オフセットで継続（次節のオフセット整合参照）

A Cutover Procedure That Leverages Offset Preservation

Cluster Linking preserves record offsets, but consumer group commits live in a separate topic, so it is safer not to assume automatic migration. The procedure below minimizes downtime while cutting over at the same offsets.

Note: relying on the default auto.offset.reset=latest/earliest behavior will shift the resume position, so align group offsets explicitly on the destination.

• Prerequisites: source and destination must agree on topic name and partition layout. Confirm the mirror lag is sufficiently small.
• Order: brief produce stop → promote → resume produces. Pause consumers if needed, and resume immediately after promotion.
• Set group offsets to the same numeric value on the destination manually (e.g. kafka-consumer-groups --reset-offsets --to-offset).

Pre-aligning group offsets (example: single topic, few partitions)

# 1) ソースで現在のコミット位置を確認\n$ kafka-consumer-groups --bootstrap-server src:9092 --describe --group app-g1 \\n  --topic orders\n# 出力から各partitionのCURRENT-OFFSETを控える（p0=1043, p1=879 など）\n\n# 2) 宛先で同じgroup.idに対し、手動で同値を設定\n$ kafka-consumer-groups --bootstrap-server dest:9092 --group app-g1 \\n  --topic orders --reset-offsets --to-offset 1043 --partition 0 --execute\n$ kafka-consumer-groups --bootstrap-server dest:9092 --group app-g1 \\n  --topic orders --reset-offsets --to-offset 879  --partition 1 --execute\n\n# 3) ミラーを昇格し、コンシューマ/プロデューサを宛先に切替\n$ kafka-mirrors --bootstrap-server dest:9092 --promote --mirror-topic orders

Operations, Monitoring Metrics, and Troubleshooting

Lag is the heart of any cutover go/no-go decision. Monitor TimeLag/RecordsLag, link health, and fetch errors, and define an acceptable lag threshold. Be careful with cases where network bandwidth or differing retention.ms values stall progress.

Configuration drift causes unexpected behavior. Verify cleanup.policy, min.insync.replicas, compression.type, and other settings at the time the mirror is created, and change them explicitly after promotion if needed. Treat ACLs/RBAC as a separate concern and distribute access rights ahead of cutover.

• Lag monitoring: MirrorTopicMetrics (TimeLag/RecordsLag), link state (Established/Failed).
• Bandwidth control: upper/lower bounds for replication bandwidth and peak-hour tuning.
• Verification: spot-check offset alignment via sample consumes on the destination.

Example: simple health check and lag retrieval

# Describeでリンクとミラーの状態を点検\n$ kafka-cluster-links --bootstrap-server dest:9092 --describe --link src-to-dest | sed -n '1,200p'\n$ kafka-mirrors --bootstrap-server dest:9092 --describe --mirror-topic orders\n\n# JMXをcurl/jolokia等で取得し、TimeLagがしきい値未満であることを確認\n# 例: curl http://jolokia:8778/jolokia/read/kafka.server:type=MirrorTopicMetrics,name=TimeLag,topic=orders

CCAAK Exam: Key Points to Remember

Cluster Linking's biggest differentiator is record offset preservation. Lock in three points: no translation step like MirrorMaker 2's Offset Sync, the mirror topic is read-only on the destination, and promotion turns it into a regular topic.

The exam frequently asks where the feature lives (broker vs. Connect), what operations are required at cutover (promote, align group offsets), and design choices like avoiding dual writes.

• Offset preservation = the consume position is numerically identical. Group commits are typically managed separately.
• While mirroring, the destination cannot accept writes. Writes are allowed only after promotion.
• Cluster Linking is server-side. MM2 is Connect-based.

Quick recap (mnemonic via pseudo-commands)

# create link -> create mirror -> catch up -> promote -> switch clients\n# オフセット保持: 変換不要。グループオフセット: 手動整合/移行を計画。

Check Your Understanding

Frequently Asked Questions