Kafka Unclean Leader Election: Balancing Data Loss Risk and Availability

Unclean Leader Election (ULE) is a mechanism that, in emergencies when no live replica remains in the ISR, elects a leader even from the OSR to restore availability. The price you pay is the rollback of unreplicated records — that is, data loss.

This article walks through ULE's behavior, the data-loss mechanism, the criteria for configuration decisions, combinations with related parameters, monitoring and drills, and the points most frequently tested on the CCAAK exam — all from a practical perspective. The explanation is grounded in stable concepts that align with Kafka's official documentation.

ULE Basics: Correctly Understanding ISR and OSR

Each Kafka partition consists of a leader and followers, and the set of replicas that are sufficiently in sync is the ISR (in-sync replicas). Normal leader elections occur only from the ISR, which preserves consistency.

ULE elects a leader even from the OSR (out-of-sync replicas) when no live replica remains in the ISR after a failure, bringing the partition back online. The default is disabled (false); while disabled, the partition stays offline until the ISR recovers.

• Purpose: emergency availability recovery (shorter downtime)
• Side effect: loss/rollback of unreplicated data that only the former leader held
• Configuration scope: broker default and per-topic override (topic value wins)
• Related: min.insync.replicas, producer acks, replication requirements

Election path during a failure (when the entire ISR is lost)

パーティション P (RF=3)

初期状態:
  R1(Leader)  R2(Follower)  R3(Follower)
  ISR = {R1, R2}   OSR = {R3}

障害発生: R1ダウン、R2も同時障害
  生存: R3のみ（ただしOSR）

分岐:
  1) ULE=false（デフォルト）
     -> リーダー不在のまま待機
     -> パーティションPは一時オフライン
     -> ISR回復後に通常の選挙

  2) ULE=true
     -> OSRのR3を暫定リーダーに選出
     -> 旧R1にだけあった未複製レコードは失われる（ロールバック）
     -> 可用性は回復するが履歴は不連続になり得る

Related properties (broker default and topic override)

# broker（server.properties あるいは動的設定）
unclean.leader.election.enable=false  # 推奨デフォルト

# トピック単位で例外的に有効化（避難ログ等）
# Kafka (kafka-configs.sh)
 kafka-configs.sh --bootstrap-server <broker:9092> \
  --alter --topic critical-telemetry \
  --add-config unclean.leader.election.enable=true

# 有効値の確認
 kafka-configs.sh --bootstrap-server <broker:9092> \
  --describe --topic critical-telemetry | grep unclean

When It Fires and What You Lose: The Election and Data-Loss Mechanism

ULE is only meaningful when 'no live replica exists in the ISR' AND 'at least one live OSR exists.' When ULE is disabled, the partition temporarily goes offline, and clients may receive errors like NotLeaderOrFollower or LEADER_NOT_AVAILABLE.

When enabled, the controller elects a new leader from the OSR. Records that were not fully replicated from the old leader to the new leader do not exist in the new leader's log, so when the old leader rejoins, truncation occurs based on the leader epoch and the high watermark (HW), and those records disappear from history. Writes that were acknowledged with acks=1 may be lost. With acks=all and a properly configured min.insync.replicas, writes that meet those conditions are in principle not lost; writes that do not meet the conditions are not ACKed and are returned as failures, so the application needs a retry design.

• Trigger conditions: 0 live replicas in ISR, >=1 live in OSR
• Data-loss unit: tail records not replicated between old and new leaders (past the high watermark)
• Observability: truncation, epoch updates, changes in offline partition count

Example controller/broker logs (key points only)

[Controller] Partition P: No live ISR; unclean leader election is enabled
[Controller] Electing new leader from OSR: R3, new leader epoch=42
[Broker R3] Became leader for P-0 at epoch 42
[Broker R1] Truncating log for P-0 from offset 105 down to HW 102 (on recovery)

Configuration Decisions: The Availability vs. Durability Trade-off

For topics where history consistency is paramount, like financial accounting or order events, generally leave ULE disabled and enforce RF>=3, min.insync.replicas>=2, and producer acks=all. Accept the downtime and protect an error-free history.

Conversely, for observability/telemetry topics where short gaps are not business-critical, depending on SLA, you may decide to enable ULE on selected topics. Even then, design min.insync.replicas, acks, alerting, and retry logic together as one package.

• Principle: keep ULE disabled for critical data. Enable per-topic only as an exception when truly unavoidable.
• Companion settings: design RF, min.insync.replicas, acks, and producer retry/ordering as a single set.
• Recovery consistency: consumers must implement duplicate/gap detection (offset management, idempotent processing).

Examples reflecting these decisions (payments = strict / telemetry = availability-first)

# 決済トピック（厳格）
 kafka-topics.sh --bootstrap-server <broker:9092> --alter --topic payments \
  --config unclean.leader.election.enable=false \
  --config min.insync.replicas=2
# プロデューサは acks=all を必須化

# テレメトリ（可用性優先）
 kafka-topics.sh --bootstrap-server <broker:9092> --alter --topic telemetry \
  --config unclean.leader.election.enable=true \
  --config min.insync.replicas=2
# ただし未達はアプリ側で再送・欠落許容の前提設計を行う

Operational Configuration and Verification

Start with an audit. Inspect both the broker defaults and per-topic overrides, and confirm there is no drift from expectations. When possible, make changes minimally and per-topic — modifying cluster-wide defaults has a much larger blast radius.

Dynamic config changes do not require a rolling restart, but you should verify in staging — via fault injection (broker shutdown) — that the change does not conflict with existing client SLAs or retry strategies.

• Audit: confirm ULE=true has not crept into critical topics
• Rollout: validate in stages — 1 topic, then a few, then all
• Risk mitigation: combine RF>=3, min.insync.replicas>=2, acks=all, and the idempotent producer

Concrete examples of audit and change

# ブローカー既定（ランタイムの動的設定も含む）
 kafka-configs.sh --bootstrap-server <broker:9092> --entity-type brokers --entity-name 0 --describe
# （クラスタ全体を見たい場合は各ブローカー、または--allや外部CMDBを使用）

# トピックの有効値
 kafka-configs.sh --bootstrap-server <broker:9092> --describe --topic <topic>

# 既定値の変更（慎重に）
 kafka-configs.sh --bootstrap-server <broker:9092> --entity-type brokers --entity-name 0 \
  --alter --add-config unclean.leader.election.enable=false

# トピック単位の上書き
 kafka-configs.sh --bootstrap-server <broker:9092> --alter --topic <topic> \
  --add-config unclean.leader.election.enable=true

# 期待するmin.insync.replicasと合わせて設定
 kafka-topics.sh --bootstrap-server <broker:9092> --alter --topic <topic> \
  --config min.insync.replicas=2

Monitoring and SLA Operation: Metrics and Drills

Whether or not ULE is enabled, what matters is having metrics and logs in place to grasp immediately what happened during a failure. In particular, monitor offline partition count, ISR shrinks/expands, and latency/error rates.

During drills, perform a planned broker shutdown in the staging environment and verify producer/consumer success/failure patterns and the occurrence (or not) of log truncation under both ULE-enabled and ULE-disabled settings.

• Key metric (example JMX name): kafka.controller:name=OfflinePartitionsCount, type=KafkaController
• ISR fluctuations: kafka.server:type=ReplicaManager,name=IsrShrinksPerSec / IsrExpandsPerSec
• Produce failures: kafka.producer:type=producer-metrics,record-error-rate / record-retry-rate
• Log audit: keywords Truncating, Leader epoch, HW

Simple drill recipe (staging only)

# 準備：テストトピック（RF=3, minISR=2）
 kafka-topics.sh --bootstrap-server <broker:9092> --create --topic t_ule_test \
  --partitions 3 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config unclean.leader.election.enable=<true|false>

# プロデュース（acks=all）
 kcat -b <broker:9092> -t t_ule_test -P -X acks=all -l msgs.txt &

# ブローカー2台を順に停止（ISRを枯渇させる）
 systemctl stop kafka@<brokerA>
 systemctl stop kafka@<brokerB>

# 観測
 kafka-topics.sh --bootstrap-server <broker:9092> --describe --topic t_ule_test | grep Leader
# JMX/メトリクス: OfflinePartitionsCount, IsrShrinksPerSec を確認
# ログに Truncating が出るか（ULE=true）/パーティション停止（ULE=false）を確認

CCAAK Exam Preparation Points

On the CCAAK exam, the ULE definition, its default, the conditions for data loss, and the relationship with min.insync.replicas and acks come up frequently. Watch out in particular for questions that target the misconception that 'acks=all is always safe.' When the conditions are not met (ISR shortage), the safe behavior is to fail without ACKing — and enabling ULE may cause history to roll back. Memorize that relationship as a single story.

There are also questions designed to confuse Preferred Leader Election (preferred-leader switching among healthy candidates) with Unclean Leader Election (emergency election from the OSR). The former does not break consistency; the latter may cause data loss.

• ULE default: false. When disabled, the partition stays offline — consistency first.
• ULE enabled: elected from OSR; availability up, possibility of unreplicated data loss.
• acks=all + min.insync.replicas>=2: ACKs that meet the conditions are robust. Failing on unmet conditions is the correct behavior.
• Preferred Leader Election is a different thing (does not break consistency).

CLI snippets worth memorizing

# ULEのトピック上書き
kafka-configs.sh --bootstrap-server <broker:9092> --alter --topic <t> \
  --add-config unclean.leader.election.enable=true

# min.insync.replicas の設定
kafka-topics.sh --bootstrap-server <broker:9092> --alter --topic <t> \
  --config min.insync.replicas=2

# Preferred Leader Election（別概念、整合性は維持）
kafka-preferred-replica-election.sh --bootstrap-server <broker:9092>

Check Your Understanding

Frequently Asked Questions