Designing Kafka Replicas and In-Sync Replicas: Fault Tolerance and Consistency

Kafka's fault tolerance rests on replication and ISR (In-Sync Replicas). Simply increasing the replication factor is not enough to prevent data loss. The combination of acks and min.insync.replicas, plus how you set unclean.leader.election.enable, determines consistency.

This article keeps certifications like CCAAK in mind while organizing design points that often confuse teams in the field, following the behavior in the official documentation. We specifically cover safe configurations assuming RF=3, ISR shrink/expand, the meaning of HW (High Watermark) and commits, and availability trade-offs during failover.

Replica and ISR Fundamentals

Each Kafka partition consists of one leader and multiple followers, with the total replica count determined by replication.factor (RF). Followers fetch from the leader in order and replicate the log in the same sequence.

ISR is "the set of replicas sufficiently in sync with the leader." Followers that fail to catch up within a configured time threshold are removed from the ISR, which shrinks and expands automatically. Producer acks=all decides commit eligibility based on the ISR.

• Leader: the sole replica responsible for accepting writes and finalizing order
• Follower: fetches from the leader and builds the identical log
• ISR: the set of replicas meeting the catch-up requirement. Removed when lag exceeds the threshold, re-added once caught up.
• Baseline guidance: use RF=3 or higher in production, with acks=all and min.insync.replicas combined for safety

Relationship between Producer, Leader, Followers, and ISR

Topic creation and minimum ISR configuration example

kafka-topics --bootstrap-server broker1:9092 \
  --create --topic orders --partitions 6 --replication-factor 3

# Set per-topic min ISR=2 (assumes RF=3)
kafka-configs --bootstrap-server broker1:9092 \
  --alter --topic orders --add-config min.insync.replicas=2

Conditions for ISR Formation, Shrink, and Expand

Followers periodically fetch data from the leader. If a follower cannot fetch or catch up for longer than a configured time, it is removed from the ISR. Once caught up again, it is automatically re-added.

ISR also acts as a safety valve for throughput control. By removing replicas that can't keep up, you avoid excessive delays waiting for replication before commit. However, when the ISR shrinks below min.insync.replicas, acks=all writes fail.

• Main causes of removal: network disconnects, broker overload, storage latency
• Configuration guideline: set replica.lag.time.max.ms to match your environment's round-trip latency and throughput
• Monitoring metrics: UnderReplicatedPartitions, IsrShrinksPerSec / IsrExpandsPerSec
• Behavior: when ISR < min.insync.replicas, returns NotEnoughReplicas or NotEnoughReplicasAfterAppend

Checking ISR (the Describe output shows the ISR)

kafka-topics --bootstrap-server broker1:9092 --describe --topic orders
# Sample output excerpt
# Partition: 0	Leader: 2	Replicas: 2,1,3	ISR: 2,3

The Meaning of High Watermark and Commit

Kafka commits are based on the High Watermark (HW). HW is the minimum LEO (Log End Offset) across replicas in the ISR; offsets below the HW are considered replicated to all ISR members.

An acks=all producer response is returned when the record is committed and the HW advances to a position that includes that record. If the ISR is slow, it waits temporarily; if the ISR shrinks and reorganizes, it can proceed.

• HW depends on the ISR. The larger the ISR, the stricter the commit requirement.
• ISR shrink can let commits resume, but the floor is min.insync.replicas
• read_committed consumers only see committed records (when transactions are used)
• Even without transactions, the HW acts as a visibility boundary

Example producer configuration favoring consistency

bootstrap.servers=broker1:9092,broker2:9092,broker3:9092
acks=all
enable.idempotence=true
retries=2147483647
max.in.flight.requests.per.connection=1
request.timeout.ms=30000
delivery.timeout.ms=120000

Design Guidance for acks and min.insync.replicas

acks is the policy that controls at which stage the leader responds, and min.insync.replicas is the lower bound on the ISR size required to commit. The common safe configuration for RF=3 is "acks=all + min.insync.replicas=2." This lets new writes safely continue with one broker down, and stops writes when a second broker is lost to avoid data loss.

If you prioritize low latency, acks=1 is an option, but a sole leader failure can lose the most recent writes. For batch jobs and important data, use acks=all together with min.insync.replicas as the default.

• With RF=3, acks=all, and minISR=2, you continue without loss when one broker fails
• When the ISR shrinks to 1, acks=all sends fail and the producer falls back to retry/wait
• Lowering minISR to prioritize availability raises the risk of loss on failure

Per-topic acks operations and configuration verification flow

# Producer side (e.g., app config)
acks=all

# Set the topic-side min ISR to 2 (assumes RF=3)
kafka-configs --bootstrap-server broker1:9092 \
  --alter --topic payments --add-config min.insync.replicas=2

# Verify the configuration
kafka-configs --bootstrap-server broker1:9092 \
  --describe --topic payments | grep min.insync.replicas

Clean / Unclean Leader Election and Data Loss

In a healthy failover, the new leader is elected only from replicas within the ISR. This is clean leader election, which preserves order and integrity. On the other hand, if you allow unclean.leader.election.enable, a lagging replica outside the ISR can become leader, and unreplicated writes get rolled back, causing data loss.

If you want strong consistency, use unclean.leader.election.enable=false. In that case, if the ISR is completely lost, the partition temporarily becomes unwritable/unreadable, but no data is lost.

• Recommended: unclean.leader.election.enable=false
• RF=3 or higher makes it easier to balance availability and consistency
• Avoid RF=2 because partition scenarios increase the number of dilemmas
• When stopping or doing rolling operations, confirm followers are caught up before proceeding

Disabling unclean leader election per topic

kafka-configs --bootstrap-server broker1:9092 \
  --alter --topic orders --add-config unclean.leader.election.enable=false

# Verify
diff <(kafka-configs --bootstrap-server broker1:9092 --describe --topic orders | sort) /dev/null | cat

Operational Essentials: Monitoring, Testing, and Common Pitfalls

Continuous monitoring of the ISR and replica lag is essential. Watch whether UnderReplicatedPartitions stays at 0, the rate of IsrShrinks/Expands, and network and disk wait times. On the producer side, you need retry strategies and alerting for NotEnoughReplicas-class errors.

Before going to production, run fault-injection tests repeatedly to confirm: acks=all + min.insync.replicas=2 keeps running with one broker down, writes stop with two brokers down, and order and integrity are preserved after recovery.

• Monitoring: UnderReplicatedPartitions, OfflinePartitionsCount, IsrShrinks/Expands, network round-trip latency
• Producer: properly tune retries, backoff, timeouts, and idempotence
• During rolling restarts, wait for the ISR to stabilize before restarting the next broker
• Chronic slow followers point to hardware or GC/IO bottlenecks

Example health checks via CLI

# Detect under-replicated partitions
kafka-topics --bootstrap-server broker1:9092 --under-replicated-partitions --describe

# List partition leaders / ISR
kafka-topics --bootstrap-server broker1:9092 --describe --topic orders

# For per-broker load, use external monitoring or JMX export

Check Your Understanding

Frequently Asked Questions