Kafka Controller Broker Deep Dive: Roles and Metadata Management

The controller broker oversees cluster-wide metadata and state transitions — topic/partition leader election, broker liveness monitoring, reassignments, and more. Clients never connect to it directly, but it is the key to stable operations and to passing the exam.

This article covers both ZooKeeper-based and KRaft (Kafka Raft) based modes, organizing the points that trip people up in the field with diagrams, comparison tables, and command examples. The target level is CCAAK-equivalent administrator.

Controller Basics and Election (ZooKeeper / KRaft)

The controller broker (henceforth, the controller) is the cluster's single active administrator. In ZooKeeper mode, one regular broker is elected via a ZooKeeper ephemeral ZNode. In KRaft mode, the leader is chosen from a controller quorum (Raft) that is independent of brokers, and that leader acts as the "active controller". When the controller fails, failover happens automatically.

One caveat: a partition leader and the cluster controller are different roles. Each partition has one leader replica, but the cluster-wide leader (= the controller) is always exactly one. The controller finalizes and distributes metadata; clients normally connect only to brokers.

• ZooKeeper mode: one broker is elected via ZK. Events are distributed via RPCs such as LeaderAndIsr/UpdateMetadata
• KRaft mode: the Raft leader of the controller quorum becomes active. Metadata is committed and replicated as a metadata log
• Even a short controller outage affects cluster operations (e.g. new topic creation, leader changes)

Controller positioning in ZooKeeper vs. KRaft

Metadata Scope and Lifecycle

Kafka "metadata" includes the cluster ID and broker registrations, topic/partition layout, ISR (In-Sync Replicas), assignments, ACLs, and various configs (topic/quota/user/broker). The controller finalizes these primarily and reflects them into each broker's metadata cache.

In ZooKeeper mode, changes are written to ZooKeeper, and the active controller distributes them to each broker via RPCs like LeaderAndIsr/UpdateMetadata. In KRaft mode, changes are committed to the controller quorum's metadata log, and brokers pull snapshots/deltas from the controller to update their caches. Clients refer to the metadata cache of the broker they connected to and use it to route requests (e.g. locate the partition leader).

• Metadata aims for near-strong consistency: the controller is the single source of truth
• Clients never query the controller directly (they go through the bootstrap broker)
• ISR changes and broker comings/goings are triggered by the controller, which re-elects leaders

ZooKeeper Mode vs. KRaft Mode (Comparison Table)

During the migration period, both architectures coexist. Even though the names look similar, their behavior and operational concerns differ — for the exam, pin down the mapping between terminology and responsibility precisely. "Where does election happen, and where is metadata persisted?" is a frequent question.

In KRaft, you can either co-locate the controller with brokers (process.roles=broker,controller) or split them onto dedicated nodes (process.roles=controller). For large clusters, separation is usually recommended.

• Election venue: ZooKeeper uses ZNodes, KRaft uses Raft
• Propagation: ZooKeeper distributes via RPC; KRaft commits to the metadata log first and brokers pull it
• Operations: KRaft does not need ZK; startup order and config keys change

Controller Responsibilities and Key Flows

The controller handles partition leader/ISR management, broker registration and heartbeat monitoring, leader reassignment on failure or planned shutdown, finalizing assignments for topic create/delete, executing reassignments and preferred-leader election, and committing metadata changes for ACLs, quotas, and the like.

It runs event-driven. In ZooKeeper mode, ControllerEventManager processes events in order; in KRaft, the commit order on the metadata log guarantees overall consistency. On inconsistencies, recovery happens via snapshots or full fetch.

• Failure detection: a missing broker heartbeat or session disconnect marks the broker offline
• Leader re-election: safely elected based on ISR; falls back to Unclean leader election if configured
• Planned shutdown: Controlled shutdown hands leadership over to other replicas in advance
• Reassignment: incremental movement with throughput throttling
• Preferred leader: Preferred leader election restores the originally assigned preferred leader

Operations: Failover, Monitoring, and Troubleshooting

Failover is automatic, but how fast detection and recovery happen depends on timeouts and network health. In ZooKeeper mode, the session timeout matters; in KRaft, the Raft heartbeat / election timeout settings matter. In real-world ops, you tune to your network so that "too-fast" switchover does not cause churn.

The monitoring fundamentals: active controller metrics, the metadata processing queue, leader movement count, ISR stability, controller channel error/latency, and metadata propagation latency. In logs, focus on ControllerEventManager (ZooKeeper mode) and QuorumController (KRaft).

• Active controller check: in ZK look at /controller; in KRaft check the metadata quorum leader
• Propagation latency: high UpdateMetadata or metadata-log apply latency increases client routing failures
• Planned maintenance: controlled shutdown for leadership handoff; throttled reassignment for redistribution

Useful operational commands (ZooKeeper / KRaft)

# ZooKeeperモード: アクティブコントローラーを確認
bin/zookeeper-shell.sh zk1:2181 get /controller
# 出力のbrokeridがアクティブコントローラー

# ZooKeeperモード: ブローカーの生存（エフェメラルノード）
bin/zookeeper-shell.sh zk1:2181 ls /brokers/ids

# KRaftモード: コントローラー・クォーラムの状態
bin/kafka-metadata-quorum.sh describe --bootstrap-server controller1:9093 --status
# Leader(=Active Controller) とVoterの健全性を確認

# KRaftモード: クォーラム内の全履歴の短縮表示
bin/kafka-metadata-quorum.sh describe --bootstrap-server controller1:9093 --log-snapshot

# 参考: パーティション配置とリーダー（どのブローカーが担当か）
bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic my-topic

# KRaftサンプル設定（同居構成）
# server.properties（一例）
process.roles=broker,controller
node.id=1
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
inter.broker.listener.name=PLAINTEXT
controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093

CCAAK Exam Tips: Avoiding Trick Questions

The exam frequently mixes up terminology — controller vs. partition leader vs. group coordinator. Clients do not connect to the controller directly. Differences in election, persistence, and propagation between ZooKeeper and KRaft are also commonly tested.

Make sure you understand operational concerns too: safe leader election (ISR-based), the risk of Unclean leader election, the effect of controlled shutdown, throttling during reassignment, and the role and limitations of the metadata cache (temporary routing failures due to propagation lag).

• There is one (active) controller per cluster, but a partition leader exists for every partition
• Clients look at the broker's metadata cache (Describe/Metadata API)
• ZooKeeper: check state via /controller and /brokers/ids. KRaft: check via metadata quorum commands
• Baseline: "existing read/write traffic mostly continues even if the controller node goes down, but administrative operations are affected"

Check with a Sample Question

Frequently Asked Questions