Kafka Topic and Partition Fundamentals: The Core of Distribution and Scalability

A Topic is a logical name and a Partition is an ordered, immutable log. Scaling and ordering guarantees are both anchored to the Partition.

Availability depends on replication and the ISR. Parallelism is capped by the rule that Consumer parallelism cannot exceed the Partition count.

Basics: Topic / Partition / Replica / Offset

In Kafka a Topic is a category name for events, and a Partition is an ordered, immutable log produced by horizontally splitting the Topic. Each record carries a monotonically increasing offset within its Partition, and ordering guarantees are confined to that Partition.

Availability comes from replication. Each Partition has one Leader and zero or more Followers, with Followers tailing the Leader's log. The set of Replicas keeping up with the Leader is the ISR (In-Sync Replicas). Write durability is governed by the combination of acks and min.insync.replicas (covered in detail later).

• The Partition is the smallest unit of parallelism and scale
• Ordering is only guaranteed within a Partition; there is no total ordering across a Topic
• Offsets are read positions managed by the Consumer (committed to __consumer_offsets)
• A replication factor of at least 3 is recommended in production

Distributing and replicating a Topic (example: partitions=3, RF=3)

Create and inspect a Topic via the CLI (3 partitions, RF=3)

kafka-topics --bootstrap-server broker1:9092 \
  --create --topic orders --partitions 3 --replication-factor 3

kafka-topics --bootstrap-server broker1:9092 \
  --describe --topic orders

Producer: Partitioning and Ordering Guarantees

The Producer uses a partitioner to choose the destination Partition. For keyed records, hashing the key sends the same key to the same Partition, preserving per-key ordering. This is a recurring CCDAK theme.

For records without a key, distribution depends on the client implementation (round-robin or sticky partitioning that improves batching). The exam and design takeaway is: use a key whenever you need ordering. Durability is tuned with acks, and duplicates are suppressed with the idempotent Producer (enable.idempotence=true).

• Same key, same Partition — per-key ordering is preserved
• Keyless records follow the client's strategy (round-robin, sticky, etc.)
• High reliability: combine acks=all with a suitable min.insync.replicas
• Deduplication: enable.idempotence=true prevents duplicate writes on retries
• Use compression.type to improve bandwidth usage and throughput

Send keyed records using the console Producer

kafka-console-producer --bootstrap-server broker1:9092 \
  --topic orders --property parse.key=true --property key.separator=:
# Input examples (key on the left, value on the right)
user-42:{"op":"create","id":1}
user-42:{"op":"update","id":1}
user-7:{"op":"create","id":2}

Consumer Groups and Parallelism

Consumers sharing the same group ID divide Partitions among themselves. Only one Consumer in a Consumer Group can process a given Partition at a time, so the maximum parallelism caps at min(partition count, consumer count).

Rebalancing kicks in when group membership or subscriptions change, briefly halting processing. Tune session timeouts and heartbeats, and adopt stable deployment strategies to avoid excessive rebalances.

• Max parallelism = min(partition count, consumers in the group)
• A Partition is exclusively owned within a Consumer Group (no concurrent processing)
• Commit offsets explicitly (enable.auto.commit=false) for better control
• auto.offset.reset=earliest/latest controls where to start when no offset is committed

Read via a Consumer Group to parallelize across processes

# Terminal 1
kafka-console-consumer --bootstrap-server broker1:9092 \
  --topic orders --group app-readers --from-beginning
# Terminal 2 (same group; Partitions are split between consumers)
kafka-console-consumer --bootstrap-server broker1:9092 \
  --topic orders --group app-readers

Design Guide: How to Choose the Partition Count

Estimate based on target throughput, the throughput a single Partition can sustain, and the Consumer parallelism you need. As a rule of thumb, provision enough to meet current demand plus headroom, and add more later if necessary (since you cannot reduce the count, choose the initial value carefully).

More Partitions mean more metadata, more files, more open file descriptors, and more network load. Workloads that need total ordering must stick to a single Partition at the cost of throughput. Make the trade-off explicit per use case.

• One Partition = one parallel slot (from a Consumer Group's perspective)
• You can add Partitions but not remove them (you would need a new Topic and migration)
• Too many Partitions raise Broker/controller load and operational complexity
• Consider a single Partition when strict total ordering is required

Replication and Availability: acks and min.insync.replicas

Write durability is controlled by acks. acks=all means the response is sent only after the record is replicated to the Leader and the required number of Replicas in the ISR. min.insync.replicas is the lower bound on the number of in-sync Replicas needed to respond when acks=all is set.

The practical recommendation for critical Topics is replication.factor=3, min.insync.replicas=2, and acks=all on the Producer. This keeps writes flowing even if one Broker fails while minimizing data-loss risk. Monitor Under-Replicated and Offline Partitions and target zero for both as cluster-health signals.

• RF >= 3 and min.insync.replicas >= 2 for critical data
• Pair with acks=all to ensure durability
• Disable unclean leader election in production
• If Replicas keep dropping out of the ISR, inspect network, disk, and load

Example durability settings for the Topic and Producer

# Set min.insync.replicas when creating the Topic
kafka-topics --bootstrap-server broker1:9092 \
  --create --topic critical-events --partitions 6 --replication-factor 3 \
  --config min.insync.replicas=2

# Producer (example properties)
acks=all
enable.idempotence=true
retries=2147483647
max.in.flight.requests.per.connection=5

Operations and Tuning: Lifecycle and Rebalance Caveats

You can grow the Partition count but cannot shrink it. After increasing it, an existing key may map to a different Partition under the new count, so the same-key-same-Partition property does not span the change boundary (the in-Partition ordering guarantee itself still holds).

Retention is controlled by retention.ms and retention.bytes, and the cleanup strategy by cleanup.policy=delete or compact. Compaction is ideal when you want to keep only the latest value per key; when you need a full event history use the delete policy with a sufficiently long retention. Reassignments and maintenance move Replicas around and consume I/O, so plan them within windows and roll them out in stages.

• You can grow but not shrink — to shrink, create a new Topic and migrate
• Reassignment: use kafka-reassign-partitions with proper planning and verification
• Monitoring: under-replicated partitions, offlinePartitionsCount, and Consumer lag
• Pick retention and compaction settings per use case (log retention vs. keeping only the latest KV)

Common operational commands

# Increase the Partition count
kafka-topics --bootstrap-server broker1:9092 \
  --alter --topic orders --partitions 12

# Adjust Topic retention
kafka-configs --bootstrap-server broker1:9092 --entity-type topics \
  --entity-name orders --alter --add-config retention.ms=604800000

# Start a reassignment (example; the plan is provided as JSON)
kafka-reassign-partitions --bootstrap-server broker1:9092 \
  --execute --reassignment-json-file plan.json

Check Your Understanding

Frequently Asked Questions