Event Sourcing with Kafka: Designing Systems Around Events, Not State

Event Sourcing is a design approach where you do not store an entity's current state directly. Instead, the stream of events that produced that state is the single source of truth. Kafka excels at appending, distributing, and replaying events, and it ships with the features Event Sourcing needs: per-partition ordering, flexible retention strategies, and transactions.

This article walks through the design considerations along the Confluent Certified Developer for Apache Kafka (CCDAK) exam domains, with a practical lens. We stick to stable concepts and features and assume the behavior documented in the official Kafka documentation.

Event Sourcing Essentials and How They Map to Kafka

In Event Sourcing, events are the single source of truth and state is treated as a derivable artifact that can be reconstructed at any time. Kafka's append-only topic log, in-partition ordering guarantees, and retention and compaction policies let you keep events long-term and replay them as needed.

Topics CCDAK loves to ask about include the difference between retention (delete) and log compaction (compact), in-partition ordering and key design, the meaning of transactional and idempotent producers, and the consumer isolation.level setting.

• Record events as immutable data; rebuild state later by replaying them
• Route to partitions by key to guarantee ordering within a key
• Mix retention (delete) and compaction (compact) to keep both history and the latest view
• Use patterns like the Outbox to draw clean boundaries with external databases

Topic, Key, and Retention Design: Setting Ordering and Lifespan

Ordering is guaranteed only within a partition. The default move is to pick a business aggregate (such as accountId or orderId) as the key. Partition count drives throughput and parallelism, so size it with the cardinality and distribution of your keys in mind.

Retention (delete) prunes the log by time or size. Compaction (compact) keeps only the latest record per key and removes prior records for that key. Using both together (cleanup.policy=compact,delete) preserves the latest view while still letting you sweep out extremely old segments.

• Pick the key as the unit that needs ordering guarantees
• Use cleanup.policy=compact to back a latest-view KTable
• Use retention.ms / retention.bytes to control the lifespan and cost of history
• Be careful when changing partition counts — rebalancing affects ordering because key-to-partition mapping can change

Duplicates and Consistency: Idempotent and Transactional Producers, and EOS

Kafka's idempotent producer prevents duplicate writes even when the network retries. Set transactional.id on top of that, and you can write to multiple topics and partitions atomically; consumers running with isolation.level=read_committed will then read only committed records. Together, these give you exactly-once semantics for processing that lives entirely inside Kafka.

Strict end-to-end EOS that extends to external systems such as databases sits outside Kafka's guarantees. The pragmatic approach for external integrations is to lean on patterns like the Outbox or a two-step pipeline and settle for at-least-once delivery plus idempotent processing.

• Think of enable.idempotence=true and acks=all as a set
• A unique transactional.id plus fencing prevents double commits
• Use isolation.level=read_committed on consumers to skip uncommitted records
• Kafka Streams EOS uses the above under the hood (version-specific details follow the official documentation)

Minimal Transactional Producer configuration (Java)

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.ByteArraySerializer.class.getName());
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5"); // 再順序化を避ける上限
props.put(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "orders-tx-1");

KafkaProducer<String, byte[]> producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
  producer.beginTransaction();
  // バッチでイベントを書き込む
  for (Event e : events) {
    ProducerRecord<String, byte[]> rec = new ProducerRecord<>("orders-events", e.key(), e.payload());
    producer.send(rec);
  }
  producer.commitTransaction();
} catch (org.apache.kafka.common.errors.ProducerFencedException fenced) {
  // 同一 transactional.id の別インスタンス起動などでフェンスされた
  producer.close();
  throw fenced;
} catch (Exception ex) {
  producer.abortTransaction();
}

Schema Management and Evolution: Keeping Events Compatible

Because events are replayed over long horizons, you have to manage schema evolution with backward and forward compatibility in mind. Use Confluent Schema Registry and set the right compatibility level. Adding fields is usually backward-compatible, while removing required fields or changing types tends to break compatibility.

Your subject naming strategy (TopicNameStrategy, RecordNameStrategy, and so on) changes the granularity and sharing of schemas. For Event Sourcing, keep a clean schema per event type and have the application side carry an upcaster — a routine that lifts old events into the latest shape — as a safety net.

• Default to backward compatibility so old events can always be replayed
• Absorb evolution by loosening required fields to optional and supplying defaults
• Subject naming strategy is a trade-off between reuse and collision avoidance
• When a breaking change is unavoidable, model it as a brand-new event type

Replay and Materialization: Deriving the Current State via KTable and State Stores

Ingest events as a KStream, aggregate by key, and materialize them into a KTable to get a local or remote store of the latest state. Compacted topics are a natural fit for KTables, and you can represent deletes with tombstones.

When you need a replay, reset the consumer offset to the beginning (for example, with auto.offset.reset=earliest or a manual seek) or read with a fresh consumer group. To shorten replay time, it is also effective to periodically publish a state-snapshot topic so you can resume from a snapshot.

• Build the current value via KStream → aggregate → KTable
• Compacted topics are ideal for a per-key latest view
• Combine snapshots with event deltas for fast recovery
• Offset management is how you spell out which point-in-time view you want

Event Sourcing data flow (conceptual diagram)

Operational Tips and CCDAK Hot Spots

In production, watch latency and lag, retention size, compaction progress, tombstone backlog, schema compatibility violations, and the impact of rebalances. Because Event Sourcing assumes data cannot simply be deleted, cost estimation and an archive or snapshot strategy are critical.

For exam prep, expect frequent questions on retention vs. compaction, keys and ordering, idempotent and transactional producer settings, read_committed, the difference between KTable and KStream, schema compatibility levels, and the basics of ACLs and security.

• Monitor consumer lag, log size, compaction lag, and error rates
• Isolate broken events in a dead-letter topic (DLT) and reprocess them separately
• Grant the right ACLs to transactional IDs
• Reprocessing strategy: full replay with a fresh group, or apply deltas starting from a snapshot

Check Your Understanding

Frequently Asked Questions