Kafka cleanup.policy: delete vs compact vs compact,delete - Choosing and Operating Each

Kafka's cleanup.policy decides whether log segments (one per partition) are deleted by time and size, compacted per-key, or both at once.

This article covers when to choose delete, compact, or compact,delete, the field pitfalls that bite real operators, how the main parameters interact, and the angles the CCAAK exam tends to focus on.

cleanup.policy Basics and Internal Behavior

cleanup.policy is a per-topic setting. delete drops whole segments once they hit a time (retention.ms) or size (retention.bytes) threshold. compact keeps only the latest value per key, thinning out duplicate updates. compact,delete applies both, holding only the latest value per key while still expiring old segments by time or size to cap disk usage.

delete never touches the active segment (the one currently being written). compact runs in the background via the broker's Log Cleaner (log.cleaner.enable, log.cleaner.threads), removing duplicate keys from segments that meet conditions such as min.cleanable.dirty.ratio. Tombstones (records with a null value) act as delete markers and are retained for delete.retention.ms.

retention.ms and retention.bytes are evaluated independently; the oldest segments are removed as soon as either threshold is crossed. The key point with compact,delete is that compaction keeps each key current while old segments themselves still disappear once the time or size cap is reached.

• delete keeps the full history and then drops it wholesale. compact keeps only the latest version per key (null-keyed records are excluded).
• Prerequisites for compact: keyed messages, log.cleaner.enable=true, and an appropriate delete.retention.ms.
• Timestamp-based retention depends on log.message.timestamp.type (CreateTime or LogAppendTime).
• Use retention.ms=-1 or retention.bytes=-1 for unlimited retention.

How segment deletion and compaction relate (conceptual view of a single partition)

Operating delete (Time/Size-Based Deletion)

delete is the most intuitive retention strategy: a whole segment is dropped once it exceeds a time or size threshold. Because the active segment is never deleted, tightening segment.ms or segment.bytes to roll segments more often gives you a clearer retention boundary.

Time-based retention depends on the message timestamp type. CreateTime uses the time the producer stamped, LogAppendTime uses when the broker received it. For workloads with late arrivals or clock skew, LogAppendTime keeps retention behavior more predictable.

• To make retention bite faster, shorten segment.ms (minutes to tens of minutes) so segments roll more frequently.
• The size cap (retention.bytes) is applied independently of time, protecting disk space during write spikes.
• retention.ms=0 approximates instant deletion, but remember the active segment stays - it is not truly immediate.
• Size your disks as partitions x replication factor x expected retention size, plus index overhead.

Operating compact (Per-Key Latest Value)

With compact, older records for the same key are thinned out, leaving only the latest version. Deletes are expressed as tombstones (null values), which are kept for delete.retention.ms so downstream consumers have a chance to observe the deletion.

Compaction is asynchronous; segments become candidates once min.cleanable.dirty.ratio (the dirty ratio since the last clean) is exceeded. min.compaction.lag.ms lets you carve out a grace window so freshly written data is not compacted immediately. On high-throughput clusters, watch the log.cleaner.threads parallelism and I/O bandwidth.

• Keys are mandatory. Records with null keys are not eligible for compaction.
• Keep delete.retention.ms long enough. Too short and a new consumer can miss a delete, leading to inconsistency.
• Record order is preserved, but since old duplicates vanish, consumers must be written to tolerate gaps in offsets.
• Large messages or oversized keys hurt compaction efficiency. Tune segment.bytes and consider compression.type as well.

compact,delete Combined Patterns and Pitfalls

compact,delete shines when you want to keep the latest snapshot but still cap overall log volume. Compaction keeps only the latest value per key, and old segments themselves are still purged via retention.ms/bytes to bound disk usage.

The catch is that compaction does not guarantee permanent retention. Once retention.ms passes, segments that still hold snapshots you assumed were safe can be dropped. If retention.ms is shorter than your new-consumer bootstrap or downstream reprocessing window, full initialization may fail.

• Recommended relationship: retention.ms should be at least as long as delete.retention.ms, so tombstones have time to propagate.
• Fits scenarios where the latest snapshot matters, the full historical log does not, and disk usage must stay bounded.
• It shows up in some Kafka Streams internal topics (for example certain state and windowing topics), but always confirm per workload.
• If you have replication or snapshot-restore requirements, either widen retention.ms or offload snapshots to external storage.

Configuration Tuning Guidelines and Examples

Typical combinations: Event logs - cleanup.policy=delete with retention.ms and retention.bytes aligned to your business SLA and disk capacity. Changelogs where the latest snapshot matters - cleanup.policy=compact with delete.retention.ms long enough to cover downstream ingestion lag. Latest snapshot plus disk cap - cleanup.policy=compact,delete with retention.ms set longer than your initial-sync or reprocessing window.

For capacity planning, estimate per-partition throughput x retention time to get rough segment volume, multiply by partition count and replication factor, then add 10-20% for index overhead. Smaller segment.bytes makes retention boundaries reflect more quickly but increases file count and metadata pressure.

• Rough disk formula: required capacity = write rate (B/s) x retention seconds x partitions x replication factor x 1.2.
• Optimize for time-based retention with a shorter segment.ms; optimize for size-based retention with a smaller segment.bytes - both speed up segment roll.
• If compaction load feels heavy, nudge min.cleanable.dirty.ratio up and scale out log.cleaner.threads or I/O bandwidth.

Representative configuration commands (script names and options vary by version)

# 1) Event log (delete): delete once 7 days OR 200 GiB is reached
kafka-topics.sh --create \
  --topic app-events \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=delete \
  --config retention.ms=$((7*24*60*60*1000)) \
  --config retention.bytes=$((200*1024*1024*1024)) \
  --config segment.ms=$((15*60*1000))

# 2) Changelog (compact): keep tombstones for 72 hours
kafka-topics.sh --create \
  --topic user-profile-changelog \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config delete.retention.ms=$((72*60*60*1000)) \
  --config min.cleanable.dirty.ratio=0.5 \
  --config min.compaction.lag.ms=$((10*60*1000))

# 3) Latest snapshot + size cap (compact,delete): keep latest, drop old segments after 14 days
kafka-topics.sh --create \
  --topic account-state \
  --partitions 24 --replication-factor 3 \
  --config cleanup.policy=compact,delete \
  --config retention.ms=$((14*24*60*60*1000)) \
  --config delete.retention.ms=$((72*60*60*1000)) \
  --config segment.bytes=$((256*1024*1024))

# 4) Altering an existing topic
kafka-configs.sh --alter --entity-type topics --entity-name app-events \
  --add-config retention.ms=$((10*24*60*60*1000))

# 5) Sending a tombstone (key:value format; omitting the value means null = delete marker)
# Note: exact syntax varies by tool and client
kafka-console-producer.sh --broker-list localhost:9092 \
  --topic user-profile-changelog \
  --property parse.key=true --property key.separator=:
user-123:

Monitoring, Troubleshooting, and CCAAK Exam Tips

On the monitoring side, track disk usage, log size per partition, segment count, and Log Cleaner health (thread backlog, processing rate). After a retention policy change, verify via broker logs and metrics that segments are rolling and being deleted as expected.

When things go wrong, re-check the topic's effective configuration (broker defaults plus topic overrides) and untangle how retention.ms, retention.bytes, segment.ms, delete.retention.ms, and min.cleanable.dirty.ratio interact. In particular, when initial sync fails under compact,delete, retention.ms is almost always too short.

• Exam takeaway: delete works at segment granularity, compact at key granularity. compact requires a key; null-keyed records are excluded.
• Exam takeaway: tombstones are retained for delete.retention.ms. Under compact,delete, segments still vanish after retention.ms - so even the latest value is not guaranteed to persist forever.
• Exam takeaway: timestamp-based behavior changes depending on whether you are using CreateTime or LogAppendTime.

Test Yourself

Frequently Asked Questions