Kafka Compression Types Compared: gzip / snappy / lz4 / zstd Selection Guide

Kafka compression is fundamentally applied per Producer record batch, and the Broker stores and replicates those batches as-is. The exception: if the topic-level compression.type is set to a concrete algorithm, the Broker may re-compress.

This article walks through the characteristics of each algorithm, configuration patterns, and monitoring/validation methods, and wraps up with the points that come up most often on CCDAK/CCAAK.

Kafka Compression Basics and Data Flow

In Kafka, the Producer batches records and compresses each batch according to compression.type. The Broker writes those compressed batches to the log almost verbatim, and replication carries the same compressed bytes. On the Consumer side, the client library decompresses on fetch.

When the topic's compression.type is set to producer, batches are stored using whatever algorithm the Producer chose. When it is set to a concrete value like gzip/snappy/lz4/zstd, the Broker re-compresses on receive, raising CPU load and latency. Compression operates on record batches, so larger batches generally improve compression efficiency.

Size limits apply to the compressed size. Specifically, the Producer's max.request.size and the Broker/topic's message.max.bytes (or max.message.bytes) are evaluated against the compressed record batch size.

• Producer config: compression.type=none|gzip|snappy|lz4|zstd (often defaults to none)
• Topic config: compression.type=producer|uncompressed|gzip|snappy|lz4|zstd
• Re-compression happens only when the topic specifies a concrete algorithm
• Batching impact: raising batch.size and linger.ms tends to improve the compression ratio at the cost of latency
• Replication uses the already-compressed bytes (saving inter-broker bandwidth)

Producer compression and Broker storage flow (branching by topic config)

Producer
   |
   | 1) Batch + compress (compression.type)
   v
[Compressed record batch]
   |
   |--> If topic compression.type = producer ----------
   |                                                |
   v                                                v
Send to Broker  ------------------------------>  Store in log as-is
   |
   |--> If topic compression.type = gzip/lz4/... ----
                                                    |
                                                    v
                                              Broker re-compresses
                                                    |
                                                    v
                                              Store in log
                                                    |
                                                    v
                                          Replication (still compressed)
                                                    |
                                                    v
                                                Consumer decompresses

Main compression types (Producer compression.type)

none
gzip
snappy
lz4
zstd

Per-Algorithm Characteristics and Data Fit

Text and repetitive JSON/Avro compress well. Binary data or already-compressed data (images, video, ZIP) sees little benefit, or can even grow. With small batches, every algorithm produces only limited gains.

For low latency and high throughput, lz4 or snappy is the safe bet. If network bandwidth is the bottleneck and you have CPU headroom, zstd wins. gzip compresses hard but is slow, so save it for latency-tolerant or overnight-batch workloads.

In mixed environments, either keep topic compression.type=producer and let each Producer optimize for itself, or — if you want a uniform algorithm — align every Producer to the same setting. Pinning it at the topic level forces Broker re-compression.

• Small or sparse messages: minimize overhead with snappy/lz4
• Large or text-heavy messages: prioritize bandwidth with zstd/gzip
• Bursty workloads: keep CPU headroom with snappy/lz4
• If you can standardize across all Producers, leaving the topic on producer is the safest choice

Basic Java Producer configuration example

Properties p = new Properties();
p.put("bootstrap.servers", "broker1:9092,broker2:9092");
p.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
p.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
// Choose the compression algorithm
p.put("compression.type", "lz4");
// Batch / latency trade-off
p.put("batch.size", "131072");      // ~128KB target
p.put("linger.ms", "10");            // wait 10ms to batch
KafkaProducer<String, byte[]> producer = new KafkaProducer<>(p);

Configuration Patterns: Producer-First or Topic-Pinned

As a rule of thumb, decide the algorithm at the Producer and leave the topic's compression.type as producer. That avoids Broker re-compression and saves both CPU and latency.

Only when uniformity across Producers is impractical — and you need it for audit or bandwidth-control reasons — pin a concrete algorithm (gzip/snappy/lz4/zstd) on the topic. Expect higher Broker CPU and latency, and plan capacity and slow-client impact accordingly.

Even with TLS or SASL in play, compression is applied before encryption, so the bandwidth savings remain intact.

• Recommended default: topic compression.type=producer, with each Producer set to snappy or lz4
• Bandwidth-constrained: pilot zstd on the Producer (verify client compatibility and CPU)
• Operational mandate to standardize: pin at the topic, but plan for re-compression cost
• Improve ratios via batching: tune batch.size and linger.ms

Concrete topic config examples (Broker CLI)

# Switch an existing topic to Producer-first (avoid re-compression)
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter --add-config compression.type=producer

# Standardize on lz4 at the topic (forces re-compression)
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter --add-config compression.type=lz4

# Create a new topic with compression.type=producer
kafka-topics --bootstrap-server localhost:9092 \
  --create --topic events --partitions 12 --replication-factor 3 \
  --config compression.type=producer

Operations, Monitoring, and Validation: Reading Throughput and Latency

Compression costs CPU and latency. When you roll it out, use load-test tools and metrics to watch CPU usage, BytesIn/Out, and request latency (RequestQueueTime/LocalTime/RemoteTime). Where re-compression is in play, expect a sharp jump in Broker CPU.

On multi-tenant clusters, pair heavy compression (gzip/zstd) with throughput control, quotas, and batch tuning so peaks from one Producer don't bleed into other workloads.

Consumer-side decompression also burns CPU, so monitor consumer-side latency, GC, and throughput regressions alongside the Broker.

• Broker metrics: kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec/BytesOutPerSec
• Producer metrics: record-send-rate, request-latency-avg, compression-rate-avg
• Consumer metrics: records-consumed-rate, fetch-latency-avg, lag
• Run load tests for both steady-state and peak conditions (verify the latency SLA)

Performance validation example (kafka-producer-perf-test)

# Send with lz4 (1KB messages, 1M records)
kafka-producer-perf-test --topic perf --num-records 1000000 \
  --record-size 1024 --throughput -1 --producer-props \
  bootstrap.servers=localhost:9092 compression.type=lz4 batch.size=131072 linger.ms=10

# Send with zstd (bandwidth-first)
kafka-producer-perf-test --topic perf --num-records 1000000 \
  --record-size 1024 --throughput -1 --producer-props \
  bootstrap.servers=localhost:9092 compression.type=zstd batch.size=131072 linger.ms=20

Selection Guide and Exam Tips (CCDAK/CCAAK)

The core of real-world selection is being clear about which dimension is your bottleneck: latency SLA, CPU, or bandwidth. Use the quick guide below as a starting point and then validate against a representative internal workload.

On the exams, the recurring themes are: where compression is applied (Producer-centric), using topic compression.type=producer to avoid re-compression, the fact that replication carries the compressed bytes, and how batching affects compression efficiency. When to pick zstd vs lz4 also comes up.

• Low latency / high TPS: lz4 or snappy
• Bandwidth-first: zstd (verify client support and CPU headroom)
• Legacy or broad compatibility: gzip
• Avoid Broker re-compression by default: topic compression.type=producer
• Maximize compression: tune batch.size + linger.ms while honoring the latency SLA
• Remember that size limits are evaluated on the compressed size

Compression algorithm cheat sheet

snappy: fast / medium ratio / low CPU      -> low latency
lz4:    even faster / medium-high ratio / low-medium CPU -> default pick
zstd:   high ratio / medium-high CPU         -> bandwidth-first
gzip:   high ratio / high CPU / slow         -> batch / latency-tolerant

Topic:    compression.type=producer (avoid re-compression)
Producer: compression.type=<one of the above>
Batching: batch.size up + linger.ms up -> ratio up (latency up)

Check Your Understanding

Frequently Asked Questions