Kafka Tiered Storage Hands-On Guide: Decoupling Brokers and External Storage for Scale and Operability

Tiered Storage is a mechanism that keeps hot data on the Kafka broker's local disk while offloading cold data to external object storage (S3/GCS/Azure Blob, etc.). This lets you scale compute (brokers) and storage independently, opening up new options for disaster recovery and cost optimization.

It is offered as Tiered Storage on Confluent Platform / Confluent Cloud, and is increasingly available in Apache Kafka itself via the Remote Log Storage API. On the CCAAK exam, you need to be able to explain that the basic produce/fetch semantics are unchanged, articulate the fundamentals of retention and segments, and accurately describe the operational benefits and constraints.

Why Decouple Brokers from Storage

The primary goal of decoupling is to scale compute resources (CPU/memory/network) and storage capacity independently. Local disks are optimized for serving the hot set, while long-term retention is delegated to object storage. This eliminates the inefficiency of adding broker nodes just to expand disk capacity.

Tiered Storage does not change the write path or ISR/acks semantics. Producers continue to append to the broker's local log as before; after a segment rolls, it is progressively offloaded to external storage. When a consumer reads an older offset, the required portion is fetched back from external storage.

• Independent scaling of capacity and throughput
• Faster recovery (less data resync when replacing brokers)
• Total cost optimization (smaller local SSDs, leveraging object storage)
• Unchanged semantics (no impact on acks/ISR/EOS)

Tiered Storage Architecture

The core components are the broker's local log (active and recent segments), an offloader (which transfers segments to external storage), the external object store, and on-demand reads at fetch time. Offload runs asynchronously after a segment is closed and the local retention conditions are met.

Reads are served from local storage by default. When the requested range is not local, the broker fetches the relevant portion of the target segment from external storage and continues serving the read. This strikes a balance between low latency for the hot working set and reachability of historical data, even when it is cold.

• Write path is unchanged; active segments stay local
• After a segment rolls, the offloader transfers it to external storage
• Older reads are fetched on demand from external storage
• Metadata and indexes are tracked on the broker side (implementation-dependent)

Conceptual diagram of Kafka Tiered Storage

Design and Configuration Essentials (Segments, Retention, Thresholds)

Segment size directly impacts throughput, offload frequency, and recovery time. Too large, and the unit of offload and fetch becomes heavy; too small, and metadata overhead explodes. A starting point of a few hundred MB to 1 GB is a reasonable default.

Think of local and global retention as two separate concepts. Local retention (the hot set) is a capacity plan for serving the most recent N hours/days of reads locally; global retention (object storage) is long-term storage for audit and analytics. Configuration keys vary by vendor and RLM implementation, but foundational keys such as log.segment.bytes and log.retention.ms/bytes remain universally important.

Internal topics (such as __consumer_offsets and __transaction_state) are typically excluded from tiering for low-latency and availability reasons. Limit tiered topics to business data, and set local retention periods based on your SLA.

• Standardize segments (align segment.bytes/segment.ms)
• Derive local retention from your SLA (e.g., 24-72 hours)
• Set global retention based on compliance requirements
• Internal topics should generally stay local-only
• Consult official documentation for vendor-specific enablement, bucket, and IAM configuration

Example server.properties (stable keys only; tiering enablement is vendor-specific)

# Foundational segment and retention settings (stable Kafka keys)
log.segment.bytes=1073741824        # 1 GiB
log.segment.ms=0                     # Disable time-based roll (set if needed)
log.retention.ms=604800000           # 7 days (example)
log.retention.bytes=-1               # Example without a capacity cap

# Keep internal topics short or local-first (operational policy: exclude from tiering)
# Example: keep __consumer_offsets retention short
offsets.retention.minutes=10080      # 7 days (adjust to your requirements)

# Tiered storage enablement and external storage settings vary by product
# Check the official documentation for Confluent Platform/Cloud or your Apache Kafka RLM implementation
# Pseudo-example:
# tiered.storage.enable=true
# tiered.storage.bucket=<your-bucket>
# tiered.storage.region=<region>
# tiered.storage.credentials.provider=<provider>

Operations: Cost, Recovery, and Scaling

From a cost perspective, the standard play is to keep the local SSD capacity that supports the hot set lean and push long-term retention to object storage. Cost per GB drops significantly for infrequently accessed data, but you need to budget for request charges and bandwidth during restore and first-fetch scenarios.

Disaster recovery and scale-out become faster. When you add or replace a broker, most historical segments already live in external storage, so you only need to sync the minimum required active data and metadata. This also shortens partition reassignment time (though the exact gain depends on implementation and data volume).

• Clearly define the SLA/durability and region selection for external storage
• Enable lifecycle management and encryption on buckets/containers
• Monitor network bandwidth (egress/ingress) and billing events
• Track resync time during broker replacement as an SLO

Observability and Troubleshooting

Monitoring focuses on offload backlog, remote read latency, local cache hit rate, and external storage error rates / IAM errors. Establish a baseline for normal operation, then define SLOs that account for peaks during overnight batch loads and similar events.

Typical failures include external storage access errors due to insufficient permissions, latency degradation (slow first-fetch), and growing offload backlog. First verify credentials, bucket policies, and network paths, then inspect the broker's offloader threads/queues and timeout settings.

• Monitoring candidates: remote read latency, remote bytes fetched/sec, offload queue depth
• Local vs. remote hit rate (heatmap)
• External storage 4xx/5xx errors and retry ratio
• Detection of IAM/key rotation failures
• Correlation analysis between consumer latency and throttling

Exam Prep and Comparison (Local-Only vs. Tiered)

For CCAAK, make sure you can articulate that Tiered Storage is purely a storage tiering mechanism and does not alter the basic semantics of produce/fetch/ISR/acks/transactions. Be ready to pair the benefits (decoupled scaling, faster recovery, cost optimization) with the trade-offs (first-time remote read latency, dependence on external storage SLA/IAM).

Interactions with compaction and topic cleanup are implementation-dependent. On the exam, frame the support story as being defined by product documentation, and be sure to correctly distinguish design decisions like excluding internal topics from tiering, and the difference between local retention and global retention.

• State explicitly that semantics are unchanged (acks/ISR/EOS)
• Distinguish the definitions of local retention and global retention
• First-time remote reads may be slower
• Internal topics should generally stay local
• Explain why disaster recovery and reassignment become faster

Check Your Understanding

Frequently Asked Questions