Kafka SSL/TLS Encryption: Cluster Communication and Certificate Management Essentials

The Kafka administrator exam frequently tests listener design, mutual authentication (mTLS), certificate handling, and verification/operations procedures. This article sticks to stable concepts from the official documentation so the material works for both the exam and production.

As a baseline, Kafka can protect both broker↔client and broker↔broker traffic with TLS. The underlying principles are the same whether you use ZooKeeper or KRaft mode: certificate validity, SAN values, and listener consistency are what matter.

TLS Scope and Architecture Overview

Kafka can apply TLS to both client traffic and inter-broker replication. The two points that tend to trip people up on the exam and in practice are which listener carries which path, and how certificate trust is wired up.

At minimum, configure the client-facing listener as SSL or SASL_SSL and also encrypt the internal replication listener referenced by inter.broker.listener.name. Each broker gets its own key and SAN entries, all signed by a single trusted CA.

• broker↔client: SSL or SASL_SSL. For mTLS, set ssl.client.auth=required.
• broker↔broker: the listener referenced by inter.broker.listener.name should be SSL or SASL_SSL.
• SAN is required: include DNS names (and IPs when needed) in the Subject Alternative Name. A CN alone is not sufficient.
• Keep the chain consistent under a single CA, and do not forget to include intermediate CAs in the truststore.

A high-level view of Kafka TLS placement

Per-broker certificate requirements at a glance

# 各ブローカーで固有の鍵/証明書
# SAN に FQDN (例: broker1.example.com) と必要なら IP を含める
# すべて同じ信頼された CA で署名
# truststore には中間 CA を含めて完全なチェーンを構成

Listener Design and inter.broker.listener Essentials

Kafka can expose multiple listeners. Separate client-facing and broker-to-broker listeners, and reference the latter via inter.broker.listener.name. Which protocol each listener uses is defined in listener.security.protocol.map.

advertised.listeners is the public address clients use when connecting. The hard rule is that it must match the certificate's SAN, otherwise hostname verification fails.

• Set both listeners and advertised.listeners. The advertised name must use a DNS entry that resolves from the outside.
• inter.broker.listener.name takes a key from the map (e.g. INTERNAL).
• Listeners that require mTLS need ssl.client.auth=required.
• Once you switch inter-broker traffic to SSL, every broker must agree on the same policy.

Representative server.properties (client and internal listeners separated)

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
advertised.listeners=INTERNAL://broker1.example.com:9092,EXTERNAL://broker1.example.com:9093
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
inter.broker.listener.name=INTERNAL

# TLS 共通設定（JKS/PKCS12 どちらでも可。近年は PEM もサポートされるが試験では JKS/PKCS12 が無難）
ssl.keystore.location=/var/private/ssl/kafka.broker1.p12
ssl.keystore.password=${FILE:/etc/kafka/keystore.pass}
ssl.keystore.type=PKCS12
ssl.truststore.location=/var/private/ssl/kafka.truststore.p12
ssl.truststore.password=${FILE:/etc/kafka/truststore.pass}
ssl.truststore.type=PKCS12

# クライアント向けリスナーで相互認証を要求する場合
ssl.client.auth=required

Mode Comparison: One-way TLS, mTLS, SASL_SSL

Choose the mode based on the use case: one-way TLS that encrypts with server certificates only, mTLS that also requires client certificates, or SASL (PLAIN/SCRAM/OAUTHBEARER) layered on top of TLS.

The administrator exam tests whether you can identify, for each mode, what is being proven, which stores are needed, and which settings are critical.

• One-way TLS: server authentication and encryption. Clients only need a truststore.
• mTLS: mutual authentication. Clients also need a keystore, and the broker side must set ssl.client.auth=required.
• SASL_SSL: TLS for encryption while identity is provided via SASL. SCRAM is the common operational choice.

Minimal additional broker-side settings for SASL_SSL (SCRAM)

# 例: INTERNAL は SSL、EXTERNAL を SASL_SSL に
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SASL_SSL
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9094
advertised.listeners=INTERNAL://broker1.example.com:9092,EXTERNAL://broker1.example.com:9094
inter.broker.listener.name=INTERNAL

# SASL 関連
sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.external.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
# 注意: 実際のユーザ作成は kafka-configs.sh --alter --add-config 'SCRAM-SHA-512=[password=...]' などで実施

Certificate Management: Issuance, Import, Rotation

Each broker carries its own private key and certificate signed by a trusted CA. Always include the FQDN in the SAN and keep it consistent with advertised.listeners. Crucially, store the full chain — including intermediate CAs — in the truststore.

Rotate in a rolling fashion: first add the new CA to the truststore → distribute the new certificates → restart nodes one by one. Manage passwords and file permissions with least privilege.

• Format: JKS or PKCS12 is the safe default. Kafka now supports PEM as well, but for the exam, stick with JKS/PKCS12.
• Missing SAN entries and hostname mismatches are the single most common cause of failure.
• Automate expiry monitoring so you do not get taken down by an expired certificate.

Representative steps with keytool/openssl

# 1) キーストア作成と鍵・CSR 生成（PKCS12）
keytool -genkeypair -alias broker1 -keyalg RSA -keystore kafka.broker1.p12 -storetype PKCS12 -storepass changeit -keypass changeit -dname "CN=broker1.example.com, OU=Kafka, O=Example, L=Tokyo, C=JP" -ext SAN=DNS:broker1.example.com,IP:10.0.0.11
keytool -certreq -alias broker1 -keystore kafka.broker1.p12 -storepass changeit -file broker1.csr -ext SAN=DNS:broker1.example.com,IP:10.0.0.11

# 2) CA で署名し、CA ルート/中間と一緒にインポート
keytool -importcert -alias CARoot -file ca-root.crt -keystore kafka.broker1.p12 -storepass changeit -noprompt
keytool -importcert -alias CAInt -file ca-int.crt -keystore kafka.broker1.p12 -storepass changeit -noprompt
keytool -importcert -alias broker1 -file broker1.signed.crt -keystore kafka.broker1.p12 -storepass changeit

# 3) truststore 作成（CA チェーンを格納）
keytool -importcert -alias CARoot -file ca-root.crt -keystore kafka.truststore.p12 -storetype PKCS12 -storepass changeit -noprompt
keytool -importcert -alias CAInt -file ca-int.crt -keystore kafka.truststore.p12 -storetype PKCS12 -storepass changeit -noprompt

# 4) 検証
keytool -list -v -keystore kafka.broker1.p12 -storepass changeit | grep -E "Alias name|Owner|Issuer|SubjectAlternativeName"

Client Configuration and Verification Tips

The minimum client configuration is security.protocol plus a truststore. When connecting to a broker that requires mTLS, the client also needs a keystore. Hostname verification is on by default (ssl.endpoint.identification.algorithm=HTTPS), so the certificate's SAN must match the connection hostname.

openssl s_client is the go-to tool for first-pass connectivity triage. It immediately surfaces certificate chain and hostname verification failures.

• Never store secrets in plaintext property files. Use an external secret store whenever possible.
• During load testing, be careful that keystore locks or DNS resolution failures do not produce false positives.
• Verify TLS version and cipher suite compatibility for your client libraries.

client.properties example and connection verification

# client.properties（単方向 TLS）
bootstrap.servers=broker1.example.com:9093,broker2.example.com:9093
security.protocol=SSL
ssl.truststore.location=/etc/client/kafka.truststore.p12
ssl.truststore.password=${FILE:/etc/client/trust.pass}
ssl.truststore.type=PKCS12
ssl.endpoint.identification.algorithm=HTTPS

# mTLS で必要な追加
ssl.keystore.location=/etc/client/client.p12
ssl.keystore.password=${FILE:/etc/client/key.pass}
ssl.keystore.type=PKCS12

# openssl で事前検証（SNI とホスト名検証の観点で FQDN を使う）
openssl s_client -connect broker1.example.com:9093 -servername broker1.example.com -showcerts

Troubleshooting and Exam Pitfalls

The most common failures are SAN/hostname mismatches, incomplete chains, inconsistent inter.broker.listener.name settings, expired certificates, and password mismatches. Look in the logs for javax.net.ssl.SSLHandshakeException, certificate_unknown, and handshake_failure.

Mapping ACLs to certificate principal names is another exam favorite. When you need to shorten the DN authenticated by mTLS into a principal name, use ssl.principal.mapping.rules.

• Confirm that the advertised.listeners host matches the certificate's SAN.
• Make sure the intermediate CA is in the truststore.
• Verify that inter.broker.listener.name points to SSL or SASL_SSL and is identical across every broker.
• Plan rolling rotations for certificates that are about to expire.
• When ZooKeeper is in the mix, configure TLS to ZooKeeper separately. With KRaft, verify TLS between controllers and brokers.

Commonly used verification commands and settings

# ブローカーの SSL ハンドシェイクエラーを抽出
grep -E "SSLHandshakeException|handshake_failure|certificate_unknown" /var/log/kafka/server.log

# 証明書の SAN を確認
keytool -list -v -keystore /var/private/ssl/kafka.broker1.p12 -storepass changeit | grep -A2 "SubjectAlternativeName"

# mTLS で DN から主体名を抽出するマッピング例（CN を user とする）
ssl.principal.mapping.rules=RULE:^CN=(.*?),.*$/$1/,DEFAULT

Check your understanding

Frequently Asked Questions