Kafka Private Networking Strategy: Design and Ops Guide for VPC Peering and PrivateLink

Kafka is designed so clients connect directly to each broker's advertised address (advertised.listeners), so the choice of private connectivity has a direct impact on client reachability and DNS design. This article focuses on AWS VPC Peering and PrivateLink, covering Kafka-appropriate designs and the points most often tested on the CCAAK exam.

The concepts here reflect the stable, generally-applicable behavior of Apache Kafka and Confluent plus AWS official specifications. Vendor-specific recent changes or constraints can be environment-dependent, so we recommend cross-checking the official documentation at adoption time. References: Apache Kafka docs, Confluent docs.

Reachability Premise and the Two Options

A Kafka client first connects to the bootstrap server, fetches metadata containing each broker's advertised name (advertised.listeners), and then reconnects to the individual brokers. So for private connectivity to succeed, two conditions are required: the client VPC must privately reach every broker's advertised name/port, and name resolution must match those advertised names.

The main options on AWS for connectivity that avoids the public internet are VPC Peering and PrivateLink. The former builds reachability via L3 routing between VPCs; the latter publishes the service through an NLB and uses Interface Endpoints on the client side. Both apply to managed Kafka (e.g., Confluent) as well as self-managed deployments, but the design and operational considerations differ significantly.

• VPC Peering: routable private-IP reachability with bidirectional traffic. Overlapping CIDRs are not allowed.
• PrivateLink: one-way access through an NLB. Overlapping CIDRs are fine. DNS and endpoint handling are the key.

Reachability requirements from the Kafka client's perspective (abstract diagram)

Minimal client configuration for verifying a Kafka connection

bootstrap.servers=broker-1.aws.internal:9093,broker-2.aws.internal:9093\nsecurity.protocol=SASL_SSL\nsasl.mechanism=PLAIN\nssl.endpoint.identification.algorithm=HTTPS\n# 実環境では認証情報やCAは適切に管理すること

Designing Kafka Connectivity with VPC Peering

VPC Peering establishes private-IP routing between VPCs and enables bidirectional traffic. With Kafka, the basic pattern is to advertise each broker's private address (or private DNS name) and align routes, security groups, and network ACLs from the client VPC.

For DNS, use a private zone (such as a Route 53 Private Hosted Zone) so that FQDNs like broker-N.aws.internal resolve in a way that matches the advertised names. Note that VPC Peering is not transitive and does not allow overlapping CIDRs. Cross-account / cross-region peering is supported, but check route counts and limits at design time.

• Set advertised.listeners to a private name/address that clients can actually reach
• Add a route to the peer VPC's CIDR in the client-side route table
• Allow the client VPC's source range in security groups
• Associate Private DNS with both VPCs so name resolution is consistent

VPC Peering traffic path (conceptual)

Typical broker-side listener configuration (assuming Peering)

# server.properties\nlisteners=SASL_SSL://0.0.0.0:9093\nadvertised.listeners=SASL_SSL://broker-1.aws.internal:9093\nlistener.security.protocol.map=SASL_SSL:SASL_SSL\ninter.broker.listener.name=SASL_SSL\n# DNSはbroker-1.aws.internal -> 10.0.1.10 (例) をPrivate Hosted Zoneで解決

Designing Kafka Connectivity with PrivateLink

PrivateLink publishes a Network Load Balancer (NLB) in the provider VPC as a "service" and provides reachability via an Interface VPC Endpoint (ENI) in the consumer VPC. It works well in environments that need to tolerate overlapping CIDRs or enforce strict segmentation.

Because Kafka reconnects to each broker after metadata, NLB and endpoint design is the key. Typically you provide at least an NLB for bootstrap, and then arrange one endpoint per broker (or equivalent name resolution) so that the advertised names the client receives are reachable via the endpoints. TLS is usually terminated at the brokers (passthrough), with the NLB running in TCP/TLS passthrough mode.

• The NLB forwards traffic to each broker's port via TCP or TLS listeners
• After creating the Interface Endpoint, make the endpoint's private DNS name resolvable on the client side
• Configure advertised names to match the endpoint's DNS (use Private DNS overrides)
• Cross-account exposure can be controlled via invitation/approval

Reachability with PrivateLink (conceptual)

Example of aligning advertised names with PrivateLink DNS

# server.properties (例。実環境のFQDNは環境固有)\nadvertised.listeners=SASL_SSL://b1.plink.example.internal:9093\n# Route53 Private Hosted Zoneなどで b1.plink.example.internal -> vpce-abcde-1.amazonaws.com をCNAMEで解決\n# NLBはb1エンドポイントからブローカー1のポートへ転送

Comparing VPC Peering and PrivateLink

Both are "private connectivity," but the network models differ, so the design and operational considerations differ as well. With Kafka, the alignment between DNS and advertised names, plus the requirement that clients connect directly to brokers, drives the key differences.

From the CCAAK exam's perspective, you'll often be asked about design trade-offs such as overlapping CIDRs, cross-account requirements, and DNS / advertised-name alignment.

• When overlapping CIDRs or strict boundaries are required: PrivateLink fits better
• Simple L3 connectivity and bidirectional traffic: VPC Peering is the straightforward choice
• DNS and naming design difficulty: tends to be higher with PrivateLink

Comparing the two approaches (abstract topology)

Example connection verification with kcat (works for either approach)

kcat -b broker-1.aws.internal:9093 -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN \\\n     -X sasl.username=... -X sasl.password=... -L\n# -Lでメタデータを確認し、各brokerの広告名に私設到達できているかを検証

Points Likely to Appear on the CCAAK Exam

From an administrator's viewpoint, what matters is translating the differences between connectivity options into Kafka's advertised names, security, and operations. Specifically, you should be able to explain how each approach satisfies name resolution and reachability, on the premise that "clients ultimately connect to each broker directly."

Certificate verification (hostname match) with public paths closed, ACLs / authorization, and least-privilege security groups also come up frequently.

• Can you explain the alignment between advertised.listeners and DNS?
• Why PrivateLink is advantageous for overlapping CIDRs and cross-account scenarios
• VPC Peering is bidirectional at L3; PrivateLink is one-way service publication
• TLS is typically terminated at the brokers; the NLB runs in passthrough mode
• Mismatched name resolution leads to client connection failures (handshake / unreachable)

Bullet-list summary of points to lock in for the exam

- クライアント->ブローカー直収が前提\n- DNSと広告名の一致\n- Peering=ルート/SG/ACL整合\n- PrivateLink=エンドポイントとNLBの対応

Example command for checking advertised.listeners

# ブローカー上で実効設定を確認 (例)\ncat /etc/kafka/server.properties | grep -E 'listeners|advertised.listeners|inter.broker.listener.name'

Troubleshooting and Verification Procedure

Isolate reachability problems by layer. Start with DNS, then L3 reachability, and finally TLS/SASL. For PrivateLink, also check NLB target health and the endpoint's status.

Confluent and Apache Kafka clients often have hostname verification enabled, and TLS handshake failures due to mismatches between certificate CN/SAN and advertised names are common. Check the logs for javax.net.ssl and SSLHandshakeException.

• DNS: does dig/nslookup resolve the advertised name to the expected private destination?
• L3: port reachability via nc/telnet; confirm VPC Flow Logs and security group hits
• PrivateLink: NLB target health, endpoint approval/association state, and Private DNS enablement
• Client: use kcat -L for metadata and identify which broker stalls when it fails

Simplified troubleshooting flowchart

Basic commands for reachability and name resolution

dig +short broker-1.aws.internal\nnc -vz broker-1.aws.internal 9093\naws ec2 describe-vpc-endpoints --filters Name=vpc-endpoint-id,Values=vpce-xxxxx

Check Yourself with a Question

Frequently Asked Questions