Practical OAuth / OIDC Authentication for Kafka: Modern Auth and Token Management

Kafka supports OAuth 2.0 / OIDC token-based authentication through the SASL/OAUTHBEARER mechanism. Integrating with an IdP lets you avoid distributing credentials and centrally manage rotation and revocation.

This article covers the token flow between producers/consumers and brokers, minimal configuration examples, token lifecycle management, and tips for ACL integration. It also lays out the comparison points you should know for the CCAAK exam.

Why OAuth / OIDC for Kafka?

With traditional SASL/PLAIN or SCRAM, distributing and protecting shared secrets becomes a heavy operational burden. Token-based auth via OAuth / OIDC consolidates authentication at the IdP, where issuance, revocation, and rotation can be managed uniformly. Kafka taps into this through SASL/OAUTHBEARER.

A Kafka client fetches an access token and presents it to the broker. The broker validates the token and then enforces authorization based on ACLs. OIDC's standard JWT + JWKS validation is stable across providers and offers strong interoperability.

• For non-interactive batch and streaming workloads, the Client Credentials flow is the de facto choice
• Brokers validate issuer, audience, and signature (JWKS), and also check expiry (exp) and not-before (nbf)
• Access tokens are assumed to be short-lived: automatic refresh and observability are essential
• On the CCAAK, expect questions comparing mechanisms (PLAIN, SCRAM, OAUTHBEARER, mTLS) and their operational fit

Big Picture and Token Flow for SASL/OAUTHBEARER

A Kafka client first authenticates to the IdP's token endpoint and obtains an access token (usually a JWT). On connect, it presents that token via SASL/OAUTHBEARER. The broker verifies the signature and claims (iss, aud, exp, etc.), and if valid, authorizes operations through Kafka ACLs.

Validation is typically done either locally via JWKS (public key set) or via token introspection. Support and configuration-key naming vary by Kafka distribution and version, so always check the official documentation for the version you are deploying.

• Producers/consumers present access tokens (not ID tokens)
• The broker validates the JWT signature, issuer, audience, and expiry
• Authorization is decided by Kafka ACLs (some distributions use RBAC)

OAuth / OIDC authentication flow in Kafka (overview)

Client Configuration: OIDC Integration from Producers/Consumers

For non-interactive Kafka clients, the standard approach is to fetch access tokens via the Client Credentials flow. Kafka's built-in OAuthBearerLoginCallbackHandler lets you delegate both token acquisition and refresh to configuration alone.

Refresh happens automatically just before expiry. To give yourself a comfortable margin, review the generic SASL refresh settings such as sasl.login.refresh.window.factor.

• Use access tokens; do not present ID tokens
• Make sure scope and audience match your IdP-side policy
• To mitigate clock skew, strictly synchronize all nodes via NTP

Minimal client-side configuration example (properties)

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler

# OIDC のトークンエンドポイント（クライアントクレデンシャル）
sasl.oauthbearer.token.endpoint.url=https://idp.example.com/oauth2/token
sasl.oauthbearer.client.id=kafka-producer
sasl.oauthbearer.client.secret=REDACTED
sasl.oauthbearer.scope=kafka.write
# audience を使う IdP の場合
authentication.oauthbearer.audience=kafka-broker

# リフレッシュ挙動（SASL 共通）
sasl.login.refresh.window.factor=0.8
sasl.login.refresh.window.jitter=0.05
sasl.login.refresh.min.period.seconds=60

Broker Configuration and Key Points for Token Validation

On the broker, enable SASL/OAUTHBEARER and configure JWT validation parameters (issuer, audience, JWKS endpoint, and so on). Configuration-key scope (global vs. per-listener) and naming vary by distribution and version, so cross-check against the official docs for your target environment.

On successful validation, the broker builds a principal from the token's claims (for example, User:<sub>) and evaluates ACLs against that principal.

• Prefer SASL_SSL (at minimum, protect the network with TLS)
• Include strict issuer / audience equality in your tests
• Verify the JWKS cache TTL and behavior during key rotation

Broker-side example (OIDC/JWKS configured per listener)

listeners=EXTERNAL://:9093
listener.security.protocol.map=EXTERNAL:SASL_SSL
sasl.enabled.mechanisms=OAUTHBEARER
sasl.server.callback.handler.class=org.apache.kafka.common.security.oauthbearer.OAuthBearerServerCallbackHandler

# リスナ別の OIDC/JWT 検証設定（配布/版によりキー表記は要確認）
listener.name.external.sasl.oauthbearer.jwks.endpoint.url=https://idp.example.com/oauth2/keys
listener.name.external.sasl.oauthbearer.expected.issuer=https://idp.example.com/
listener.name.external.sasl.oauthbearer.expected.audience=kafka-broker

# 認可は ACL で管理。例: kafka-acls.sh で付与（運用環境に合わせて）

Token Lifetime, Refresh, and Monitoring

Design access tokens to be short-lived and leave no gap by relying on automatic refresh. Kafka clients try to refresh just before expiry, but build in retries and buffers to account for IdP rate limits and transient failures.

In operations, instrument the typical failure modes — token expiry, issuer/audience mismatch, signature verification failure — as metrics. Continuously visualize broker and client logs and metrics (authentication failure count, refresh failure count) on dashboards.

• Set sasl.login.refresh.window.factor to 0.7-0.9 to leave breathing room
• Time synchronization across all nodes (NTP) and TLS expiry monitoring are mandatory
• When the IdP rotates JWKS, verify cache expiry and revocation propagation
• Failure drills: validate client retries and queue buildup when the IdP is temporarily unavailable

Mechanism Comparison and What the CCAAK Targets

The exam frequently asks which mechanism best fits which requirement. Frame your comparison around credential distribution/protection, ease of rotation, interoperability, and the client-side refresh mechanism to keep your thinking organized.

Watch for support differences between Confluent Cloud and self-managed Confluent Platform / Apache Kafka. The cloud data plane is mostly API keys over SASL/PLAIN, while OIDC is used mainly for SSO into the management UI. On the platform side, OAUTHBEARER is the common choice.

• Axes for organizing requirements: presence of secret distribution, rotation frequency, integration with IdP / enterprise stack
• When using a cloud service, always check which mechanisms are supported on the data plane

Example of granting ACLs (the principal is derived from token claims)

# 例: トークンの sub が "alice" の場合にトピックへの Write を許可
kafka-acls.sh \
  --bootstrap-server broker:9093 \
  --add --allow-principal User:alice \
  --operation Write --topic orders

Check Yourself with a Question

Frequently Asked Questions