Kafka ACL Design Guide: Combining Principal, Resource, and Operation Correctly

Kafka access control is built from five elements: Principal (who), Resource (what), Operation (what action), Permission (allow/deny), and Pattern (how it matches). What trips people up on the exam and in production is how to combine these five elements and how the evaluation order works.

This article distills the design fundamentals and minimal CLI examples based on the official documentation. With CCAAK in mind, it covers naming conventions, prefix matching, and Deny rule handling that you can apply directly in production.

ACL Basics: The 5 Elements and a Safe Initial Setup

A Kafka ACL is defined by a combination of Principal (User:app, CN=client, a Kerberos principal, etc.), Resource (Topic, Group, Cluster, TransactionalId, etc.), Operation (READ, WRITE, CREATE, etc.), Permission (ALLOW or DENY), and Pattern (LITERAL or PREFIXED). The evaluation rule is simple: if any matching DENY exists, deny; otherwise if any matching ALLOW exists, allow; otherwise deny.

For the initial rollout, enable StandardAuthorizer (the default Authorizer in recent Kafka versions), explicitly declare super-users for operations, and grant only the minimum required privileges to business apps.

• Main Resources: Topic, Group, Cluster, TransactionalId
• Key Operations and what they mean: READ (fetch / group join / offset commit), WRITE (produce), CREATE/DELETE (create/delete), DESCRIBE (read metadata), ALTER/ALTER_CONFIGS (modify), DESCRIBE_CONFIGS (read configuration), IDEMPOTENT_WRITE (permit idempotent produce)
• Pattern: LITERAL (exact match), PREFIXED (prefix match). The wildcard "*" is also available, but it affects a wide scope so use it carefully.
• DENY is powerful. Limit it to exceptional blocks; design primarily around ALLOW rules.

Granting a basic ACL (Broker 3.x and later: --bootstrap-server is recommended)

bin/kafka-acls.sh \
  --bootstrap-server broker1:9092 \
  --add --allow-principal User:app-writer \
  --operation WRITE --operation DESCRIBE \
  --topic payments

# Consumer (READ on topic + READ on group)
bin/kafka-acls.sh \
  --bootstrap-server broker1:9092 \
  --add --allow-principal User:app-reader \
  --operation READ --topic payments \
  --operation READ --group pay-consumer

Principal Design: Service Accounts and Naming Conventions

A Principal is the identity passed in from the authentication layer. Common forms are SASL/SCRAM usernames, SASL/PLAIN users, mTLS certificate subjects (CN, etc.), and Kerberos principals. ACLs are ultimately evaluated against a string such as User:xxx.

Design service accounts per app and per role, and embed the environment and system name in the prefix to avoid collisions and accidental cross-cutting grants. Always separate human-use principals from service principals so incident investigation stays clean.

• Example names: prod-appA-writer, stg-appA-reader. Include env-system-role.
• One account per app (no sharing) to make rotation and revocation easy
• For mTLS, lock down the rule that maps certificate CN/OU to Principal
• Enumerate super-users explicitly, e.g., super.users=User:ops-admin

Creating a SASL/SCRAM user and granting an ACL (example)

# Create a SCRAM user (on the broker)
bin/kafka-configs.sh --bootstrap-server broker1:9092 \
  --alter --add-config 'SCRAM-SHA-512=[password=secret-pass]' \
  --entity-type users --entity-name prod-appA-writer

# Write ACL
auth_user=User:prod-appA-writer
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal "$auth_user" \
  --operation WRITE --operation DESCRIBE \
  --topic payments-orders

Resource Design: Namespaces and When to Use PREFIXED

Carving out namespaces for topics and groups dramatically simplifies ACL design. Standardize on a convention like system-domain-purpose-environment and grant in bulk with prefix matching (PREFIXED).

That said, PREFIXED tends to cover more than you think. Rather than papering over the issue with Deny rules, it is safer to revisit the prefix design on the ALLOW side.

• Example names: payments-orders-prod, payments-refunds-prod, analytics-raw-stg
• Include env-system-role in group names too, e.g., prod-appA-consumer
• Do not mix operational and business workloads under the same prefix (reduces human error)
• As a rule, disable auto topic creation and manage topics through an admin

Conceptual view of namespaces and prefix-matched ACLs

Granting a PREFIXED ACL (apply across topics/groups at once)

bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:prod-appA-reader \
  --resource-pattern-type prefixed \
  --operation READ --topic payments- \
  --operation READ --group prod-appA-

Operation Design: Standard Least-Privilege Sets

The required Operations are essentially fixed per typical role: producer, consumer, or admin. If you use an idempotent producer or transactions, you also need to grant on Cluster and/or TransactionalId.

In production, auto.create.topics.enable is typically disabled, so restrict topic creation to admin principals.

• Producer: WRITE + DESCRIBE on the target Topic. For idempotent producers, also add IDEMPOTENT_WRITE on Cluster.
• Transactional producer: the above plus WRITE on the target TransactionalId
• Consumer (with group management): READ on the target Topic and READ on the target Group
• Topic admin: CREATE/DELETE/ALTER/DESCRIBE/ALTER_CONFIGS/DESCRIBE_CONFIGS on the target Topic. Add CREATE on Cluster as needed.
• Monitoring use: grant only DESCRIBE and DESCRIBE_CONFIGS (the minimum)

ACL examples for typical roles

# Idempotent producer
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:prod-appA-writer \
  --operation WRITE --operation DESCRIBE --topic payments-orders-prod
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:prod-appA-writer \
  --operation IDEMPOTENT_WRITE --cluster

# Transactional producer (transactional.id=tx-payments)
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:prod-appA-writer \
  --operation WRITE --transactional-id tx-payments

# Consumer (per group)
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:prod-appA-reader \
  --operation READ --topic payments-orders-prod \
  --operation READ --group prod-appA-consumer

Evaluation Order and Deny: What Happens on a Conflict

The Kafka Authorizer follows a simple rule: if any matching DENY exists, deny; if there is no DENY but any matching ALLOW exists, allow; otherwise deny. There is no concept of one pattern being "more specific" than another. If you place a narrow LITERAL DENY against a broad PREFIXED ALLOW, the DENY always wins and the request is blocked.

One exception: principals listed in the broker's super.users config bypass ACLs entirely. Restrict super.users to operators only and always keep audit logs enabled.

• Treat Deny as a last resort. Don't use it to paper over design mistakes; revisit your namespaces and ALLOW rules instead.
• Even when multiple patterns match, the order is Deny > Allow > Default deny
• Audit: enable authorizer.log.enabled and monitor deny events

Concrete Deny example (block one specific topic)

# Broad allow (READ on payments-*)
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:prod-appA-reader \
  --resource-pattern-type prefixed \
  --operation READ --topic payments-

# But block refunds (Deny wins)
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --deny-principal User:prod-appA-reader \
  --operation READ --topic payments-refunds-prod

Operations, Verification, and Governance: Safety Valves for Changes

ACLs take effect immediately, so misconfigurations have a big blast radius. Make changes safer by visualizing the impact in advance and codifying naming rules and grant policies as code (IaC) for reviewability. On Confluent you can also combine ACLs with RBAC, but the exam focuses on basic ACL evaluation.

For auditing, reconcile deny logs against who was denied which operation and which ACL matched, on a daily basis. During app migrations, the practical approach is to grant broadly via PREFIXED temporarily, then tighten the scope once the migration completes.

• Current state: inventory by Principal/Resource using kafka-acls.sh --list
• Make changes via Pull Request, managed together with the naming convention
• Pre-production deny test: verify blocking with a dummy Principal
• Environment isolation: separate prefixes and clusters between stg and prod

Inventory, deletion, and safe rollback

# List
bin/kafka-acls.sh --bootstrap-server broker1:9092 --list

# Remove a specific ACL (undo a mistake)
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --remove --allow-principal User:prod-appA-reader \
  --operation READ --group prod-appA-consumer

# Temporarily grant a broad PREFIXED allow to help migration, then narrow it later
bin/kafka-acls.sh --bootstrap-server broker1:9092 \
  --add --allow-principal User:migration-helper \
  --resource-pattern-type prefixed --operation READ --topic payments-

Check Your Understanding

Frequently Asked Questions