Audit Log Design in Confluent: CCAAK Practical and Exam Essentials

Audit logs consistently record "who did what, where, when, and with what result." In a Kafka environment, the scope includes API calls (Produce/Fetch/Metadata), permission changes, schema operations, and connection information.

Confluent implements and delivers audit logs differently in Cloud and Platform. This article explains the design decisions and implementation pitfalls based on those differences.

Purpose and Scope of Audit Logs

Audit logs serve a different purpose than observability logs or metrics: their primary goal is to retain a consistent event history for after-the-fact verification and compliance evidence. In a Kafka environment, you especially need to comprehensively record authorization decisions (allow/deny), administrative operations (Topic/ACL/RBAC/Connector/ksqlDB/Schema Registry), and client identity (principal, client ID, authentication method, source IP).

In Confluent Cloud, audit logs are delivered to external sinks via a managed delivery mechanism. In Confluent Platform (self-managed), brokers and surrounding components publish audit events as Kafka topics, which are then forwarded externally. Both use schematized events, and the principle of least privilege along with tamper-resistant design is essential.

• Target actions: authentication/authorization, administrative operations on Topic/ACL/RBAC/Schema/Connect/ksqlDB, and data plane actions like Produce/Fetch
• Required attributes: actor (executing principal), action (operation), resource (target), outcome (allowed/denied), timestamp, and request metadata (client.id, etc.)
• Separate audit logs from operational logs, and prioritize WORM characteristics (tamper-resistant) for storage

Confluent Cloud Audit Logs: Design Basics

Confluent Cloud collects audit logs as a managed service and delivers them in near real time to your chosen sinks (for example S3, GCS, Azure Blob, Datadog, Splunk). Users do not subscribe directly from a Kafka topic; storage and analysis happen at the delivery destination.

Events include organization/environment/cluster identifiers, resource type, action, principal, outcome, and timestamp. The basic design assumes retention is managed by the sink's policy; do not expect long-term retention on the Cloud side.

• Simple to adopt: the management plane is absorbed by Cloud, and delivery reliability and schema consistency are easier to guarantee
• Access control: operations on audit log settings are restricted to organization-level admin roles
• Design point: when regions/accounts are separated, separate the sinks as well and aggregate horizontally on the SIEM side

Confluent Platform Audit Logs: Design Basics

In Confluent Platform (including Confluent Server), audit events are published as internal Kafka topics. The scope covers RBAC/ACL authorization decisions, administrative APIs, and major data plane operations. Audit topics are protected with high availability (sufficient replicas and partitions) and strict access control (read access limited to audit personnel).

External systems (SIEM, object storage) are reached via various Kafka Connect sinks (S3, GCS, Azure, Splunk, etc.). Observability of the forwarding pipeline itself and a retry design (at least at-least-once) are essential.

• Mirroring the internal topics to a separate cluster (audit-dedicated) is also an effective design
• Denial events and administrative operations are top monitoring priorities; aggregate/thin successful events according to requirements
• Strictly synchronize time (NTP) across all nodes to ensure timeline consistency

Design Patterns and Architecture

Cloud uses managed delivery as-is and standardizes storage and analysis at the sink. For Platform, evaluate redundancy and capacity sizing (growth during high throughput) in advance for the internal topic → Connect → sink path.

Critical events (denials, permission changes, data definition changes) should be routed to separate topics/storage tiers and given longer retention. This makes it easier to meet audit requirements while controlling operational cost.

• Cloud: split sinks by environment/organization and use tags for cost allocation
• Platform: separate the audit-dedicated Connect worker to isolate backpressure from production traffic
• Common to all approaches: explicitly define an isolation buffer (DLQ) and retry policy for write failures

Standard Cloud/Platform audit log paths (overview)

Audit Event Schema and Filtering

Audit events generally consist of the following fields: actor (principal: user/service account), action (operation: CREATE_TOPIC/ALTER_ACL/PRODUCE, etc.), resource (type, name, scope), outcome (ALLOWED/DENIED), reason (denial reason), network/auth information, client.id, correlation.id, and timestamp. There are representation differences between Cloud and Platform, but this skeleton is stable.

High-volume allowed events drive up long-term storage costs. If your requirements permit, prioritize storing denied events and administrative operations, and aggregate (sampling/daily roll-up) successful events. Mask or tokenize confidential data (token fragments, IP segments, user attributes) on the sink side.

• Inventory fields that may contain PII/sensitive information before collection
• To prepare for schema evolution, do not drop unknown fields; pass them through transparently
• Normalize time to a single reference (UTC) and make it explicit at ingestion

Example: Long-term storage of an audit topic with Kafka Connect (S3 Sink)

{
  "name": "audit-s3-sink",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "tasks.max": "2",
    "topics": "audit-log-events",
    "s3.bucket.name": "org-audit-logs",
    "s3.part.size": "5242880",
    "flush.size": "1000",
    "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
    "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
    "path.format": "year=YYYY/month=MM/day=dd/hour=HH",
    "locale": "en_US",
    "timezone": "UTC",
    "timestamp.extractor": "Record",
    "behavior.on.null.values": "ignore",
    "rotate.schedule.interval.ms": "900000",
    "schema.compatibility": "BACKWARD"
  }
}

Operations, Protection, and CCAAK Exam Essentials

In operations, record audits of the audit path itself (who disabled or modified it) through a separate path. Keep access permissions on the audit topic minimal; on the Platform side, apply encryption (in transit/at rest) and WORM settings for log storage destinations. On the Cloud side, strictly manage delivery destination IAM, storage class, and lifecycle.

On CCAAK, the differences between Cloud and Platform on "where to configure and where it outputs," the differences in RBAC/ACL and their impact on audit events, and the responsibility boundary when integrating with a SIEM are commonly tested. A frequent pitfall is treating broker standard logs or OS logs as substitutes for audit logs.

• Cloud: audit logs use managed delivery. Direct consumption by a Kafka consumer is not assumed
• Platform: durability/protection of the internal topic is the user's responsibility. Clearly define the sink's retry design
• Performance: the overhead of enabling auditing is small but non-zero. Plan internal topic partitions/replicas accordingly

Example: Audit event (excerpt)

{
  "actor": {"type": "User", "name": "alice"},
  "action": "CREATE_TOPIC",
  "resource": {"type": "Topic", "name": "payments", "scope": {"cluster_id": "lkc-xxxx"}},
  "outcome": "ALLOWED",
  "request": {"client_id": "producer-1", "ip": "203.0.113.10"},
  "auth": {"mechanism": "SASL_SSL", "principal": "User:alice"},
  "correlation_id": "c-7b3d",
  "ts": "2026-04-18T04:21:34Z"
}

Practice Question

Frequently Asked Questions