Kafka Sink Connectors Practical Guide: Writing from Kafka to External Systems

Kafka Connect Sink Connectors consume records from Kafka topics and write them to external systems such as databases, search engines, and object storage.

This article summarizes delivery guarantees, duplicate handling, error handling, scaling, and operational best practices grounded in the official Confluent/Apache Kafka documentation, alongside the points most often tested on the CCDAK exam.

Sink Connectors: Overview and Architecture

Kafka Connect is a framework that runs connectors and tasks on workers (standalone or distributed). Sink Connectors read from Kafka and write to external systems. Offsets are stored in internal topics and committed after successful processing.

Data transformation is handled by converters (Avro/JSON Schema/Protobuf, etc.) and SMTs (Single Message Transform). On error, you can configure retries, skipping, or forwarding to a DLQ (Dead Letter Queue).

• A connector is the logical definition; tasks are the units of parallel execution. Use tasks.max to control parallelism.
• Internal topics (connect-configs / connect-offsets / connect-status) are stored on Kafka.
• Ordering is guaranteed per partition. A single partition is processed sequentially within one task.
• Converters handle serialization and schema management; SMTs perform lightweight preprocessing and shaping.
• DLQs are effective for isolating problematic records. Control behavior via the errors.* settings.

Logical architecture of a Kafka Sink Connector

Delivery Semantics and Consistency Model: Handling Duplicates and Ordering

Sink Connectors are fundamentally at-least-once. Retries on failure and rebalances can cause the same record to be written more than once. Designing without assuming end-to-end Exactly-once is the safer approach.

Ordering is preserved within a Kafka partition. Tasks process records in order within their assigned partitions, but ordering across partitions is not guaranteed. Meet ordering requirements through topic and key design.

• Default guarantee: at-least-once (offset commit after successful confirmation).
• Dedup: target-side UPSERT/primary keys, using the key as document ID, downstream dedup.
• Read consistency: to avoid uncommitted records on ingest, set isolation.level=read_committed (via consumer.override.isolation.level).
• Delete propagation: use tombstones (null value) and enable the connector's delete feature (supported connectors only).
• Order within a partition is preserved. Achieve throughput via parallelism and distribution.

Design and Configuration Tips by Major Sink

Connector-specific settings directly affect delivery guarantees, schema evolution, and throughput. Build on the defaults and constraints in the official documentation, and choose the safer design.

Here we list key points using the commonly deployed JDBC / Elasticsearch / S3 / Snowflake connectors as examples.

• JDBC Sink: set insert.mode=upsert, pk.mode=record_key, and explicitly declare pk.fields. Use delete.enabled with behavior.on.null.values=delete for delete propagation. Validate auto.create/auto.evolve in staging before production.
• Elasticsearch Sink: set key.ignore=false to use the key as _id for idempotency. Tune bulk.size and linger to the workload. Watch for mapping conflicts and type conversions.
• S3 Sink: organize rotation (flush.size, rotate.interval.ms) and partitioning (topics.dir, partitioner.class). Tolerate duplicates and design dedup/MERGE downstream.
• Snowflake Sink: operationalize post-ingest MERGE using the target table's keys/unique constraints. Watch out for cost from very small batches. Apply schema changes incrementally for safety.

Error Handling, Retries, and DLQ Implementation Guidelines

Kafka Connect lets you control error handling via configuration. Regardless of whether failures occur during record conversion, serialization, or destination writes, designing for progress via retries, skipping, or DLQ forwarding is the realistic approach.

Always provision a DLQ in production, and build monitoring and reprocessing workflows for DLQ records (such as re-ingest after fixes or one-off corrections) into operations.

• errors.tolerance=all keeps processing past failed records; none stops immediately.
• Specify the DLQ target via errors.deadletterqueue.topic.name. Use context.headers.enable to attach cause information to headers.
• Tune persistence against transient errors with errors.retry.timeout and errors.retry.delay.max.ms.
• Conversion errors: revisit SMT/converter settings (schemas.enable, types, required fields).
• Target-specific limits (constraint violations, timeouts): adjust batch size, concurrency, and retries.

Scaling and Performance: Tasks and Partition Design

Throughput is largely determined by the product of topic partition count and tasks.max. A task can own multiple partitions, but a single partition is always processed sequentially within one task.

Connector-specific batch settings (e.g. JDBC's batch.size) and fine-tuning consumption via Connect's consumer.override.* are also effective. Under overload, the connector pauses the consumer to apply backpressure.

• Setting tasks.max higher than the partition count caps out (depends on assignment limits).
• When scaling up, plan partition count increases and verify key distribution.
• For bulk writes such as JDBC, tune batch.size and the connection pool / commit interval.
• Tune records-per-poll with consumer.override.max.poll.records (balance against processing time).
• Frequent rebalances cause stalls and lag. Ensure a stable worker count and cooperative rebalancing operationally.

Operations, Monitoring, and CCDAK Exam Points

Distributed mode is standard for production. Properly configure replication factor and cleanup policy (such as compact) on internal topics. Monitor using both the Connect REST API and JMX metrics.

On CCDAK, Connect's components, internal topics, delivery guarantees, DLQ, the roles of SMTs/converters, and representative connector settings are frequently tested. Build on the defaults and constraints from the official documentation and pay close attention to the wording of answer choices.

• REST: /connectors, /connectors/<name>/status, /connectors/<name>/tasks
• Internal topics: connect-configs (compact), connect-offsets (compact), connect-status (compact).
• Security: use SSL/SASL for Kafka connections; store connector-specific destination credentials securely.
• Change management: restart tasks in rolling fashion; version-manage configs via REST.
• Exam tips: at-least-once is the baseline, duplicates handled on the target side. Memorize the combination of DLQ and errors.* settings.

Example: creating a JDBC Sink Connector via REST (UPSERT + deletes + DLQ)

POST /connectors HTTP/1.1
Host: connect:8083
Content-Type: application/json

{
  "name": "inventory-jdbc-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "tasks.max": "3",
    "topics": "inventory.customers,inventory.orders",
    "connection.url": "jdbc:postgresql://db:5432/warehouse",
    "connection.user": "app",
    "connection.password": "******",
    "auto.create": "true",
    "auto.evolve": "true",
    "insert.mode": "upsert",
    "pk.mode": "record_key",
    "pk.fields": "id",
    "delete.enabled": "true",
    "behavior.on.null.values": "delete",
    "batch.size": "3000",
    "max.retries": "10",
    "retry.backoff.ms": "5000",
    "errors.tolerance": "all",
    "errors.deadletterqueue.topic.name": "dlq.inventory",
    "errors.deadletterqueue.context.headers.enable": "true",
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "http://schema-registry:8081",
    "value.converter.schema.registry.url": "http://schema-registry:8081",
    "consumer.override.isolation.level": "read_committed"
  }
}

Check Your Understanding

Frequently Asked Questions