Apache Kafka

TwinCommit publishes each source transaction to Kafka inside a Kafka transaction, so consumers using isolation.level=read_committed never see a partial transaction. By default each table gets its own topic, and messages are keyed by primary key so per-key ordering is preserved.

Supported: Apache Kafka 3.x and 4.x, Confluent Cloud and Platform, Amazon MSK, Redpanda.

Configuration

target:
  type: kafka
  brokers: [kafka-1:9093, kafka-2:9093, kafka-3:9093]
  topic: "replica.{schema}.{table}"     # template
  format: avro                          # json | avro | protobuf | debezium
  schema_registry: https://schema-registry:8081
  transactional_id: twincommit-prod-db2
  security:
    protocol: SASL_SSL
    sasl_mechanism: SCRAM-SHA-512
    username: ${KAFKA_USER}
    password: ${KAFKA_PASSWORD}
  compression: zstd

Formats

`format`	Payload
`json`	`{"op":"U","table":"SALES.ORDERS","commit_ts":"…","before":{…},"after":{…}}`
`avro`	Same fields; schemas registered per topic in Schema Registry
`protobuf`	Same fields; schemas registered per topic in Schema Registry
`debezium`	Debezium-compatible envelope (`before`, `after`, `op`, `source`, `ts_ms`) so existing consumers keep working

ACLs

kafka-acls --bootstrap-server kafka-1:9093 --add --allow-principal User:twincommit \
  --operation Write --operation Describe --operation Create --topic replica. --resource-pattern-type prefixed
kafka-acls --bootstrap-server kafka-1:9093 --add --allow-principal User:twincommit \
  --operation Write --operation Describe --transactional-id twincommit-prod-db2

Checkpoint

The checkpoint for Kafka targets is stored in a compacted topic (__twincommit_checkpoints by default) and written inside the same producer transaction as the messages.