Exactly-once delivery
TwinCommit delivers every committed source transaction to the target exactly once. It does this by writing the replication checkpoint in the same target transaction as the data, so the data and the record of “how far we got” can never disagree.
How it works
Section titled “How it works”For each batch, TwinCommit opens one transaction on the target and:
- applies every row change in the batch, as upserts and deletes by primary key;
- updates the checkpoint row in
state_schemawith the log position of the last source transaction in the batch; - commits.
If the process, the host or the target fails at any point before the commit, the target rolls back both the data and the checkpoint. On restart TwinCommit reads the checkpoint and resumes from the transaction after it. If the commit succeeded, the checkpoint already reflects it, so nothing is applied twice.
Why upserts matter
Section titled “Why upserts matter”Because every change is an upsert or a delete keyed on the primary key, re-applying a change the target already contains leaves the row unchanged. TwinCommit relies on this during the snapshot handoff, where the stream may replay changes the snapshot already copied.
How each target achieves it
Section titled “How each target achieves it”| Target | How exactly-once is achieved |
|---|---|
| PostgreSQL, SQL Server, Oracle, Db2, MySQL, MariaDB, SAP HANA | Data and checkpoint in one database transaction |
| MongoDB | Data and checkpoint document in one multi-document transaction |
| Amazon Redshift, Google BigQuery, Azure Synapse | Checkpoint updated in the same transaction as the MERGE from staging |
| Azure Data Lake Storage, Amazon S3, Google Cloud Storage | Checkpoint stored in the table commit (Delta txn action, Iceberg snapshot summary), so each batch is committed once |
| Databricks, Hive | Data and checkpoint committed in the same Delta / ACID MERGE commit |
| Snowflake | Checkpoint updated in the same transaction as the MERGE from the change table |
| Kafka | Kafka transactions; the checkpoint is committed as part of the producer transaction |
| Spark | Offsets map to log positions and are tracked by Spark’s own checkpoint |
| Cassandra, Redis, Elasticsearch | Effectively-once: the checkpoint is stored after each acknowledged batch, and writes are idempotent (source commit timestamps on Cassandra, external versions on Elasticsearch, full-row replacement on Redis), so a replayed batch leaves the same end state |