MongoDB

TwinCommit captures changes from MongoDB with change streams. It opens one cluster-wide stream, keeps the resume token in its checkpoint, and groups changes from multi-document transactions so they are applied as a unit. Nothing is installed in the cluster.

Supported: MongoDB 6.0, 7.0 and 8.x · replica sets · sharded clusters · MongoDB Atlas · Amazon DocumentDB (change streams enabled).

Prerequisites

Replica set or sharded cluster

Change streams need a replica set or a sharded cluster. A standalone mongod is not supported. Convert one with:

mongod --replSet rs0 ...
mongosh --eval 'rs.initiate()'

Full documents and pre-images

Enable pre- and post-images on each captured collection. TwinCommit can then emit full before and after images for updates and deletes:

db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: { enabled: true } })

Oplog window

Size the oplog for the longest outage you want to survive:

db.adminCommand({ replSetResizeOplog: 1, minRetentionHours: 72 })

User

db.getSiblingDB("admin").createUser({
  user: "twincommit",
  pwd: "********",
  roles: [
    { role: "read", db: "app" },
    { role: "clusterMonitor", db: "admin" }
  ]
})

On Atlas, use the built-in Read Any Database role plus clusterMonitor.

Configuration

source:
  type: mongodb
  uri: mongodb+srv://twincommit:${MONGO_PASSWORD}@cluster0.example.mongodb.net/?readPreference=secondaryPreferred
  database: app

List collections under tables: as database.collection. With relational targets, set flatten: true per table to map top-level fields to columns, or keep the default to store the document in a single JSON column.

How capture works

• The snapshot reads each collection at a cluster time, then the change stream starts at that startAtOperationTime, so there is no gap and no overlap.
• Events that share a lsid and txnNumber belong to one multi-document transaction and are applied together.
• The _id becomes the target primary key.

Limitations

• Time-series collections and capped collections are not captured.
• Documents larger than 16 MiB after the update cannot be emitted with full post-images.
• If the resume token falls off the oplog, affected collections must be re-snapshotted.