Snapshots

A snapshot copies a table’s current contents to the target and records the exact log position from which streaming must continue. You run it once per table at setup, and again for any table that becomes broken.

What happens during a snapshot

1. TwinCommit takes a brief share lock on the table.
2. While no writer is mid-transaction on the table, it records the current log position.
3. It releases the lock. Writers are blocked only for this short window.
4. It bulk-copies the rows to the target, for example with binary COPY on PostgreSQL or staged Parquet plus MERGE on Databricks.
5. It marks the table streaming from the recorded position.

Changes committed while the copy runs are picked up by the stream. Because apply is idempotent, a change that is in both the copy and the stream leaves the same result. See exactly-once.

Commands

# all tables in the pipeline
twincommit snapshot -config pipeline.yaml

# just one or more tables
twincommit snapshot -config pipeline.yaml -tables SALES.ORDERS,SALES.ORDER_LINES

Note

run reads table definitions at startup. Restart run after snapshotting a table whose definition changed.

Tips for large tables

• Run snapshots during quieter periods; the copy reads the whole table.
• Make sure the source keeps enough log to cover the snapshot’s duration.
• Use exclude_columns for unsupported column types rather than skipping the table.