Skip to content
TwinCommitDocs

Spill to disk

TwinCommit buffers each source transaction until it commits. A transaction larger than batch.max_txn_memory_bytes (default 256 MiB) is written to spill_dir instead of memory, then read back and applied atomically at commit. Memory use stays flat no matter how big a batch job’s transaction is.

Configuration

Section titled “Configuration”
batch:
  max_txn_memory_bytes: 268435456   # 256 MiB
spill_dir: /var/lib/twincommit/spill

Sizing

Section titled “Sizing”
  • spill_dir needs free space for the largest open transaction. Spilled files are deleted once the transaction is applied or rolled back.
  • Put spill_dir on local SSD, not a network filesystem.
  • In Kubernetes, the Helm chart mounts an emptyDir by default. Set spill.persistentVolume.enabled=true if your largest transaction exceeds node ephemeral storage.

Observing spills

Section titled “Observing spills”

twincommit_open_transactions shows how many transactions are buffered, and /status lists each open transaction with its size and whether it has spilled. A large spilled transaction shows up as rising lag that drops back as soon as it commits.