How to Stream Db2 to Databricks in Real Time

To stream Db2 to Databricks in real time, enable archive logging and DATA CAPTURE CHANGES on Db2, point TwinCommit at a Databricks SQL warehouse, then run twincommit snapshot followed by twincommit run. Every committed Db2 transaction lands in Delta Lake with MERGE semantics.

Prepare Db2

-- once per database
UPDATE DB CFG FOR PRODDB USING LOGARCHMETH1 DISK:/db2/archive;
-- once per replicated table
ALTER TABLE SALES.ORDERS DATA CAPTURE CHANGES;

Run twincommit check -config db2-dbx.yaml and it reports anything missing, together with the exact command to fix it.

Configure the pipeline

source:
  type: db2
  database: PRODDB
  user: ${DB2_USER}
  password: ${DB2_PASSWORD}
target:
  type: databricks
  host: adb-1234567890.12.azuredatabricks.net
  warehouse: replication-wh
  catalog: main
  schema: db2
  staging_volume: /Volumes/main/db2/_twincommit
tables:
  - source: SALES.ORDERS
  - source: SALES.ORDER_LINES

Snapshot, then stream

twincommit ddl -config db2-dbx.yaml -apply
twincommit snapshot -config db2-dbx.yaml
twincommit run -config db2-dbx.yaml

The snapshot records the exact log position before copying, so streaming picks up from that point with no gap and no overlap.

Why MERGE matters

Appending change rows to a bronze table is easy. Getting a correct silver table is harder: you need deletes applied, updates collapsed and ordering preserved per key. TwinCommit does that work for you. Each micro-batch is a single Delta MERGE, committed together with the replication checkpoint.