Databricks

TwinCommit streams committed transactions into Delta Lake tables on Databricks. Each micro-batch is staged as Parquet in a Unity Catalog volume or external location, then applied with a Delta MERGE through a SQL warehouse. Deletes, updates and per-key ordering are preserved, and the checkpoint commits with the data.

Supported: Databricks on AWS, Azure and GCP; Unity Catalog; Delta Lake 3+.

Configuration

target:
  type: databricks
  host: adb-1234567890123456.7.azuredatabricks.net
  warehouse_id: 4b9c1f0e2a7d3c55
  catalog: prod
  schema: replica
  staging: /Volumes/prod/replica/_twincommit_staging
  auth:
    client_id: ${DATABRICKS_CLIENT_ID}
    client_secret: ${DATABRICKS_CLIENT_SECRET}
  merge_interval: 10s

Key	Description
`host`	Workspace hostname
`warehouse_id`	SQL warehouse that runs the MERGE
`catalog`, `schema`	Default Unity Catalog location for tables
`staging`	Volume path or external location for Parquet staging files
`auth`	OAuth machine-to-machine credentials for a service principal
`merge_interval`	How often staged batches are merged (default `10s`)

Grants

GRANT USE CATALOG ON CATALOG prod TO `twincommit-sp`;
GRANT USE SCHEMA, CREATE TABLE, MODIFY, SELECT ON SCHEMA prod.replica TO `twincommit-sp`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME prod.replica._twincommit_staging TO `twincommit-sp`;

Also grant the service principal CAN USE on the SQL warehouse.

Table layout

Tables are created as managed Delta tables with the source primary key as a liquid clustering key. Two metadata columns are added:

Column	Meaning
`_tc_commit_ts`	Source commit timestamp
`_tc_op`	Last operation (`I`, `U`; deleted rows are removed)