Skip to content
TwinCommitDocs

Azure Data Lake Storage target

TwinCommit writes database changes to Azure Data Lake Storage Gen2 as open table formats: Delta Lake, Apache Iceberg or plain Parquet. Each micro-batch becomes one atomic table commit, and the TwinCommit checkpoint is stored inside that commit, so a restart never writes a batch twice.

The tables can be queried directly by Synapse serverless SQL, Microsoft Fabric (OneLake shortcuts), Databricks, Spark, Trino and DuckDB.

Configuration

Section titled “Configuration”
target:
  type: adls
  account: acmelake
  container: lake
  path: replica/{schema}/{table}
  format: delta              # delta | iceberg | parquet
  mode: merge                # merge (current state) | append (change log)
  auth: service_principal    # or managed_identity, account_key
  client_id: ${AZURE_CLIENT_ID}
  client_secret: ${AZURE_CLIENT_SECRET}
  tenant_id: ${AZURE_TENANT_ID}
  commit_interval: 60s
  target_file_size: 128MiB

Iceberg tables can be registered in a catalog with catalog: { type: rest, uri: https://catalog.internal/api } or catalog: { type: hive, uri: thrift://hms:9083 }.

Write modes

Section titled “Write modes”
mode Result Use it for
merge Table mirrors the source’s current state. Updates and deletes are applied with merge-on-read deletion vectors (Delta) or position deletes (Iceberg). BI, reporting and lakehouse silver tables
append Every change is a row with _tc_op, _tc_lsn, _tc_commit_ts and _tc_txn columns. Bronze or audit tables, and history and point-in-time queries

Parquet format supports append only, because plain Parquet has no delete mechanism.

How exactly-once works

Section titled “How exactly-once works”
  • Delta Lake: each commit carries a txn action with appId = twincommit/<pipeline> and version = <log position>. On restart, TwinCommit reads the latest txn version from the Delta log and skips anything already committed.
  • Iceberg: the log position is written into the snapshot summary (twincommit.position), and commits use optimistic concurrency against the catalog.
  • Parquet: file names are derived from the log position range, and a _twincommit/_checkpoint object is written after the files. A replayed batch overwrites identical files rather than creating duplicates.

Maintenance

Section titled “Maintenance”

TwinCommit compacts small files and runs OPTIMIZE (Delta) or rewrite_data_files (Iceberg) on the maintenance_interval you set (default 6h). It also expires old snapshots after retain_snapshots (default 7 days).

Permissions

Section titled “Permissions”

Grant the service principal or managed identity Storage Blob Data Contributor on the container, or ACL rwx on the target path with hierarchical namespace enabled.