Skip to content
TwinCommitDocs

Google Cloud Storage data lake target

TwinCommit writes database changes to Google Cloud Storage as Apache Iceberg, Delta Lake or Parquet tables. Each micro-batch is one atomic table commit carrying the TwinCommit checkpoint, so delivery is exactly-once. Iceberg tables can be registered in the BigLake metastore and queried from BigQuery, Dataproc and Spark.

Configuration

Section titled “Configuration”
target:
  type: gcs
  bucket: acme-lake
  path: replica/{schema}/{table}
  format: iceberg            # iceberg | delta | parquet
  mode: merge                # merge | append
  catalog:
    type: biglake            # biglake | rest | hive
    project: acme-analytics
    location: us
    catalog: replica
  credentials_file: /secrets/gcs.json   # or workload identity
  commit_interval: 60s

Write modes and exactly-once

Section titled “Write modes and exactly-once”

Write modes and per-format exactly-once behaviour are the same as for the Azure Data Lake Storage target.

Permissions

Section titled “Permissions”

Grant the service account Storage Object Admin on the bucket (or a managed folder). For BigLake registration, also grant BigLake Admin on the catalog.