Skip to content
TwinCommitDocs

Google BigQuery target

TwinCommit loads Google BigQuery by streaming changes through the Storage Write API into a per-table staging table. It then runs a MERGE into the target on a short interval. The MERGE and the checkpoint update run in one multi-statement transaction, so delivery is exactly-once.

Supported: BigQuery (all editions and regions).

Configuration

Section titled “Configuration”
target:
  type: bigquery
  project: acme-analytics
  dataset: replica
  location: US
  credentials_file: /etc/twincommit/bq-sa.json   # or workload identity
  merge_interval: 60s
  partition_by: ""          # optional: column for time partitioning of new tables

How apply works

Section titled “How apply works”
  1. Changes are appended to replica._tc_stage_<table> with the Storage Write API’s committed-type streams.
  2. Every merge_interval, TwinCommit runs BEGIN TRANSACTION; MERGE ...; UPDATE checkpoint ...; COMMIT TRANSACTION;.
  3. Applied staging rows are deleted in the same transaction.

IAM

Section titled “IAM”

Grant the service account:

  • roles/bigquery.dataEditor on the target dataset
  • roles/bigquery.jobUser on the project