Amazon S3 data lake target

TwinCommit writes database changes to Amazon S3 as Apache Iceberg, Delta Lake or Parquet tables. Each micro-batch is one atomic table commit carrying the TwinCommit checkpoint, so delivery is exactly-once. Tables are immediately queryable from Athena, EMR, Redshift Spectrum, Databricks, Snowflake and Trino.

Configuration

target:
  type: s3
  bucket: acme-lake
  path: replica/{schema}/{table}
  region: us-east-1
  format: iceberg            # iceberg | delta | parquet
  mode: merge                # merge | append
  catalog:
    type: glue               # glue | s3tables | rest | hive
    database: replica
  auth: iam_role             # instance profile, IRSA or access keys
  commit_interval: 60s

For Amazon S3 Tables, set catalog.type: s3tables and table_bucket_arn. TwinCommit then writes managed Iceberg tables, and S3 handles compaction.

Write modes and exactly-once

Write modes (merge and append) and the exactly-once mechanism for each format are the same as for the Azure Data Lake Storage target.

IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::acme-lake", "arn:aws:s3:::acme-lake/replica/*"] },
    { "Effect": "Allow", "Action": ["glue:GetDatabase", "glue:GetTable", "glue:CreateTable", "glue:UpdateTable"],
      "Resource": "*" }
  ]
}