Amazon Redshift target

TwinCommit loads Amazon Redshift by staging each micro-batch as Parquet in S3. It then runs COPY into a staging table and a MERGE into the target. The MERGE and the checkpoint update run in one Redshift transaction, so delivery is exactly-once.

Supported: Amazon Redshift (RA3) · Redshift Serverless.

Configuration

target:
  type: redshift
  dsn: postgres://twincommit:${REDSHIFT_PASSWORD}@analytics.abc123.us-east-1.redshift.amazonaws.com:5439/dev
  schema: replica
  s3_staging: s3://acme-twincommit-staging/redshift/
  iam_role: arn:aws:iam::123456789012:role/twincommit-redshift-copy
  merge_interval: 60s

How apply works

1. Changes are accumulated for up to merge_interval and written as Parquet to s3_staging.
2. COPY loads the batch into a temporary staging table.
3. MERGE applies inserts, updates and deletes using the latest change per key, then the checkpoint is updated and the transaction commits.
4. Staged files are deleted.

Grants

CREATE USER twincommit PASSWORD '********';
GRANT CREATE ON DATABASE dev TO twincommit;
GRANT ALL ON SCHEMA replica TO twincommit;

The IAM role needs s3:GetObject and s3:ListBucket on the staging prefix. TwinCommit itself needs s3:PutObject and s3:DeleteObject.