Elasticsearch target

TwinCommit keeps Elasticsearch indices in sync with your source tables. It sends _bulk requests in which the document ID is the primary key and the version is the source log position. Elasticsearch rejects stale writes, so replays and retries can never move a document backwards.

Supported: Elasticsearch 8.x and 9.x · Elastic Cloud · OpenSearch 2.x and 3.x · Amazon OpenSearch Service.

Configuration

target:
  type: elasticsearch
  urls: [https://es-1.internal:9200, https://es-2.internal:9200]
  api_key: ${ES_API_KEY}
  index_template: "{schema}_{table}"   # lower-cased
  refresh: false                       # true = refresh after each bulk (slower)

How apply works

Inserts and updates become index actions with _id = pk, version = <log position> and version_type = external.
Deletes become delete actions with the same versioning.
A version_conflict_engine_exception means the document already has a newer change. It is counted, not treated as an error.

Checkpoint and delivery

The checkpoint is stored in a small .twincommit-state index after each successful bulk. A crash between the bulk and the checkpoint can re-send that bulk, but external versioning makes the replay a no-op. The index therefore ends in the same state as the source (effectively-once).

Privileges

{
  "indices": [
    { "names": ["sales_*", ".twincommit-state"], "privileges": ["create_index", "index", "delete", "write", "read", "view_index_metadata"] }
  ]
}