Prometheus metrics

TwinCommit exposes Prometheus metrics on /metrics while twincommit run is active. Every metric carries a pipeline label.

Metrics

Metric	Type	Description
`twincommit_lag_seconds`	gauge	Time between the source commit of the last applied transaction and now
`twincommit_caught_up`	gauge	`1` when the reader has reached the end of the log and all buffered work is applied
`twincommit_broken_tables`	gauge	Number of tables in the `broken` state
`twincommit_open_transactions`	gauge	Source transactions currently buffered (in memory or spilled)
`twincommit_log_records_total`	counter	Log records read
`twincommit_transactions_total`	counter	Transactions applied
`twincommit_rows_total{op}`	counter	Rows applied, by `op` = `insert`, `update`, `delete`, `truncate`
`twincommit_errors_total`	counter	Errors that caused a retry

Scrape config

scrape_configs:
  - job_name: twincommit
    static_configs:
      - targets: ['replication-01:9108', 'replication-02:9108']

On Kubernetes, the Helm chart can create a ServiceMonitor: --set serviceMonitor.enabled=true.

Alerting rules

groups:
  - name: twincommit
    rules:
      - alert: TwinCommitLagHigh
        expr: twincommit_lag_seconds > 300
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Replication lag on {{ $labels.pipeline }} is {{ $value | humanizeDuration }}"

      - alert: TwinCommitBrokenTables
        expr: twincommit_broken_tables > 0
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "{{ $value }} broken table(s) on {{ $labels.pipeline }}; re-snapshot required"

      - alert: TwinCommitErrors
        expr: increase(twincommit_errors_total[15m]) > 5
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.pipeline }} is retrying repeatedly"

      - alert: TwinCommitDown
        expr: up{job="twincommit"} == 0
        for: 2m
        labels: { severity: critical }