Change Data Capture

Stream row-level changes from a database. A CDC source captures inserts, updates and deletes, optionally hands each subscriber a consistent snapshot of existing rows first, and delivers everything as driver-neutral change events. Sources are addressable by their entry ID and consumed from Lua via the cdc module.

Entry Kinds

Both kinds expose the same Lua API, the same source info record and the same change event shape. What differs is the guarantee set, published per source as capabilities.

Postgres Configuration

- name: pg_cdc
  kind: db.cdc.postgres
  host: ${env:DB_HOST}
  port: 5432
  database: app
  username: ${env:DB_USER}
  password: ${env:app.secrets:db_password}
  slot_name: wippy_slot
  publication: wippy_pub
  tables:
    - public.users
    - public.orders
  snapshot: true
  streaming: true
  standby_interval: "10s"
  status_interval: "10s"
  lifecycle:
    auto_start: true

Zero on any max_* field selects the default; the decoder is never unbounded. Negative values are rejected.

Credentials resolve ${env:NAME} placeholders through the environment registry at decode time.

SQLite Configuration

A SQLite source does not open its own database. It borrows an existing db.sql.sqlite resource and subscribes to that resource's committed-mutation observer, so it captures exactly the writes made through that Wippy SQL resource — writes by another process, another connection or an external tool are not observed.

- name: cdcdb
  kind: db.sql.sqlite
  file: /var/data/app.db
  lifecycle:
    auto_start: true

- name: changes
  kind: db.cdc.sqlite
  db_resource: app:cdcdb
  tables:
    - users
    - orders
  snapshot: true
  status_interval: "30s"
  lifecycle:
    auto_start: true

The source declares the SQL resource as a lifecycle requirement, so the supervisor starts the database first and restarts the source when the database generation is replaced.

Subscription Limits

Each source admits a bounded number of subscribers and reserves their worst-case backlog up front. A snapshot slot stays reserved until the snapshot-enabled stream closes.

subscriptions:
  max_subscriptions: 1024
  max_snapshot_subscriptions: 4
  max_bytes: 268435456

Zero selects the default; negative values are rejected. Exhausting a limit fails the subscription with a retryable errors.UNAVAILABLE.

How It Works

1. A Postgres source connects as a replication user and creates (or resumes) the slot named by slot_name. A SQLite source borrows its db_resource and subscribes to that resource's committed-mutation observer.
2. Row changes are decoded into driver-neutral change events with op of insert, update, delete or truncate.
3. A subscriber whose stream has snapshot enabled — from the entry's snapshot field or from opts.snapshot on the stream — first receives the existing rows as events with op = "snapshot", then continues into live changes with no gap between the two.
4. A Postgres source periodically acknowledges the LSN so the server can release WAL segments (standby_interval).
5. The source registers under its entry ID; Lua code subscribes with cdc.stream.

Capabilities

Every source publishes what it guarantees, so consumers branch on capabilities rather than on the entry kind.

Capability flags describe source progress, not durable delivery: no driver replays events for an individual subscriber that fell behind or disconnected.

Source Info

Each source is described by an info record, returned by cdc.source and cdc.list_sources.

admission counts reservations, not queue fill: active is the admitted subscription count, snapshots the snapshot-enabled subset, reserved_bytes the reserved backlog budget, and rejected the cumulative number of subscriptions refused by the limits.

Permissions

CDC authority is separate from database access: a source can expose every captured row, including before images. Stream filters narrow delivery only; they never grant access to a source.

- name: cdc_reader
  kind: security.policy
  policy:
    effect: allow
    actions: [cdc.source, cdc.subscribe]
    resources: [app:changes]

See Also

• CDC Module - Lua streaming API
• Database - SQL database services
• Environment - Resolving credentials via ${env:NAME}
• Security - Policies and actions