Change Data Capture for a Search Index: Keeping Downstream Systems in Sync

How do I keep another system in sync with my search index?

You have a second system that needs to know what changed: a warehouse doing analytics, a cache that has to invalidate, a downstream index, a reconciliation job that checks nothing was dropped. The index is being written to continuously by extraction pipelines, imports and user edits.

There are three ways to find out what changed, and they fail differently.

Polling by timestamp, webhooks, or a change feed?

Approach

How it works

Where it loses data

Poll by updated_at	Query for rows newer than the last timestamp you saw	Clock skew and same-millisecond writes. Two rows written in the same tick, you read between them, you never see the second
Webhooks	The producer pushes each change to your endpoint	Your endpoint is down, the delivery is dropped or retried out of order, and you cannot tell which happened
Ordered change feed	You pull a monotonic sequence and store your position	Only if you resume outside the retention window, or change filters mid-cursor

Polling by timestamp is the one most teams reach for and the one that loses data quietest. It looks correct in testing, because in testing the writes are far apart. Under real write pressure, two documents committed inside the same timestamp granularity are indistinguishable, and a reader that has already advanced past that timestamp will never come back for the second one. Nothing errors. The row is simply absent downstream, and you find out when someone notices a count that does not reconcile.

Webhooks solve the latency problem and introduce a delivery problem. A push you did not receive looks exactly like a change that did not happen, which means a webhook consumer cannot answer "am I complete?" without a second mechanism.

An ordered feed inverts the responsibility. The producer keeps a sequence, the consumer keeps a position, and completeness becomes a property the consumer can check for itself.

How does a cursor-resumable feed work?

Every change gets a position in one monotonic, organization-wide sequence. You read a page, you process it, and you store the next_cursor the response hands back. If your consumer dies mid-page, it restarts from the last cursor it committed and re-reads nothing it already processed.

In Mixpeek that is GET /v1/events. Omit the cursor to start from the beginning of the retention window; pass the one you stored to resume. Pages are up to 1000 events, defaulting to 100, and you can filter to a namespace or an event type.

The important property is that the cursor is yours to persist. Commit it AFTER you have durably processed the page, never before. A consumer that stores the cursor first and then crashes has told itself it processed work it did not, and that is unrecoverable without a full resync.

The two ways a change feed consumer still loses data

Both of these are properties of cursors in general, not of any one vendor, and both are silent.

Resuming outside the retention window

Retention is 90 days. Events older than that are expired rather than archived, so a consumer that has been down longer cannot resume from where it stopped: there is nothing at that position any more. It has to resync current state and start a fresh cursor.

The failure mode to design against is a consumer that has been broken for months, comes back, gets an empty or partial response, and reports success. If your consumer can be down for longer than the retention window, it needs an explicit "how long have I been gone" check before it trusts a resume.

Changing filters while reusing a cursor

This one is genuinely counterintuitive and worth stating precisely: the cursor encodes a position in the organization's overall sequence, not a position within whatever filter you happened to pass.

So if you read with event_type=document.created, store the cursor, then resume with event_type=document.updated and the same cursor, you do not get the updates you skipped. You get updates from that sequence position forward, and everything the new filter would have matched before that point is gone, silently.

Whenever the filter set changes, start a fresh cursor. Treat a cursor as bound to the exact filter combination that produced it.

How do I prove I have not missed anything?

This is the question that separates a feed you can build on from one you hope is working. Sequence position alone does not prove completeness, because a gap and a quiet period look the same from the consumer's side.

Two mechanisms help:

Reconciliation markers. Batch operations emit terminal markers into the

feed, so a consumer can detect that a batch of N documents finished and check that it saw N events for it. A missing marker or a short count is a detectable gap rather than an inference.

Periodic count reconciliation. Independently of the feed, compare the

document count on each side. The feed tells you what changed; a count tells you whether you agree about the total. Disagreement means resync.

Do both. The feed makes incremental sync cheap; the count check is what makes it trustworthy, and it is the one people skip.

What events should I subscribe to?

Start narrower than you think you need. A consumer that filters to the event types it acts on has less to process and a smaller blast radius when something changes. Add types when you have a reason, and remember that adding one means starting a fresh cursor.

Scope matters too. The feed is organization-wide by default and can be filtered to a namespace, which is usually what you want if a downstream system only cares about one workload.

When is a change feed the wrong tool?

You need sub-second reaction. A feed is pull-based. Use webhooks for

latency and the feed for completeness, rather than choosing one.

You only need current state. If a downstream system rebuilds from scratch

on every run, list the documents and skip the feed entirely.

You are replicating everything. If the downstream copy is meant to be a

full mirror, a periodic bulk export is simpler to reason about than a continuously-consumed sequence, and easier to verify.