What is the manifest pattern in object-storage databases?

All the hard coordination problems get pushed into one small object called the manifest. Every other object in the bucket is immutable: log segments and data pages append and never change. The manifest names the current head of the log and the current writer's epoch, and committing means swapping the manifest with a conditional write. Chroma's wal3 documents the protocol end to end: read the manifest's ETag, write your pages, then compare-and-swap the manifest as the commit point.

What happens when two writers try to commit at the same time?

One swap lands and one bounces. The loser's conditional write fails because the manifest changed since it last read it, so its data objects become orphans and the log stays clean. Nothing corrupts and no write silently disappears, which is the improvement over plain S3's last-writer-wins overwrite. A crash before the swap has the same footprint: orphan objects plus an intact manifest, and recovery is deleting the orphans.

Why did databases on S3 only take off after 2024?

S3 added conditional writes in 2024: If-None-Match in August and If-Match compare-and-swap in November. Before that, two writers racing on the same key was last-writer-wins, so any real system needed a database or lock service next to the bucket to serialize commits. Once the bucket itself can reject a stale swap, that coordinator disappears. walgit runs a whole Git server as one binary against S3 with no database, and MVS scopes its own use of the bucket to sealed WAL segments and a durable snapshot frontier, with a full compare-and-swap fencing protocol in flight.

Every New Database on S3 Is Secretly the Same Design

Name: Every New Database on S3 Is Secretly the Same Design
Uploaded: 2026-08-27T20:00:00Z
Duration: 91 s

Multimodal Search in Action

Ethan

August 27, 2026

Summary

Databases built on object storage keep converging on one shape: a pile of immutable objects under a single small mutable manifest. Writes land as objects that never change. The manifest names the current head of the log and the epoch of the writer, and a commit is a compare-and-swap of that one small object. If the swap lands the write happened; if another writer got there first the swap bounces and nothing corrupts. A crash before the swap leaves orphan objects and a clean manifest, so recovery is deleting the orphans. The video walks this shape through four systems that arrived at it independently: Chroma's wal3 write-ahead log, walgit (the Shopify CEO's weekend Git server, one binary on S3 with no database), Neon's Postgres storage, and MVS, Mixpeek's vector store, which ships sealed WAL segments to the bucket and advances its safe-to-delete line only after a snapshot provably commits. The unlock was S3 adding conditional writes in 2024, which let the bucket coordinate its own writers. How ours works: mixpeek.com/mvs

object-storagemanifestcompare-and-swaps3-conditional-writeswrite-ahead-logmvs

Every New Database on S3 Is Secretly the Same Design

Summary

About this video

Frequently Asked Questions