Everyone Posts WAL-on-S3 Designs. Here Are Production Counters.
Summary
Half the internet is designing write-ahead logs on object storage; this is one running in production with the counters to prove it. Scraped from Mixpeek's primary at the end of August: 7,134 WAL segments shipped to object storage over 2.5 days of uptime, roughly one every 31 seconds under real load, with about 2.9 GB of log behind them. Recovery is snapshot plus replay of the shipped chain, and replicas tail the same objects. The parts nobody blogs about: garbage collection deletes a segment only when its writes are provably inside a committed snapshot, so a stalled snapshot pipeline deletes nothing at all, by design; and every ship checks a leader lease and the writer's boot identity so a deposed writer cannot fork the chain. The counters: mixpeek.com/mvs
About this video
Half the internet is designing write-ahead logs on object storage; this is one running in production with the counters to prove it. Scraped from Mixpeek's primary at the end of August: 7,134 WAL segments shipped to object storage over 2.5 days of uptime, roughly one every 31 seconds under real load, with about 2.9 GB of log behind them. Recovery is snapshot plus replay of the shipped chain, and replicas tail the same objects. The parts nobody blogs about: garbage collection deletes a segment only when its writes are provably inside a committed snapshot, so a stalled snapshot pipeline deletes nothing at all, by design; and every ship checks a leader lease and the writer's boot identity so a deposed writer cannot fork the chain. The counters: mixpeek.com/mvs