Back to Videos
10M Objects/Day Broke Our Pipeline — Here's How We Fixed It
0:60
Short Form
Ethan
April 14, 2026
Summary
We hit 10 million objects per day at Mixpeek. Videos, images, PDFs — all hitting the pipeline at once. Here's what broke: 1. Celery queue saturated at 2M jobs — backed up for hours
short-form
About this video
We hit 10 million objects per day at Mixpeek. Videos, images, PDFs — all hitting the pipeline at once. Here's what broke: 1. Celery queue saturated at 2M jobs — backed up for hours 2. Ray OOM on files over 500MB 3. 47-second cold starts costing $14K/month The fixes: priority queues, streaming extraction, pre-warmed clusters. The queue is the ceiling — fix the ceiling. See the architecture: mixpeek.com/mvs #DataEngineering #MLOps #ScaleProblems #AI #Infrastructure #RayComputing #StartupEngineering