Batch Processing
Multi-tier DAG processing that transforms bucket objects into searchable documents with feature extraction
Why do anything?
Raw objects need feature extraction to become searchable. Without batch processing, you can't generate embeddings or enrich content at scale.
Why now?
AI search requires vector embeddings. Manual processing doesn't scale.
Why this feature?
Multi-tier DAG processing handles complex pipelines: Tier 0 (bucket→collection) and Tier N (collection→collection) with Celery workers and Ray inference.
How It Works
Batch processing uses a multi-tier DAG architecture. Tier 0 processes bucket objects, Tier N processes upstream collection outputs.
Batch Creation
Create batch record, validate source and collection config
Task Routing
Route to Celery process_tier queue based on tier level
Feature Extraction
Ray engine runs feature extractor, generates embeddings
Document Storage
Documents stored in the Mixpeek Vector Store with vectors indexed
Why This Approach
DAG enables complex multi-stage pipelines (e.g., video→frames→faces→embeddings). Celery provides reliable task execution. Ray handles ML inference at scale.
Where This Is Used
Recent updates
Full changelog- Oct 13, 2026Image extraction on CPU batch workers runs about 50 times fasterA large image batch on CPU workers had been embedding about 185 images an hour. The model ran in bfloat16 on processors with no hardware support for it, so each image took about 20 seconds, and only one of the batch's worker CPUs was ever put to work. CPU batches now embed in fp32 and start one model copy per worker CPU from the beginning. In our test at the same worker shape, 3,414 images took 20 minutes, about 10,250 an hour, where the previous code managed about 180 an hour. GPU batches are unchanged.
- Oct 13, 2026Failed audio fingerprint and Vertex embedding objects say why they failedWhen the audio fingerprint extractor could not read a file, the batch failed with a message saying it returned 0 rows, followed by a list of possible causes. Each object now carries a named error: an unsupported format, a video with no audio track, or a decode failure with FFmpeg's last lines of output. A video with no audio track is not retried. When a Vertex multimodal embedding fails, the object's error now carries the provider's message, where it had read "None/empty". A finished batch also takes its extracted and failed counts from its per-object record, so a row that failed extraction is counted as failed.
- Sep 26, 2026Batch status is right when a tier runs several extractorsWhen one tier ran two extractors, such as passthrough and text on the same bucket, the first to finish could close the tier, and the batch ended as completed with errors saying no documents were written, then the documents arrived a minute later. Each extractor job now reports its own job id, so a tier waits for all of them. A batch that was re-dispatched from the queue also marks its objects completed; they used to stay pending.
- Sep 27, 2026Small CPU batches start soonerA CPU batch sized its workers from the processing profile's maximum rather than from the number of objects, so a one-document batch started several workers and could wait for capacity it never needed. Workers are now sized from the objects in the batch and from what the shared pool can hold.