Batch Processing
Multi-tier DAG processing that transforms bucket objects into searchable documents with feature extraction
Why do anything?
Raw objects need feature extraction to become searchable. Without batch processing, you can't generate embeddings or enrich content at scale.
Why now?
AI search requires vector embeddings. Manual processing doesn't scale.
Why this feature?
Multi-tier DAG processing handles complex pipelines: Tier 0 (bucket→collection) and Tier N (collection→collection) with Celery workers and Ray inference.
How It Works
Batch processing uses a multi-tier DAG architecture. Tier 0 processes bucket objects, Tier N processes upstream collection outputs.
Batch Creation
Create batch record, validate source and collection config
Task Routing
Route to Celery process_tier queue based on tier level
Feature Extraction
Ray engine runs feature extractor, generates embeddings
Document Storage
Documents stored in Qdrant with vectors indexed
Why This Approach
DAG enables complex multi-stage pipelines (e.g., video→frames→faces→embeddings). Celery provides reliable task execution. Ray handles ML inference at scale.
Where This Is Used
Recent updates
Full changelog- Jul 26, 2026A batch can no longer report success while writing nothingThe universal fast-path write leg called the vector store and discarded the result, so a batch could log per-object write lines and finish COMPLETED without checking that anything persisted. A successful call is not a successful write: the store silently drops a point that supplies fewer named vectors than the namespace declares. One observed batch processed 47 objects, reported 464 documents created and 47 write lines, and finished COMPLETED with zero documents in either store. The write result is now checked, a zero-point upsert logs the datasink-drop diagnosis instead of passing silently, and the guardrail is anchored on documents actually produced rather than points sent. Three independent success indicators were agreeing on a false result; all three now fail loudly.
- Jul 24, 2026Resubmit a failed batch directly from StudioWhen a batch ends in a failed state, Studio now shows a Resubmit action so you can retry it in place, without rebuilding the request or dropping to the API. A failed ingestion or processing run is one click from another attempt.