Skip to main content
Processing a large bucket (100k–5M objects) is a batching decision, not one API call. This page gives you the sizing model the platform itself uses, so you can pick a chunk size and batch count with numbers instead of guesses.

​
Two submission paths

POST /v1/collections/{collection_id}/trigger creates one batch containing every object in the source bucket. There is no server-side splitting. Use it for buckets up to a few thousand objects. POST /v1/buckets/{bucket_id}/batches/bulk-submit splits the bucket into batches of chunk_size objects (default 1,000, maximum 50,000) and submits each one. Use it for everything larger.
Do not trigger a collection over a very large bucket. The single batch it creates runs as one job with a fixed worker ceiling and a 24-hour execution deadline, so it under-parallelizes and can time out. Bulk-submit is the large-corpus path.

​
Picking a chunk size

Each batch runs as its own job. A job scales its workers with object count — roughly one CPU worker per 500 objects — up to a per-job worker ceiling. A batch of 16,000–20,000 objects saturates one job’s parallelism. Larger batches do not run faster; they only raise the cost of a mid-run failure. Going too small hurts the other way. Every batch pays a cluster cold-start, and only a fixed number of batches run at once (see the tier table). 1,200 batches of 1,000 objects serialize into deep queue waves and spend a large share of wall-clock on startup. Calibrate before you commit: run one small batch (1,000 objects) and one at your target size, compare objects/hour and credits from the Billing usage page, then extrapolate. Media mix changes cost more than batch size does.

​
Concurrency and queueing

Submission is accept-and-queue. Batches past your concurrency limit park as QUEUED and promote automatically, first-in-first-out, as running batches finish. You do not pace submissions yourself. Submission returns 429 only when a queue is truly full. Each running job also has a 24-hour execution deadline.

​
Automatic recovery

The platform heals transient failures without your involvement:
  • Failed tasks retry 3 times with exponential backoff.
  • A worker that dies mid-task has its work redelivered to another worker.
  • Reapers sweep every 1–10 minutes for orphaned, stuck, or stalled batches. They re-drive recoverable ones (bounded retries) and mark the rest failed.
  • Failures tagged safe-to-resubmit are resubmitted automatically every 30 minutes.
  • Batches that failed on transient errors retry hourly, up to 3 times, with 1/2/4-hour backoff.
Progress is durable per object, not per batch. Each object is recorded in a processing ledger as it completes, so resubmitting a partially-failed batch skips finished objects and reprocesses only the remainder. You do not pay for extraction twice. Some failures never self-heal by design: authentication errors, schema validation errors, and hard provider quota exhaustion (for example, an exhausted OpenAI balance). These end the affected batches in FAILED or COMPLETED_WITH_ERRORS and wait for you to fix the cause, then resubmit.

​
Monitoring a run

Watch progress through the API, not by polling logs:
  • GET /v1/buckets/{bucket_id}/batches/{batch_id} — status, per-tier progress, heartbeat freshness, documents_written, failed_object_count, and error_summary.
  • GET .../batches/{batch_id}/failed-documents — every failed object with its error, for targeted retries.
  • Batch diagnostics — deeper triage when a batch misbehaves.
Enable the batch system alerts — batch_failed, batch_stalled, and batch_error_rate — before a large run. They evaluate every 5 minutes and notify in-app plus any webhook, Slack, or email channel you configure. Without them, a failed batch is visible only if you go looking for it.
A batch can finish as COMPLETED_WITH_ERRORS. Treat that as a work list, not a verdict: read failed-documents, fix the cause, resubmit, and the ledger skips everything already done.