> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Batch ingestion at scale
> How to size, submit, and monitor large batch ingestions — chunking, concurrency, automatic recovery, and the limits that apply at each plan tier.
Processing a large bucket (100k–5M objects) is a batching decision, not one API call. This page gives you the sizing model the platform itself uses, so you can pick a chunk size and batch count with numbers instead of guesses.
## Two submission paths
**`POST /v1/collections/{collection_id}/trigger`** creates **one batch containing every object** in the source bucket. There is no server-side splitting. Use it for buckets up to a few thousand objects.
**`POST /v1/buckets/{bucket_id}/batches/bulk-submit`** splits the bucket into batches of `chunk_size` objects (default 1,000, maximum 50,000) and submits each one. Use it for everything larger.
Do not trigger a collection over a very large bucket. The single batch it creates
runs as one job with a fixed worker ceiling and a 24-hour execution deadline, so
it under-parallelizes and can time out. Bulk-submit is the large-corpus path.
```python Python theme={null}
from mixpeek import Mixpeek
mp = Mixpeek(api_key="YOUR_API_KEY")
result = mp.buckets.batches.bulk_submit(
bucket_id="bkt_123",
collection_ids=["col_abc"],
chunk_size=20000,
)
print(result.batch_ids)
```
```javascript JavaScript theme={null}
import { Mixpeek } from "@mixpeek/sdk";
const mp = new Mixpeek({ apiKey: "YOUR_API_KEY" });
const result = await mp.buckets.batches.bulkSubmit({
bucketId: "bkt_123",
collectionIds: ["col_abc"],
chunkSize: 20000,
});
console.log(result.batchIds);
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/bkt_123/batches/bulk-submit" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace" \
-H "Content-Type: application/json" \
-d '{"collection_ids": ["col_abc"], "chunk_size": 20000}'
```
## Picking a chunk size
Each batch runs as its own job. A job scales its workers with object count — roughly one CPU worker per 500 objects — up to a per-job worker ceiling. A batch of 16,000–20,000 objects saturates one job's parallelism. Larger batches do not run faster; they only raise the cost of a mid-run failure.
Going too small hurts the other way. Every batch pays a cluster cold-start, and only a fixed number of batches run at once (see the tier table). 1,200 batches of 1,000 objects serialize into deep queue waves and spend a large share of wall-clock on startup.
| Corpus size | Suggested `chunk_size` | Resulting batches |
| ----------- | ---------------------- | ----------------- |
| under 5k | one trigger call | 1 |
| 5k – 100k | 5,000–10,000 | 1–20 |
| 100k – 1M | 20,000 | 5–50 |
| 1M+ | 20,000–50,000 | 20–100+ |
Calibrate before you commit: run one small batch (1,000 objects) and one at your target size, compare objects/hour and credits from the [Billing usage page](/docs/platform/billing), then extrapolate. Media mix changes cost more than batch size does.
## Concurrency and queueing
Submission is accept-and-queue. Batches past your concurrency limit park as `QUEUED` and promote automatically, first-in-first-out, as running batches finish. You do not pace submissions yourself.
| Plan | Max objects per batch | Concurrent running batches |
| ---------- | --------------------- | -------------------------- |
| Free | 10,000 | 1 |
| Pro | 50,000 | 5 |
| Team | 500,000 | 10 |
| Enterprise | 5,000,000 | 50 |
Submission returns `429` only when a queue is truly full. Each running job also has a 24-hour execution deadline.
## Automatic recovery
The platform heals transient failures without your involvement:
* Failed tasks retry 3 times with exponential backoff.
* A worker that dies mid-task has its work redelivered to another worker.
* Reapers sweep every 1–10 minutes for orphaned, stuck, or stalled batches. They re-drive recoverable ones (bounded retries) and mark the rest failed.
* Failures tagged safe-to-resubmit are resubmitted automatically every 30 minutes.
* Batches that failed on transient errors retry hourly, up to 3 times, with 1/2/4-hour backoff.
Progress is durable per object, not per batch. Each object is recorded in a processing ledger as it completes, so resubmitting a partially-failed batch skips finished objects and reprocesses only the remainder. You do not pay for extraction twice.
Some failures never self-heal by design: authentication errors, schema validation errors, and hard provider quota exhaustion (for example, an exhausted OpenAI balance). These end the affected batches in `FAILED` or `COMPLETED_WITH_ERRORS` and wait for you to fix the cause, then [resubmit](/docs/api-reference/bucket-batches/submit-batch-for-processing).
## Monitoring a run
Watch progress through the API, not by polling logs:
* `GET /v1/buckets/{bucket_id}/batches/{batch_id}` — status, per-tier progress, heartbeat freshness, `documents_written`, `failed_object_count`, and `error_summary`.
* `GET .../batches/{batch_id}/failed-documents` — every failed object with its error, for targeted retries.
* [Batch diagnostics](/docs/troubleshoot/batch-diagnostics) — deeper triage when a batch misbehaves.
Enable the batch [system alerts](/docs/api-reference/alerts) — `batch_failed`, `batch_stalled`, and `batch_error_rate` — before a large run. They evaluate every 5 minutes and notify in-app plus any webhook, Slack, or email channel you configure. Without them, a failed batch is visible only if you go looking for it.
A batch can finish as `COMPLETED_WITH_ERRORS`. Treat that as a work list, not a
verdict: read `failed-documents`, fix the cause, resubmit, and the ledger skips
everything already done.