> ## Documentation Index > Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Bulk-submit a whole bucket as N auto-chunked, auto-queued batches > One call that streams the bucket's objects (paginated, de-duplicated), chunks them server-side into `chunk_size`-object batches, and submits each (accept-and-queue). Replaces the client-side chunk loop + cursor pacing — no 409 cursor-overlap, no 5k count-timeout, no manual 429 handling. Poll the returned batch ids / `batch_group_id` for QUEUED -> PROCESSING. ## OpenAPI ````yaml post /v1/buckets/{bucket_identifier}/batches/bulk-submit openapi: 3.1.0 info: title: Mixpeek API description: >- This is the Mixpeek API, providing access to various endpoints for data processing and retrieval. termsOfService: https://mixpeek.com/terms contact: name: Mixpeek Support url: https://mixpeek.com/contact email: info@mixpeek.com version: '0.82' servers: - url: https://api.mixpeek.com description: Production security: - BearerAuth: [] paths: /v1/buckets/{bucket_identifier}/batches/bulk-submit: post: tags: - Bucket Batches summary: Bulk-submit a whole bucket as N auto-chunked, auto-queued batches description: >- One call that streams the bucket's objects (paginated, de-duplicated), chunks them server-side into `chunk_size`-object batches, and submits each (accept-and-queue). Replaces the client-side chunk loop + cursor pacing — no 409 cursor-overlap, no 5k count-timeout, no manual 429 handling. Poll the returned batch ids / `batch_group_id` for QUEUED -> PROCESSING. operationId: >- bulk_submit_batches_v1_buckets__bucket_identifier__batches_bulk_submit_post parameters: - name: bucket_identifier in: path required: true schema: type: string description: The bucket to ingest. title: Bucket Identifier description: The bucket to ingest. requestBody: content: application/json: schema: $ref: '#/components/schemas/BulkSubmitBatchesRequest' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/BulkSubmitBatchesResponse' '400': description: Bad Request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '401': description: Unauthorized content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '403': description: Forbidden content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Not Found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' security: - BearerAuth: [] NamespaceHeader: [] components: schemas: BulkSubmitBatchesRequest: properties: collection_ids: anyOf: - items: type: string type: array - type: 'null' title: Collection Ids description: >- Collections to process. Omit to auto-discover all collections sourced from this bucket (plus downstream dependencies). filters: anyOf: - $ref: '#/components/schemas/LogicalOperator-Input' - type: 'null' description: >- Optional filter to scope which objects are submitted. Omit to submit EVERY object in the bucket (the whole-bucket case). chunk_size: type: integer maximum: 50000 minimum: 1 title: Chunk Size description: >- Objects per batch. Clamped down to your tier's max_batch_size. Default 1000 — a good balance of parallelism and per-batch overhead. Sizing model: each batch runs as its own job whose workers scale at ~1 CPU worker per 500 objects up to a per-job worker ceiling, so 16k-20k objects saturates one job's parallelism — for large corpora (100k+ objects) prefer chunk_size 20000. Larger chunks don't run faster; they only raise the cost of a mid-run failure (progress is still resumable per-object via the processing ledger). Much smaller chunks pay one cluster cold-start each and queue behind your tier's concurrent-batch limit. Full guide: docs.mixpeek.com/operations/batch-ingestion-at-scale. default: 1000 dedup_strategy: $ref: '#/components/schemas/DedupStrategy' description: How already-processed objects are handled (skip/replace/force). default: skip max_objects: anyOf: - type: integer minimum: 1 - type: 'null' title: Max Objects description: >- Optional cap on total objects streamed (safety bound for very large buckets). Omit to submit the whole bucket. type: object title: BulkSubmitBatchesRequest description: >- Submit a WHOLE bucket as N auto-chunked, auto-queued batches in one call. The server streams the bucket's objects (paginated), de-duplicates them, chunks them into ``chunk_size``-object batches, and submits each — so the client makes ONE call instead of a paced chunk-loop. This kills the client-side cursor-overlap (409 storm), the 5k object-count timeout, and the no-whole-bucket frustration at once (phase 4). Each batch is accepted and QUEUED (accept-and-queue admission); poll the returned ``batch_group_id`` / batch ids for QUEUED -> PROCESSING. BulkSubmitBatchesResponse: properties: batch_group_id: type: string title: Batch Group Id description: Correlates all batches created by this call. total_objects: type: integer title: Total Objects description: Distinct objects streamed (post-dedup). total_batches: type: integer title: Total Batches description: Batches created. submitted: type: integer title: Submitted description: Batches accepted/queued successfully. failed: type: integer title: Failed description: Batches that failed admission/submit. chunk_size: type: integer title: Chunk Size description: Effective per-batch object count used. batches: items: $ref: '#/components/schemas/BulkSubmittedBatch' type: array title: Batches description: Per-batch results with status + queue position. type: object required: - batch_group_id - total_objects - total_batches - submitted - failed - chunk_size - batches title: BulkSubmitBatchesResponse description: Result of a whole-bucket bulk submit. ErrorResponse: properties: success: type: boolean title: Success description: Always false for error responses default: false status: type: integer title: Status description: HTTP status code for this error error: $ref: '#/components/schemas/ErrorDetail' description: Error details payload type: object required: - status - error title: ErrorResponse description: Error response model. examples: - error: details: id: ns_123 resource: namespace message: Namespace not found type: NotFoundError status: 404 success: false HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError LogicalOperator-Input: properties: AND: anyOf: - items: anyOf: - $ref: '#/components/schemas/LogicalOperator-Input' - $ref: '#/components/schemas/FilterCondition' type: array - type: 'null' title: And description: Logical AND operation - all conditions must be true example: - field: name operator: eq value: John - field: age operator: gte value: 30 OR: anyOf: - items: anyOf: - $ref: '#/components/schemas/LogicalOperator-Input' - $ref: '#/components/schemas/FilterCondition' type: array - type: 'null' title: Or description: Logical OR operation - at least one condition must be true example: - field: status operator: eq value: active - field: role operator: eq value: admin NOT: anyOf: - items: anyOf: - $ref: '#/components/schemas/LogicalOperator-Input' - $ref: '#/components/schemas/FilterCondition' type: array - type: 'null' title: Not description: Logical NOT operation - all conditions must be false example: - field: department operator: eq value: HR - field: location operator: eq value: remote case_sensitive: anyOf: - type: boolean - type: 'null' title: Case Sensitive description: Whether to perform case-sensitive matching default: false example: true additionalProperties: true type: object title: LogicalOperator description: >- Represents a logical operation (AND, OR, NOT) on filter conditions. Allows nesting with a defined depth limit. Also supports shorthand syntax where field names can be passed directly as key-value pairs for equality filtering (e.g., {"metadata.title": "value"}). DedupStrategy: type: string enum: - skip - replace - force title: DedupStrategy description: |- Controls how duplicate objects are handled during batch processing. Dedup is scoped to (bucket_id, collection_id): an object is considered a duplicate if the target collection already has documents produced from the same source object in any prior batch. BulkSubmittedBatch: properties: batch_id: type: string title: Batch Id status: type: string title: Status object_count: type: integer title: Object Count queue_position: anyOf: - type: integer - type: 'null' title: Queue Position error: anyOf: - type: string - type: 'null' title: Error type: object required: - batch_id - status - object_count title: BulkSubmittedBatch description: One batch in a bulk-submit group. ErrorDetail: properties: message: type: string title: Message description: Human-readable error message type: type: string title: Type description: Stable error type identifier (machine-readable) code: anyOf: - type: string - type: 'null' title: Code description: >- Fine-grained error code for programmatic handling (e.g., namespace_name_taken, feature_extractor_not_found). Present only when consumers may need to branch on a specific error condition. details: anyOf: - additionalProperties: true type: object - type: 'null' title: Details description: >- Optional structured details to help debugging (validation errors, IDs, etc.) type: object required: - message - type title: ErrorDetail description: Error detail model. ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError FilterCondition: properties: field: type: string title: Field description: Field name to filter on operator: $ref: '#/components/schemas/FilterOperator' description: Comparison operator default: eq value: anyOf: - $ref: '#/components/schemas/DynamicValue' - {} title: Value description: Value to compare against type: object required: - field - value title: FilterCondition description: |- Represents a single filter condition. Attributes: field: The field to filter on operator: The comparison operator value: The value to compare against FilterOperator: description: Supported filter operators across database implementations. enum: - eq - ne - gt - lt - gte - lte - in - nin - contains - starts_with - ends_with - regex - exists - is_null - text - phrase - geo_radius - geo_bounding_box - geo_polygon title: FilterOperator type: string DynamicValue: description: A value that should be dynamically resolved from the query request. properties: type: type: string const: dynamic title: Type default: dynamic field: type: string title: Field description: >- The dot-notation path to the value in the runtime query request, e.g., 'inputs.user_id' examples: - inputs.query_text - filters.AND[0].value required: - field title: DynamicValue type: object securitySchemes: BearerAuth: type: http scheme: bearer description: >- Mixpeek API key, sent as `Authorization: Bearer mxp_sk_...`. Create one in Studio under Settings → API Keys, or with an admin key via `POST /v1/organizations/users/{user_email}/api-keys`. A missing header returns 403; an invalid or revoked key returns 401. NamespaceHeader: type: apiKey in: header name: X-Namespace description: >- Namespace id (`ns_...`), not the namespace name. This scopes the request rather than authenticating it, and it is required on every operation marked `x-mixpeek-namespace-scoped`. ````