> ## Documentation Index > Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Execute Retriever (Auto-Optimized) > Execute a retriever and return matching documents. The pipeline is automatically optimized before execution for best performance. **Automatic Optimization:** Your pipeline stages are automatically transformed for optimal performance: - Filters pushed down to reduce expensive operations - Redundant stages merged or eliminated - Grouping operations pushed to database layer (10-100x faster) - Operations reordered for efficiency **Streaming Support:** Set stream=true in the request body to receive real-time stage updates via SSE: - Response uses text/event-stream content type - Each stage emits stage_start and stage_complete events - Final event contains complete results and pagination - Useful for progress tracking and debugging **Response Includes (when stream=false):** - documents: Final matching documents - pagination: Pagination metadata - stage_statistics: Per-stage execution metrics - budget: Credit/time consumption - optimization_applied: Whether optimizations were applied - optimization_summary: Details about transformations (when applied) **Optimization Summary Example:** ```json { "optimization_applied": true, "optimization_summary": { "original_stage_count": 5, "optimized_stage_count": 3, "optimization_time_ms": 8.2, "rules_applied": ["push_down_filters", "group_by_push_down"], "stage_reduction_pct": 40.0 } } ``` Use the /explain endpoint to see the optimized execution plan before running. ## OpenAPI ````yaml post /v1/retrievers/{retriever_id}/execute openapi: 3.1.0 info: title: Mixpeek API description: >- This is the Mixpeek API, providing access to various endpoints for data processing and retrieval. termsOfService: https://mixpeek.com/terms contact: name: Mixpeek Support url: https://mixpeek.com/contact email: info@mixpeek.com version: '0.82' servers: - url: https://api.mixpeek.com description: Production security: [] paths: /v1/retrievers/{retriever_id}/execute: post: tags: - Retrievers summary: Execute Retriever (Auto-Optimized) description: >- Execute a retriever and return matching documents. The pipeline is automatically optimized before execution for best performance. **Automatic Optimization:** Your pipeline stages are automatically transformed for optimal performance: - Filters pushed down to reduce expensive operations - Redundant stages merged or eliminated - Grouping operations pushed to database layer (10-100x faster) - Operations reordered for efficiency **Streaming Support:** Set stream=true in the request body to receive real-time stage updates via SSE: - Response uses text/event-stream content type - Each stage emits stage_start and stage_complete events - Final event contains complete results and pagination - Useful for progress tracking and debugging **Response Includes (when stream=false):** - documents: Final matching documents - pagination: Pagination metadata - stage_statistics: Per-stage execution metrics - budget: Credit/time consumption - optimization_applied: Whether optimizations were applied - optimization_summary: Details about transformations (when applied) **Optimization Summary Example:** ```json { "optimization_applied": true, "optimization_summary": { "original_stage_count": 5, "optimized_stage_count": 3, "optimization_time_ms": 8.2, "rules_applied": ["push_down_filters", "group_by_push_down"], "stage_reduction_pct": 40.0 } } ``` Use the /explain endpoint to see the optimized execution plan before running. operationId: execute_retriever_v1_retrievers__retriever_id__execute_post parameters: - name: retriever_id in: path required: true schema: type: string description: >- Retriever ID or name. Pipeline will be automatically optimized before execution. title: Retriever Id description: >- Retriever ID or name. Pipeline will be automatically optimized before execution. - name: return_presigned_urls in: query required: false schema: type: boolean description: >- Generate presigned URLs for S3-backed blobs and url-shaped fields. Also accepted as a body field — if either source is true, presigning is enabled. default: false title: Return Presigned Urls description: >- Generate presigned URLs for S3-backed blobs and url-shaped fields. Also accepted as a body field — if either source is true, presigning is enabled. - name: return_vectors in: query required: false schema: type: boolean description: >- Include vector embeddings in result documents. Also accepted as a body field — if either source is true, vectors are returned. default: false title: Return Vectors description: >- Include vector embeddings in result documents. Also accepted as a body field — if either source is true, vectors are returned. - name: explain in: query required: false schema: type: boolean description: >- Return the inline `explain_plan` (per-stage timings + optimization summary) in the response. Also accepted as a body field — if either source is true, the plan is included. default: false title: Explain description: >- Return the inline `explain_plan` (per-stage timings + optimization summary) in the response. Also accepted as a body field — if either source is true, the plan is included. - name: include_legacy_results in: query required: false schema: type: boolean description: >- DEPRECATED. Restore the legacy `results` alias (a byte-for-byte copy of `documents`). Off by default — `results` previously duplicated the entire document array in every response (~half the payload). Prefer `documents`; use this only as a temporary migration shim. default: false title: Include Legacy Results description: >- DEPRECATED. Restore the legacy `results` alias (a byte-for-byte copy of `documents`). Off by default — `results` previously duplicated the entire document array in every response (~half the payload). Prefer `documents`; use this only as a temporary migration shim. requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ExecuteRetrieverRequest' description: >- Execution request with inputs, filters, pagination, and optional stream parameter. Set stream=true to receive real-time stage updates via Server-Sent Events. responses: '200': description: >- Execution results with documents, pagination, statistics, and optimization details. When stream=true, returns Server-Sent Events. When stream=false, returns JSON response. content: application/json: schema: {} '400': description: Bad Request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '401': description: Unauthorized content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '403': description: Forbidden content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Not Found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' components: schemas: ExecuteRetrieverRequest: properties: inputs: additionalProperties: true type: object title: Inputs description: >- Runtime inputs for the retriever mapped to the input schema. Keys must match the retriever's input_schema field names. Values depend on field types (text, vector, filters, etc.). REQUIRED unless all retriever inputs have defaults. Common input keys: - 'query': Text search query - 'embedding': Pre-computed vector for search - 'top_k': Number of results to return - 'min_score': Minimum relevance threshold - Any custom fields defined in input_schema **Template Syntax** (Jinja2): Namespaces (uppercase or lowercase): - `INPUT` / `input`: Query inputs (e.g., `{{INPUT.query}}`) - `DOC` / `doc`: Document fields (e.g., `{{DOC.payload.title}}`) - `CONTEXT` / `context`: Execution context - `STAGE` / `stage`: Stage configuration - `SECRET` / `secret`: Vault secrets (e.g., `{{SECRET.api_key}}`) Accessing Data: - Dot notation: `{{DOC.payload.metadata.title}}` - Bracket notation: `{{DOC.payload['special-key']}}` - Array index: `{{DOC.items[0]}}`, `{{DOC.tags[2]}}` - Array first/last: `{{DOC.items | first}}`, `{{DOC.items | last}}` Array Operations: - Iterate: `{% for item in DOC.tags %}{{item}}{% endfor %}` - Extract key: `{{DOC.items | map(attribute='name') | list}}` - Join: `{{DOC.tags | join(', ')}}` - Length: `{{DOC.items | length}}` - Slice: `{{DOC.items[:5]}}` Conditionals: - If: `{% if DOC.status == 'active' %}...{% endif %}` - If-else: `{% if DOC.score > 0.8 %}high{% else %}low{% endif %}` - Ternary: `{{'yes' if DOC.enabled else 'no'}}` Built-in Functions: `max`, `min`, `abs`, `round`, `ceil`, `floor` Custom Filters: `slugify` (URL-safe), `bool` (truthy coercion), `tojson` (JSON encode) S3 URLs: Internal S3 URLs (s3://bucket/key) are automatically presigned when accessed via DOC namespace. examples: - query: artificial intelligence top_k: 25 - min_score: 0.7 query: customer feedback top_k: 50 - category: blog embedding: - 0.1 - 0.2 - 0.3 top_k: 10 filters: anyOf: - additionalProperties: true type: object - type: 'null' title: Filters description: >- Optional ad-hoc filters applied at execution time. Merged (AND) with any filters already defined in the retriever's stages. Uses the standard LogicalOperator format: {"AND": [{"field": "brand", "operator": "eq", "value": "Acme"}]}. Supports operators: eq, ne, in, nin, gt, gte, lt, lte, contains, exists, is_null. examples: - AND: - field: brand operator: eq value: Acme - AND: - field: status operator: in value: - active - pending - field: score operator: gte value: 0.5 pagination: oneOf: - $ref: '#/components/schemas/OffsetPaginationParams' - $ref: '#/components/schemas/CursorPaginationParams' - $ref: '#/components/schemas/ScrollPaginationParams' - $ref: '#/components/schemas/KeysetPaginationParams' title: Pagination description: > Pagination strategy configuration. Defaults to cursor-based pagination with limit=20. IMPORTANT: Pagination params do NOT support template variables ({{INPUT.x}} or {{DOCUMENT.x}}). Pagination is a request-level parameter for slicing results, separate from pipeline business logic. Pass cursor/limit values directly from your client code. Cursor values come from the previous response's pagination.cursor field. Supported Methods: - CURSOR (default): Best for infinite scroll, stateless, opaque token - KEYSET: Most efficient, requires stable sort, stateless - OFFSET: Traditional page numbers, can have drift issues - SCROLL: Server-side state, best for bulk exports Use CURSOR for: - Infinite scroll UIs (mobile apps, feeds, timelines) - Real-time updates where consistency matters - When you can't jump to arbitrary pages Use KEYSET for: - Maximum performance with large result sets - Stable sort fields (e.g., score DESC, id ASC) - When you need truly stateless pagination Use OFFSET for: - Traditional page UIs with page numbers - When users need to jump to specific pages - Smaller result sets where drift is acceptable Use SCROLL for: - Bulk exports or processing large datasets - When you need to iterate through all results - Background jobs with progress tracking Example (cursor - first page): {"method": "cursor", "limit": 20, "cursor": null} Example (cursor - next page, using cursor from previous response): {"method": "cursor", "limit": 20, "cursor": "eyJvZmZzZXQiOjIwfQ=="} Example (offset): {"method": "offset", "page_size": 25, "page_number": 2} Example (keyset): {"method": "keyset", "limit": 20, "after": {"score": 0.73, "id": "doc_20"}} discriminator: propertyName: method mapping: cursor: $ref: '#/components/schemas/CursorPaginationParams' keyset: $ref: '#/components/schemas/KeysetPaginationParams' offset: $ref: '#/components/schemas/OffsetPaginationParams' scroll: $ref: '#/components/schemas/ScrollPaginationParams' stream: type: boolean title: Stream description: >- Enable streaming execution to receive real-time stage updates via Server-Sent Events (SSE). NOT REQUIRED - defaults to False for standard execution. When stream=True: - Response uses text/event-stream content type - Each stage completion emits a StreamStageEvent - Events include: stage_start, stage_complete, stage_error, execution_complete - Clients receive intermediate results and statistics as stages execute - Useful for progress tracking, debugging, and partial result display When stream=False (default): - Response returns after all stages complete - Returns a single RetrieverExecutionResponse with final results - Lower overhead for simple queries Use streaming when: - You want to show real-time progress to users - You need to display intermediate results - Pipeline has many stages or long-running operations - Debugging or monitoring pipeline performance Example streaming client (JavaScript): ```javascript const eventSource = new EventSource('/v1/retrievers/ret_123/execute?stream=true'); eventSource.onmessage = (event) => { const stageEvent = JSON.parse(event.data); if (stageEvent.event_type === 'stage_complete') { console.log(`Stage ${stageEvent.stage_name} completed`); console.log(`Documents: ${stageEvent.documents.length}`); } }; ``` Example streaming client (Python): ```python import requests response = requests.post('/v1/retrievers/ret_123/execute', json={'inputs': {...}, 'stream': True}, stream=True) for line in response.iter_lines(): if line.startswith(b'data: '): event = json.loads(line[6:]) print(f"Stage {event['stage_name']}: {event['event_type']}") ``` default: false examples: - false - true expand: anyOf: - items: type: string type: array - type: 'null' title: Expand description: >- OPTIONAL. List of fields containing document IDs to resolve inline. Referenced documents are fetched and attached under an '_expanded' key in each result document. Supports dot-notation for nested fields (e.g., 'items.product_id'). Max 50 unique references per request. Depth is limited to 1 (no recursive expansion). examples: - - customer_id - - customer_id - items.product_id skip_cache: type: boolean title: Skip Cache description: >- OPTIONAL. Bypass stage result cache for this execution. When True, all stages execute fresh without cache lookup. Useful after corpus updates, retriever config changes, or engine deploys. Results are still written to cache for future requests. default: false examples: - false - true return_presigned_urls: type: boolean title: Return Presigned Urls description: >- Generate presigned URLs for S3-backed blobs and url-shaped fields in result documents. Also accepted as a `return_presigned_urls` query parameter; if either source is true, presigning is enabled. default: false examples: - false - true return_vectors: type: boolean title: Return Vectors description: >- Include vector embeddings in result documents. Also accepted as a `return_vectors` query parameter; if either source is true, vectors are returned. default: false examples: - false - true write_token: anyOf: - type: string - type: 'null' title: Write Token description: >- OPTIONAL. Pass the `write_token` returned by a prior direct upsert (options.write_token=true) to get read-your-writes consistency: the read is routed to the primary shard, where your just-written document is immediately searchable, instead of an eventually-consistent replica that can lag several seconds behind. Omit for normal (eventual) reads. explain: type: boolean title: Explain description: >- OPTIONAL. When true, the response includes `explain_plan` — the query profile for THIS execution: per-stage timings + input/output counts, MVS execution stats, and the optimizer summary (and, once wired, the shard ExecutionTrace: chosen legs, fusion, push-downs, nprobe, served/shadow planner). Analogous to SQL `EXPLAIN ANALYZE` — the query still runs and returns documents; the plan is attached alongside. The same profile is auto-logged for every execution regardless of this flag (so Studio can read it); `explain=true` simply returns it inline in the response. default: false type: object title: ExecuteRetrieverRequest description: |- Request to execute a retriever with optional streaming support. Inherits all fields from RetrieverExecutionRequest including: - inputs: Runtime input values matching the retriever's input_schema - pagination: Pagination configuration (cursor, offset, etc.) - stream: Enable SSE streaming for real-time stage updates Streaming Execution (stream=True): When streaming is enabled, the response uses Server-Sent Events (SSE) format with Content-Type: text/event-stream. Each stage emits events as it executes: Event Types: - stage_start: Emitted when a stage begins execution - stage_complete: Emitted when a stage finishes with results - stage_error: Emitted if a stage encounters an error - execution_complete: Emitted after all stages finish successfully - execution_error: Emitted if the entire execution fails Each event is a StreamStageEvent containing: - event_type: The type of event - execution_id: Unique execution identifier - stage_name: Human-readable stage name - stage_index: Zero-based stage position - total_stages: Total number of stages - documents: Intermediate results (for stage_complete) - statistics: Stage metrics (duration, counts, etc.) - budget_used: Cumulative resource consumption Example streaming client: ```python response = requests.post( '/v1/retrievers/{id}/execute', json={'inputs': {...}, 'stream': True}, stream=True ) for line in response.iter_lines(): if line.startswith(b'data: '): event = json.loads(line[6:]) print(f"{event['event_type']}: {event.get('stage_name')}") ``` Standard Execution (stream=False, default): Returns a single ExecuteRetrieverResponse with final documents, pagination, and aggregate statistics after all stages complete. examples: - description: Simple query inputs: query: artificial intelligence trends top_k: 25 - description: Query with stage-driven filtering via inputs inputs: categories: - AI - ML min_score: 0.7 published_after: '2024-01-01' query: machine learning top_k: 100 - description: Advanced query with multiple input parameters inputs: date_range: end: '2024-12-31' start: '2024-01-01' query: customer testimonials sentiment: positive top_k: 20 - description: Infinite scroll with cursor pagination (first page) inputs: query: AI trends top_k: 50 pagination: limit: 20 method: cursor - description: Infinite scroll (next page using cursor from previous response) inputs: query: AI trends top_k: 50 pagination: cursor: eyJvZmZzZXQiOjIwfQ== limit: 20 method: cursor - description: Traditional pagination with page numbers inputs: query: ML papers pagination: method: offset page_number: 2 page_size: 25 - description: High-performance keyset pagination inputs: query: customer reviews top_k: 100 pagination: after: id: doc_20 score: 0.73 limit: 20 method: keyset - description: Scroll pagination for bulk export inputs: query: all documents pagination: limit: 100 method: scroll scroll_ttl: 300 ErrorResponse: properties: success: type: boolean title: Success description: Always false for error responses default: false status: type: integer title: Status description: HTTP status code for this error error: $ref: '#/components/schemas/ErrorDetail' description: Error details payload type: object required: - status - error title: ErrorResponse description: Error response model. examples: - error: details: id: ns_123 resource: namespace message: Namespace not found type: NotFoundError status: 404 success: false HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError OffsetPaginationParams: properties: method: type: string const: offset title: Method description: Constant identifying offset pagination (REQUIRED). default: offset page_size: type: integer maximum: 500 minimum: 1 title: Page Size description: 'Number of documents per page (REQUIRED). Default: 10.' default: 10 page_number: type: integer minimum: 1 title: Page Number description: '1-based page index to retrieve (REQUIRED). Default: 1.' default: 1 type: object title: OffsetPaginationParams description: |- Offset-based pagination using page number sizing. Best for: Traditional page UIs with page number navigation How it works: - Uses page numbers (1, 2, 3...) and page size - Calculates offset as: (page_number - 1) * page_size - Simple and familiar for users - Can jump to any page directly Tradeoffs: - Can have "page drift" if data changes between requests - Example: Items added/deleted causes duplicates or gaps - Less efficient for large offsets (database must skip N rows) Use when: - Building traditional page-numbered UIs - Users need to jump to specific pages - Result set is relatively stable - Working with smaller datasets Example: Page 1: {"method": "offset", "page_size": 25, "page_number": 1} Page 2: {"method": "offset", "page_size": 25, "page_number": 2} CursorPaginationParams: properties: method: type: string const: cursor title: Method description: Constant identifying cursor pagination (REQUIRED). default: cursor limit: type: integer maximum: 500 minimum: 1 title: Limit description: >- Maximum number of documents to return per page (REQUIRED). Default: 10. default: 10 cursor: anyOf: - type: string - type: 'null' title: Cursor description: >- Opaque base64 cursor from previous response (OPTIONAL). null for first page, then use cursor from response.pagination.cursor type: object title: CursorPaginationParams description: >- Cursor-based pagination referencing last seen position. Best for: Infinite scroll UIs, mobile apps, real-time feeds How it works: - First request: cursor=null - Response includes next cursor token - Next request: pass cursor from previous response - Stateless: no server-side state - Consistent: no duplicates/gaps even with concurrent writes Use when: - Building infinite scroll interfaces - Users scroll through results sequentially - You need consistency across pages - You don't need to jump to arbitrary pages Example flow: 1. Request: {"method": "cursor", "limit": 20, "cursor": null} 2. Response: {"documents": [...], "pagination": {"cursor": "abc123", "has_next": true}} 3. Request: {"method": "cursor", "limit": 20, "cursor": "abc123"} ScrollPaginationParams: properties: method: type: string const: scroll title: Method description: Constant identifying scroll pagination (REQUIRED). default: scroll limit: type: integer maximum: 1000 minimum: 1 title: Limit description: >- Number of documents to fetch per scroll page (REQUIRED). Default: 100. default: 100 scroll_id: anyOf: - type: string - type: 'null' title: Scroll Id description: >- Server-issued scroll session identifier (OPTIONAL). null for first request, then use scroll_id from response scroll_ttl: type: integer maximum: 3600 minimum: 60 title: Scroll Ttl description: >- Seconds to keep scroll context alive (REQUIRED). Default: 300 (5 minutes). default: 300 type: object title: ScrollPaginationParams description: >- Scroll-style pagination maintaining server-side context for TTL. Best for: Bulk exports, batch processing, iterating through all results How it works: - Server maintains a snapshot of results - First request: scroll_id=null, returns scroll_id - Subsequent requests: use scroll_id from response - Context expires after scroll_ttl seconds - Consistent view of data (point-in-time snapshot) Tradeoffs: - Requires server-side state (memory/cache) - TTL means sessions can expire - Not suitable for long-lived sessions - Good for background jobs, not user-facing UIs Use when: - Exporting large datasets - Batch processing all results - Background jobs iterating through results - You need consistent point-in-time view Example flow: 1. Request: {"method": "scroll", "limit": 100, "scroll_id": null} 2. Response: {"documents": [...], "scroll_id": "xyz789", "has_next": true} 3. Request: {"method": "scroll", "limit": 100, "scroll_id": "xyz789"} KeysetPaginationParams: properties: method: type: string const: keyset title: Method description: Constant identifying keyset pagination (REQUIRED). default: keyset limit: type: integer maximum: 500 minimum: 1 title: Limit description: >- Maximum number of documents to return per page (REQUIRED). Default: 10. default: 10 after: anyOf: - additionalProperties: true type: object - type: 'null' title: After description: >- Last seen keyset marker from previous response (OPTIONAL). Must include all sort fields. Example: {'score': 0.73, 'id': 'doc_20'}. null for first page, then use next_cursor from response type: object title: KeysetPaginationParams description: >- Stateless keyset pagination relying on last seen sort key. Best for: High-performance pagination, large result sets, stable sorting How it works: - Uses actual field values as pagination markers - Database can use indexes efficiently (WHERE score < 0.73) - No offset calculation or server state - Requires deterministic sort order (e.g., score DESC, id ASC) - Most efficient pagination method Requirements: - Results must be sorted consistently - Sort fields must be in the "after" marker - Example: sorted by (score DESC, id ASC) → after: {score: 0.73, id: "doc_20"} Advantages: - No server-side state (truly stateless) - Consistent even with concurrent writes - Database can use indexes (fast for large datasets) - No offset performance degradation Use when: - You have stable, deterministic sort fields - Working with large result sets (10k+ docs) - Maximum performance is critical - You need infinite scroll with best efficiency Example flow: 1. Request: {"method": "keyset", "limit": 20, "after": null} 2. Response: {"documents": [...], "next_cursor": {"score": 0.85, "id": "doc_20"}} 3. Request: {"method": "keyset", "limit": 20, "after": {"score": 0.85, "id": "doc_20"}} ErrorDetail: properties: message: type: string title: Message description: Human-readable error message type: type: string title: Type description: Stable error type identifier (machine-readable) code: anyOf: - type: string - type: 'null' title: Code description: >- Fine-grained error code for programmatic handling (e.g., namespace_name_taken, feature_extractor_not_found). Present only when consumers may need to branch on a specific error condition. details: anyOf: - additionalProperties: true type: object - type: 'null' title: Details description: >- Optional structured details to help debugging (validation errors, IDs, etc.) type: object required: - message - type title: ErrorDetail description: Error detail model. ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type type: object required: - loc - msg - type title: ValidationError ````