# Mixpeek + Databricks
Source: https://docs.mixpeek.com/docs/integrations/databricks-warehouse
Multimodal data warehouse meets data lakehouse -- complementary layers for the modern data stack
## Overview
Mixpeek and Databricks occupy different layers of the data stack. Mixpeek ingests unstructured multimodal files and extracts structured features, embeddings, and classifications. Databricks provides the lakehouse platform -- Delta Lake for storage, Unity Catalog for governance, and integrated ML for training and serving models. Together, they give you a complete path from raw media files to governed, analytics-ready data.
Ingests unstructured files, extracts features (embeddings, transcripts, classifications, metadata), and powers multimodal retrieval.
Stores structured outputs in Delta tables, enforces governance via Unity Catalog, and runs ML training and serving at scale.
## Architecture
```
Mixpeek Databricks
+-----------------------+ +------------------------+
| | | |
Files -----> | Buckets & Collections| | Delta Lake Tables |
(images, | | | |
video, | Decompose files into | export | - classifications |
audio, | features: | ---------> | - extracted metadata |
PDFs) | - embeddings | | - taxonomy labels |
| - transcripts | | - document payloads |
| - classifications | | |
| - metadata | enrich | Unity Catalog governs |
| | <--------- | all tables. Databricks|
| Retrieval & Search | | ML retrains models. |
+-----------------------+ +------------------------+
```
## Use Cases
### Write extracted features as Delta tables
After Mixpeek processes your files, export the structured outputs -- transcripts, object detections, taxonomy labels, metadata -- as Delta tables. This makes them queryable with Spark SQL, joinable with your existing business data, and available to any tool in the Databricks ecosystem.
### Use Unity Catalog for governance
Unity Catalog provides fine-grained access control, lineage tracking, and audit logging for all data assets. Once Mixpeek outputs land in Delta tables, Unity Catalog governs who can access them and how they flow through your organization.
### Combine Mixpeek retrieval with Databricks ML
Use Mixpeek to power real-time multimodal search and retrieval. Feed the same structured features into Databricks ML for batch training -- fine-tune classifiers, build recommendation models, or run large-scale analytics on extracted content.
## Quick Start
Export Mixpeek document metadata to a Delta table using the Mixpeek Python SDK and the Databricks SQL Connector.
```bash theme={null}
pip install mixpeek databricks-sql-connector
```
```python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
# List documents from a collection
documents = client.collections.documents.list(
collection_id="your-collection-id",
page_size=100
)
```
```python theme={null}
from databricks import sql
import json
connection = sql.connect(
server_hostname="YOUR_WORKSPACE.cloud.databricks.com",
http_path="/sql/1.0/warehouses/YOUR_WAREHOUSE_ID",
access_token="YOUR_ACCESS_TOKEN"
)
cursor = connection.cursor()
# Create table if it does not exist
cursor.execute("""
CREATE TABLE IF NOT EXISTS mixpeek_catalog.default.documents (
document_id STRING,
source_url STRING,
content_type STRING,
metadata STRING,
created_at TIMESTAMP
)
""")
# Insert each document
for doc in documents:
cursor.execute(
"""
INSERT INTO mixpeek_catalog.default.documents
(document_id, source_url, content_type, metadata, created_at)
VALUES (%s, %s, %s, %s, %s)
""",
(
doc.get("document_id"),
doc.get("source", {}).get("url"),
doc.get("content_type"),
json.dumps(doc.get("metadata", {})),
doc.get("created_at"),
)
)
connection.commit()
cursor.close()
connection.close()
```
For production workloads, write Mixpeek outputs to cloud storage (S3 or ADLS) and use Databricks Auto Loader to incrementally ingest new files into Delta tables.
## When to Use Each
| Capability | Mixpeek | Databricks |
| ----------------------------------------------------------- | ------------------------- | --------------------------------- |
| Ingest unstructured files (video, images, audio, PDFs) | Yes | No |
| Extract features (embeddings, transcripts, classifications) | Yes | No |
| Multimodal semantic search | Yes | No |
| Structured SQL analytics | No | Yes (Spark SQL) |
| Data governance and lineage | Document-level ACL | Unity Catalog |
| ML model training and serving | No | Yes (MLflow, Model Serving) |
| Streaming ingestion | Webhooks + batch triggers | Structured Streaming, Auto Loader |
Mixpeek handles everything before the data is structured. Databricks handles everything after. Use both to bridge the gap between raw multimodal files and governed, ML-ready data.
## Related
* [Taxonomies](/docs/enrichment/taxonomies) -- classify content and export labels
* [SQL Lookup Stage](/docs/retrieval/stages/sql-lookup) -- query external databases from retriever pipelines
* [API Call Stage](/docs/retrieval/stages/api-call) -- call external APIs during retrieval
* [Webhooks](/docs/operations/webhooks) -- trigger Databricks jobs when Mixpeek processing completes
# Email
Source: https://docs.mixpeek.com/docs/integrations/email
Receive emails as bucket objects with a dedicated inbound address for document intake
## Overview
The Email connector lets you ingest documents by forwarding emails to a dedicated address. Each email becomes a bucket object with the body as a text blob, each attachment as a typed blob, and the original `.eml` preserved for chain of custody.
This is built for compliance-oriented workflows — legal document intake, healthcare record forwarding, secure support inboxes — where email is the transport and the documents (attachments) are the payload.
## Prerequisites
* A Mixpeek account with an active namespace
* A bucket and sync configured for the email connection
* **Cloudflare account** with `mixpeek.com` (or your custom domain) in Cloudflare DNS — Email Routing is free on all plans
## How It Works
1. **Create an email connection** — Mixpeek assigns a unique inbound address (e.g., `conn_abc123@inbound.mixpeek.com`)
2. **Cloudflare receives the email** — MX records point to Cloudflare Email Routing, which routes to a Worker
3. **Worker POSTs raw .eml** — The Cloudflare Email Worker reads the raw RFC 2822 bytes and POSTs them to the Mixpeek webhook
4. **Mixpeek parses and stores** — MIME parsing extracts headers → metadata, body → text blob, attachments → S3-backed blobs, raw .eml → S3
```
Customer Email Client
│
▼
Cloudflare MX (inbound.mixpeek.com)
│
▼
Cloudflare Email Routing (catch-all)
│
▼
mixpeek-email-ingest Worker
│ POST raw .eml bytes
▼
api.mixpeek.com/v1/webhooks/email/{connection_id}
│ MIME parse → S3 upload → object + blobs
▼
Bucket Objects (S3 + MongoDB)
```
## Configuration
### Connection-level fields
| Field | Required | Default | Description |
| ----------------- | -------- | ---------- | -------------------------------------------------------------------------------------------- |
| `allowed_senders` | No | `[]` (all) | Sender allowlist. Exact addresses or domain wildcards (`*@company.com`). Empty = accept all. |
| `store_raw_eml` | No | `true` | Store the original `.eml` file as an additional blob for chain of custody. |
### Auto-provisioned fields (read-only)
| Field | Description |
| ----------------- | ------------------------------------------------------------------------------------------- |
| `inbound_address` | System-assigned email address for this connection (e.g., `conn_abc123@inbound.mixpeek.com`) |
| `webhook_secret` | Auto-generated HMAC-SHA256 signing secret for webhook verification |
## Setup
```python Python theme={null}
from mixpeek import Mixpeek
mp = Mixpeek("your_api_key")
connection = mp.connections.create(
name="Legal Intake Inbox",
provider_type="email",
provider_config={
"credentials": {"type": "webhook_secret"},
"allowed_senders": ["*@lawfirm.com", "paralegal@partner.com"],
"store_raw_eml": True,
},
)
print(f"Inbound address: {connection.provider_config['inbound_address']}")
print(f"Webhook URL: https://api.mixpeek.com/v1/webhooks/email/{connection.connection_id}")
```
```javascript JavaScript theme={null}
const { Mixpeek } = require("mixpeek");
const mp = new Mixpeek("your_api_key");
const connection = await mp.connections.create({
name: "Legal Intake Inbox",
provider_type: "email",
provider_config: {
credentials: { type: "webhook_secret" },
allowed_senders: ["*@lawfirm.com", "paralegal@partner.com"],
store_raw_eml: true,
},
});
console.log(`Inbound address: ${connection.provider_config.inbound_address}`);
console.log(`Webhook URL: https://api.mixpeek.com/v1/webhooks/email/${connection.connection_id}`);
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer your_api_key" \
-H "Content-Type: application/json" \
-d '{
"name": "Legal Intake Inbox",
"provider_type": "email",
"provider_config": {
"credentials": {"type": "webhook_secret"},
"allowed_senders": ["*@lawfirm.com", "paralegal@partner.com"],
"store_raw_eml": true
}
}'
```
Link the email connection to a bucket so incoming emails become objects:
```python Python theme={null}
sync = mp.buckets.syncs.create(
bucket_id="your_bucket_id",
connection_id=connection.connection_id,
source_path="inbox",
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/your_bucket_id/syncs \
-H "Authorization: Bearer your_api_key" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_abc123",
"source_path": "inbox"
}'
```
The Email Worker receives emails at `*@inbound.mixpeek.com` and POSTs the raw `.eml` bytes to the Mixpeek webhook. The worker source is in `server/infra/cloudflare/email-worker/`.
```bash theme={null}
cd server/infra/cloudflare/email-worker
npm install
wrangler login
wrangler deploy
```
Optionally set a global signing key:
```bash theme={null}
wrangler secret put WEBHOOK_SIGNING_KEY
```
In the Cloudflare Dashboard:
1. Go to your domain (`mixpeek.com`) → **Email Routing**
2. **Enable Email Routing** — Cloudflare auto-adds MX records for `inbound.mixpeek.com`
3. Go to **Routing rules** → **Catch-all address**
4. Set action to **Send to a Worker** → select `mixpeek-email-ingest`
Cloudflare Email Routing is free on all plans. MX records are managed automatically — no manual DNS configuration needed.
Send an email with an attachment to the inbound address and verify the object appears in your bucket.
## Object Structure
Each email becomes **one bucket object** with multiple blobs:
| Blob Property | Type | Content |
| ----------------------------------- | ------ | ----------------------------------------------------------------- |
| `email_body` | `text` | Email body (plain text preferred, HTML fallback) |
| `attachment_0`, `attachment_1`, ... | varies | Each attachment, typed by MIME (image, pdf, video, etc.) |
| `raw_eml` | `text` | Original `.eml` file stored in S3 (if `store_raw_eml` is enabled) |
### Email metadata fields
These are set as **root-level fields** on the object and can be mapped to your bucket schema:
| Field | Type | Description |
| ------------------------ | ----------------- | -------------------------------------------- |
| `email_from` | string | Sender address |
| `email_to` | list\[string] | Recipient addresses |
| `email_cc` | list\[string] | CC addresses |
| `email_subject` | string | Subject line |
| `email_date` | string (ISO 8601) | Date the email was sent |
| `email_message_id` | string | RFC 2822 Message-ID (used for deduplication) |
| `email_in_reply_to` | string | Parent message ID (for threading) |
| `email_references` | list\[string] | Thread reference IDs |
| `email_attachment_count` | integer | Number of attachments |
## Schema Mapping
Map email fields to your collection schema to make them searchable:
```json theme={null}
{
"schema": {
"fields": [
{"name": "email_from", "type": "text"},
{"name": "email_subject", "type": "text"},
{"name": "email_date", "type": "text"},
{"name": "email_attachment_count", "type": "text"}
]
}
}
```
Use `attribute_filter` in your retriever to query by sender, date, or subject:
```json theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "email_from", "operator": "contains", "value": "@lawfirm.com"},
{"field": "email_date", "operator": "gte", "value": "2026-01-01"}
]
}
}
}
}
```
## Security
| Feature | Description |
| ------------------------- | ------------------------------------------------------------------ |
| **Sender allowlist** | Only accept emails from specified addresses or domains |
| **Webhook signature** | HMAC-SHA256 verification of inbound payloads |
| **Deduplication** | Duplicate emails (same Message-ID) are skipped |
| **Chain of custody** | Raw `.eml` uploaded to S3 with SHA-256 hash for forensic integrity |
| **Credential encryption** | Webhook secret encrypted at rest (Fernet / CSFLE) |
| **Audit logging** | All connection events logged to ClickHouse (365-day retention) |
Email headers and bodies may contain PII (names, email addresses, phone numbers). Consider enabling PII redaction in your collection pipeline or restricting access to the namespace containing email data.
## Compliance Notes
| Requirement | How Mixpeek addresses it |
| --------------------------------- | ----------------------------------------------------------------------------------------------- |
| **HIPAA — encryption in transit** | Cloudflare enforces TLS on MX; webhook endpoint requires HTTPS (TLS 1.2+) |
| **HIPAA — encryption at rest** | Credentials encrypted via CSFLE; all blobs (body, attachments, raw .eml) stored in encrypted S3 |
| **HIPAA — audit trail** | All access logged to ClickHouse audit service |
| **eDiscovery — immutability** | Raw `.eml` in S3 with SHA-256 hash, stored alongside parsed content |
| **eDiscovery — chain of custody** | Source tracking: `source_provider=email`, `source_object_id=email://{message_id}` |
| **SOC 2 — access control** | Per-namespace RBAC (ADMIN, MEMBER, VIEWER) with granular operations |
Mixpeek does not currently hold a HIPAA BAA. If you need a BAA for PHI handling, contact us at [sales@mixpeek.com](mailto:sales@mixpeek.com) to discuss your requirements.
## Troubleshooting
| Issue | Solution |
| --------------------------------- | -------------------------------------------------------------------------------------------------- |
| **403 — sender not in allowlist** | Add the sender's address or domain to `allowed_senders` |
| **404 — connection not found** | Verify the `connection_id` in the webhook URL matches an active email connection |
| **400 — no active bucket sync** | Create a bucket sync linked to this email connection |
| **Duplicate emails skipped** | Expected behavior — emails with the same Message-ID are deduplicated |
| **Attachments not appearing** | Check that the email service is sending the full raw RFC 2822 message, not a stripped-down version |
## Related
* [Buckets](/docs/ingestion/buckets) — Bucket schemas and objects
* [Create Sync Configuration](/docs/api-reference/bucket-syncs/create-sync-configuration) — Link connections to buckets
* [Attribute Filter](/docs/retrieval/stages/attribute-filter) — Filter by email metadata
# Azure Blob Storage
Source: https://docs.mixpeek.com/docs/integrations/object-storage/azure-blob
Connect Mixpeek to your Azure Blob Storage containers to ingest and process your data.
This guide explains how to connect your Azure Blob Storage containers to Mixpeek, enabling automated data ingestion and processing.
## Prerequisites
* An active Azure subscription.
* A Storage Account with Blob Storage containers containing the data you want Mixpeek to process.
* Permissions to create and manage storage account access keys or managed identities.
## Configuration Steps
Connecting Mixpeek to Azure Blob Storage requires granting Mixpeek read access to your containers. We recommend using a managed identity or storage account access key for authentication.
First, you need to configure access to your Azure Blob Storage containers. Choose one of the following methods:
**Option A: Storage Account Access Key (Simpler Setup)**
1. Navigate to your Storage Account in the Azure Portal.
2. Go to **Access keys** under **Security + networking**.
3. Copy either the **key1** or **key2** value. You'll need this to configure the connection in Mixpeek.
4. Store the key securely.
**Option B: Managed Identity (Recommended for Security)**
1. Navigate to your Storage Account in the Azure Portal.
2. Go to **Access Control (IAM)**.
3. Click **Add** > **Add role assignment**.
4. Select the **Storage Blob Data Reader** role.
5. Assign access to the managed identity that Mixpeek will use.
Ensure your container has the appropriate access level:
1. Navigate to your container in the Azure Portal.
2. Go to **Change access level**.
3. Set the access level to either:
* **Private (no anonymous access)** - Use this if you're using access keys or managed identity
* **Blob (anonymous read access for blobs only)** - Use this if you want to allow public read access
4. Click **OK** to save the changes.
1. Navigate to the **Integrations** or **Data Sources** section in your Mixpeek dashboard (or Mixpeek Studio).
2. Click **Add Connection** or **New Source** and select **Azure Blob Storage**.
3. Enter the required details:
* **Storage Account Name:** The name of your Azure Storage Account.
* **Container Name:** The name of your Blob container.
* **Authentication:**
* If using Access Key: Provide the **Access Key** obtained in Step 1A.
* If using Managed Identity: Provide the **Managed Identity Client ID**.
* Optionally, specify a **Prefix** if you only want Mixpeek to process files within a specific folder in your container.
4. Click **Test Connection** (if available) to verify the credentials and permissions.
5. Click **Save** or **Connect**.
## Verification
Once connected, Mixpeek should start discovering files in your specified Azure Blob container (and prefix, if provided). You can monitor the ingestion status within the Mixpeek Studio. Depending on your pipeline configuration, feature extraction and indexing will begin automatically for supported file types.
If you encounter issues, double-check the access permissions and the credentials provided in Mixpeek. Ensure the storage account name, container name, and authentication details are correct.
# Backblaze B2
Source: https://docs.mixpeek.com/docs/integrations/object-storage/backblaze
Sync files from Backblaze B2 Cloud Storage into Mixpeek buckets using the S3-compatible API.
Backblaze B2 is an S3-compatible object storage service. Mixpeek auto-discovers your regional endpoint — you only need your application key ID and application key.
## Overview
The Backblaze B2 integration lets Mixpeek sync objects from any B2 bucket into your Mixpeek buckets for processing. Each file becomes a bucket object with its metadata, ready for feature extraction in your collections.
Mixpeek uses Backblaze's S3-compatible API via the standard AWS SDK. On connection creation, it calls the B2 authorization API to discover your account's correct regional endpoint (e.g., `s3.us-east-005.backblazeb2.com`) automatically.
## Prerequisites
* An active [Backblaze](https://www.backblaze.com/b2/cloud-storage.html) account with B2 Cloud Storage enabled.
* An application key with at minimum: `listBuckets`, `listFiles`, and `readFiles` capabilities.
* The key ID and application key (shown once at creation — copy it immediately).
## Configuration
### Connection-Level Fields
| Field | Required | Description |
| ----------------- | -------- | --------------------------------------------------------- |
| `key_id` | Yes | Backblaze B2 application key ID |
| `application_key` | Yes | Backblaze B2 application key (secret) — encrypted at rest |
### Sync-Level Fields
| Field | Required | Description |
| -------------------------- | -------- | ---------------------------------------------------------- |
| `source_path` | Yes | `b2://bucket-name/optional/prefix` or `bucket-name/prefix` |
| `sync_mode` | No | `continuous`, `one_time`, or `scheduled` |
| `polling_interval_seconds` | No | Seconds between scheduled runs |
| `include_patterns` | No | Glob patterns to include (e.g., `["*.mp4", "*.jpg"]`) |
| `exclude_patterns` | No | Glob patterns to exclude (e.g., `["*.tmp"]`) |
| `modified_since` | No | Only sync files modified after this ISO 8601 timestamp |
## Setup
1. Log in to the [Backblaze Console](https://secure.backblaze.com/b2_buckets.htm).
2. Go to **App Keys** in the left sidebar.
3. Click **Add a New Application Key**.
4. Set a name (e.g., `mixpeek-sync`) and choose the buckets to grant access to.
5. Enable: **Read and Write Files**, **List Buckets**, **List Files**, **Read Files**.
6. Click **Create New Key** and **copy both the keyID and applicationKey immediately** — the key is shown only once.
The application key is shown only once at creation. If you lose it, you must create a new key.
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-mixpeek-api-key")
connection = client.organizations.connections.create(
name="Backblaze B2 Production",
provider_type="backblaze",
provider_config={
"credentials": {
"type": "application_key",
"key_id": "005870eff85b6c60000000001",
"application_key": "K005...",
},
},
)
print(f"Created connection: {connection['connection_id']}")
```
```javascript JavaScript theme={null}
import { Mixpeek } from 'mixpeek-sdk'
const client = new Mixpeek({ apiKey: 'your-mixpeek-api-key' })
const connection = await client.organizations.connections.create({
name: 'Backblaze B2 Production',
provider_type: 'backblaze',
provider_config: {
credentials: {
type: 'application_key',
key_id: '005870eff85b6c60000000001',
application_key: 'K005...',
},
},
})
console.log('Created connection:', connection.connection_id)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Backblaze B2 Production",
"provider_type": "backblaze",
"provider_config": {
"credentials": {
"type": "application_key",
"key_id": "005870eff85b6c60000000001",
"application_key": "K005..."
}
}
}'
```
```python Python theme={null}
sync = client.buckets.syncs.create(
bucket_id="bkt_your_bucket_id",
connection_id=connection["connection_id"],
source_path="b2://my-backblaze-bucket/videos/",
sync_mode="scheduled",
polling_interval_seconds=3600, # Hourly
)
print(f"Sync created: {sync['sync_config_id']}")
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/bkt_your_bucket_id/syncs \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "X-Namespace: ns_your_namespace_id" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_your_connection_id",
"source_path": "b2://my-backblaze-bucket/videos/",
"sync_mode": "scheduled",
"polling_interval_seconds": 3600
}'
```
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/bkt_your_bucket_id/syncs/SYNC_CONFIG_ID/trigger \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "X-Namespace: ns_your_namespace_id"
```
## Advanced Configuration
### Scoped Key (Recommended)
For production, create a key scoped to a specific bucket:
1. In Backblaze Console → App Keys → Add New Key
2. Under **Allow access to Bucket(s)**, select a single bucket
3. This limits the key's blast radius if it's ever compromised
### File Filtering
Filter which files are synced using glob patterns:
```python theme={null}
sync = client.buckets.syncs.create(
bucket_id="bkt_your_bucket_id",
connection_id=connection["connection_id"],
source_path="b2://my-bucket/",
sync_mode="scheduled",
polling_interval_seconds=86400,
include_patterns=["*.mp4", "*.mov", "*.jpg"],
exclude_patterns=["*_thumbnail.*", "*.tmp"],
)
```
### Incremental Sync
Only sync files added or modified after a specific date:
```python theme={null}
from datetime import datetime, timezone
sync = client.buckets.syncs.create(
bucket_id="bkt_your_bucket_id",
connection_id=connection["connection_id"],
source_path="b2://my-bucket/",
sync_mode="continuous",
polling_interval_seconds=300,
modified_since="2024-01-01T00:00:00Z",
)
```
## Source Path Format
The `source_path` supports multiple formats:
| Format | Example | Description |
| -------------------- | ---------------------- | ------------------------------ |
| `b2://bucket/prefix` | `b2://my-videos/2024/` | Preferred — explicit B2 scheme |
| `s3://bucket/prefix` | `s3://my-videos/2024/` | S3-style URI also accepted |
| `bucket/prefix` | `my-videos/2024/` | Bare bucket + prefix |
| `bucket` | `my-videos` | Entire bucket |
## Sync Modes
| Mode | Description | When to Use |
| ------------ | -------------------------------------- | ------------------------------------ |
| `continuous` | Polls every `polling_interval_seconds` | Real-time monitoring, active uploads |
| `one_time` | Single import, then completes | Historical backfills, migrations |
| `scheduled` | Runs on a fixed interval | Regular batch processing |
## Troubleshooting
Your key ID or application key is incorrect:
1. In the Backblaze Console → App Keys, verify the keyID column matches what you entered.
2. The application key is only shown once — if you lost it, create a new key.
3. Ensure the key has not expired or been deleted.
* Verify the `source_path` matches the correct bucket name (case-sensitive).
* Check that your application key has **listFiles** and **readFiles** capabilities.
* If the key is scoped to a specific bucket, ensure the bucket name in `source_path` matches exactly.
* Check `include_patterns` — make sure they match your file extensions.
Your application key may have **listBuckets** but not **listFiles** on a specific bucket:
1. Create a new key with **List Files** and **Read Files** enabled for the target bucket.
2. Update the connection with the new key.
Backblaze B2 uses flat key namespacing (like S3). Folders are just key prefixes.
* Use `b2://bucket/` (trailing slash) to sync all files recursively.
* Use `b2://bucket/subfolder/` to scope to a specific prefix.
## Related
* [Bucket Syncs](/docs/api-reference/bucket-syncs/create-sync-configuration)
* [Storage Connections](/docs/api-reference/organization-connections/create-storage-connection)
* [Amazon S3 Integration](/docs/integrations/object-storage/s3)
* [Tigris Integration](/docs/integrations/object-storage/tigris)
# Box
Source: https://docs.mixpeek.com/docs/integrations/object-storage/box
Connect Mixpeek to your Box account to sync and process files from Box folders.
This guide explains how to connect your Box storage to Mixpeek, enabling automated ingestion and processing of files from Box folders.
## Prerequisites
* An active Box account (Business or Enterprise plan recommended for CCG auth)
* A Box application created at [developer.box.com](https://developer.box.com)
* Client ID and Client Secret from your Box app
## Authentication Methods
Box supports two authentication methods:
| Method | Best For | Token Expiry |
| ---------------------------------- | ----------------------- | -------------- |
| **OAuth (Developer Token)** | Development and testing | 60 minutes |
| **CCG (Client Credentials Grant)** | Production deployments | Auto-refreshes |
Developer tokens expire after 60 minutes and **cannot be refreshed**. Use CCG authentication for production workloads.
## Configuration Steps
1. Go to [developer.box.com](https://developer.box.com) and sign in
2. Click **My Apps** → **Create New App**
3. Select **Custom App** and choose your authentication method:
* For testing: **User Authentication (OAuth 2.0)**
* For production: **Server Authentication (Client Credentials Grant)**
4. Name your app and click **Create App**
5. In the app's **Configuration** tab, note down your **Client ID** and **Client Secret**
For quick testing with OAuth:
1. Open your app in the Box Developer Console
2. Go to the **Configuration** tab
3. Scroll down to **Developer Token**
4. Click **Generate Developer Token**
5. Copy the token — it's valid for 60 minutes
Generating a new developer token invalidates the previous one. Store it immediately.
For production use with Client Credentials Grant:
1. In your Box app configuration, select **App Access Only** under **App Access Level**
2. Enable **Generate User Access Tokens** under **Advanced Features** (if needed)
3. In the Box Admin Console, approve the app:
* Go to **Admin Console** → **Apps** → **Custom Apps Manager**
* Authorize your CCG app
4. Note your **Enterprise ID** from **Admin Console** → **Enterprise Settings** → **Account Info**
1. Navigate to **Settings** → **Connections** in Mixpeek Studio
2. Click **Add Connection** and select **Box**
3. Enter your connection details:
* **Connection Name:** A descriptive name (e.g., "Production Box")
* **Auth Method:** Choose OAuth or CCG
* **Client ID:** From your Box app configuration
* **Client Secret:** From your Box app configuration
* For OAuth: **Access Token** (developer token)
* For CCG: **Enterprise ID** (from Box Admin Console)
* **Folder ID:** Box folder to sync (use `0` for root folder)
4. Click **Test Connection** to verify credentials
5. Click **Create Connection**
## API Configuration
You can also create a Box connection directly via the API:
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
# OAuth (developer token)
connection = client.organizations.connections.create(
name="My Box Storage",
provider_type="box",
provider_config={
"provider_type": "box",
"credentials": {
"type": "oauth",
"client_id": "your-client-id",
"client_secret": "your-client-secret",
"access_token": "your-developer-token"
},
"folder_id": "0" # Root folder
}
)
# CCG (production)
connection = client.organizations.connections.create(
name="My Box Storage (CCG)",
provider_type="box",
provider_config={
"provider_type": "box",
"credentials": {
"type": "ccg",
"client_id": "your-client-id",
"client_secret": "your-client-secret",
"enterprise_id": "your-enterprise-id"
},
"folder_id": "0"
}
)
```
```bash cURL theme={null}
# OAuth (developer token)
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "My Box Storage",
"provider_type": "box",
"provider_config": {
"provider_type": "box",
"credentials": {
"type": "oauth",
"client_id": "your-client-id",
"client_secret": "your-client-secret",
"access_token": "your-developer-token"
},
"folder_id": "0"
}
}'
# CCG (production)
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "My Box Storage (CCG)",
"provider_type": "box",
"provider_config": {
"provider_type": "box",
"credentials": {
"type": "ccg",
"client_id": "your-client-id",
"client_secret": "your-client-secret",
"enterprise_id": "your-enterprise-id"
},
"folder_id": "0"
}
}'
```
## Setting Up a Sync
Once your connection is created, configure a sync to start ingesting files:
```python theme={null}
# Create a bucket sync from Box
sync = client.buckets.syncs.create(
bucket_id="your-bucket-id",
connection_id="your-connection-id",
source_path="/", # Folder path within Box (relative to folder_id)
sync_mode="continuous", # Keep syncing new/changed files (vs initial_only)
)
```
## Finding Your Folder ID
To sync a specific Box folder (instead of root):
1. Open Box and navigate to the folder
2. Look at the URL: `https://app.box.com/folder/123456789`
3. The number at the end is your **Folder ID**
Use `0` for the root folder.
## Supported File Types
Mixpeek processes all file types from Box. Common supported formats include:
| Type | Formats |
| --------- | -------------------------- |
| Documents | PDF, DOCX, XLSX, PPTX, TXT |
| Images | PNG, JPEG, GIF, WEBP |
| Video | MP4, MOV, AVI |
| Audio | MP3, WAV, M4A |
| Data | JSON, CSV, XML |
## Troubleshooting
Developer tokens expire after 60 minutes. Generate a new token from the Box Developer Console and update your connection credentials.
For production use, switch to **CCG authentication** to avoid expiry issues.
In `continuous` sync mode, only new or modified files since the last sync are processed. If you want to re-process all files, delete and recreate the sync configuration to reset the `last_sync_at` timestamp.
Ensure your CCG app has been authorized in the Box Admin Console:
1. Go to **Admin Console** → **Apps** → **Custom Apps Manager**
2. Find your app and click **Authorize**
Also verify that the **Enterprise ID** matches your Box enterprise account.
For OAuth connections, the developer token grants access based on the Box user's permissions. Ensure the user has access to the target folder.
For CCG connections, the app acts as a service account. Use the Box Admin Console to grant the app access to specific folders if needed.
## Related
* [S3 Integration](/docs/integrations/object-storage/s3)
* [S3 Integration](/docs/integrations/object-storage/s3)
* [Bucket Syncs](/docs/ingestion/buckets)
# Google Cloud Storage
Source: https://docs.mixpeek.com/docs/integrations/object-storage/gcs
Connect Mixpeek to your Google Cloud Storage buckets to ingest and process your data.
This guide explains how to connect your Google Cloud Storage (GCS) buckets to Mixpeek, enabling automated data ingestion and processing.
## Prerequisites
* An active Google Cloud project.
* A GCS bucket containing the data you want Mixpeek to process.
* Permissions to create service accounts and manage IAM policies in your Google Cloud project.
## Configuration Steps
Connecting Mixpeek to GCS requires granting Mixpeek read access to your bucket. We recommend using a service account with appropriate IAM roles for enhanced security.
First, create a service account that Mixpeek will use to access your GCS bucket.
1. Navigate to the IAM & Admin > Service Accounts section in your Google Cloud Console.
2. Click **Create Service Account**.
3. Enter a name for the service account (e.g., `mixpeek-gcs-access`).
4. Click **Create and Continue**.
5. Grant the service account the following roles:
* `Storage Object Viewer` (roles/storage.objectViewer)
* `Storage Legacy Bucket Reader` (roles/storage.legacyBucketReader)
6. Click **Done**.
To authenticate with GCS, you'll need to create and download a service account key.
1. Find your newly created service account in the list and click on it.
2. Go to the **Keys** tab.
3. Click **Add Key** > **Create new key**.
4. Choose **JSON** as the key type.
5. Click **Create**.
6. **Important:** The JSON key file will be downloaded to your computer. Store this file securely as you'll need it to configure the connection in Mixpeek.
1. Navigate to the **Integrations** or **Data Sources** section in your Mixpeek dashboard (or Mixpeek Studio).
2. Click **Add Connection** or **New Source** and select **Google Cloud Storage**.
3. Enter the required details:
* **Bucket Name:** The name of your GCS bucket (e.g., `YOUR_BUCKET_NAME`).
* **Project ID:** Your Google Cloud project ID.
* **Service Account Key:** Upload the JSON key file you downloaded in the previous step.
* Optionally, specify a **Prefix** if you only want Mixpeek to process files within a specific folder in your bucket.
4. Click **Test Connection** (if available) to verify the credentials and permissions.
5. Click **Save** or **Connect**.
## Verification
Once connected, Mixpeek should start discovering files in your specified GCS bucket (and prefix, if provided). You can monitor the ingestion status within the Mixpeek Studio. Depending on your pipeline configuration, feature extraction and indexing will begin automatically for supported file types.
If you encounter issues, double-check the service account permissions and the credentials provided in Mixpeek. Ensure the bucket name and project ID are correct.
# Google Drive
Source: https://docs.mixpeek.com/docs/integrations/object-storage/google-drive
Sync files from a Google Drive folder or shared drive into Mixpeek
Mixpeek reads files directly from Google Drive — personal folders, shared drives, and team collaboration folders. You create a storage **connection** (how to authenticate), then a bucket **sync** that points at a folder. New and changed files are picked up automatically.
Mixpeek only **reads** from your Drive. It never writes or deletes files in your Drive.
## 1. Authenticate
Two auth methods are supported. **Service account** is recommended for unattended, server-to-server sync.
1. In Google Cloud Console, create a **service account** and download its JSON key.
2. **Share the target Drive folder (or shared drive) with the service account's `client_email`** — this is what grants Mixpeek access. Viewer permission is enough.
3. Create the connection with the fields from the JSON key file:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/organizations/connections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Marketing Google Drive",
"provider_type": "google_drive",
"provider_config": {
"provider_type": "google_drive",
"credentials": {
"type": "service_account",
"project_id": "my-project",
"private_key_id": "",
"private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
"client_email": "sync@my-project.iam.gserviceaccount.com",
"client_id": ""
},
"shared_drive_id": "0AH-Xabc123"
}
}'
```
`shared_drive_id` is optional — include it only when syncing a **shared drive** (find it in the drive's URL). For a folder in a regular My Drive, omit it and point the sync at the folder ID instead (step 2). `private_key` and the other secrets are encrypted at rest.
Use OAuth when syncing a specific user's Drive. Supply a client ID/secret and a long-lived refresh token:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/organizations/connections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "My Drive",
"provider_type": "google_drive",
"provider_config": {
"provider_type": "google_drive",
"credentials": {
"type": "oauth",
"client_id": "",
"client_secret": "",
"refresh_token": ""
}
}
}'
```
The response includes a `connection_id` (`conn_...`).
List folders and files the connection can see with `GET /v1/organizations/connections/{connection_id}/folders` and `…/files` — handy for finding the folder ID to sync.
## 2. Sync a folder into a bucket
Create a [bucket](/docs/platform/data-model) first, then attach a sync. `source_path` is the **Drive folder ID** (the trailing segment of the folder's URL, e.g. `1A2b3C...`):
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets/$BUCKET_ID/syncs" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_abc123",
"source_path": "1A2b3C4d5E6f7G8h9I",
"sync_mode": "continuous",
"polling_interval_seconds": 3600
}'
```
Then trigger the first sync:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/trigger" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
After the initial sync, new and changed files in the folder are picked up automatically at the polling interval. See [Syncs](/docs/platform/syncs) for file/metadata filters, `on_delete` cascade behavior, and monitoring.
## Next steps
* [Syncs reference](/docs/platform/syncs) — filters, reconciliation, lifecycle, and robustness
* [Object Storage overview](/docs/integrations/object-storage/overview) — all supported providers
* [Ingest Data](/docs/platform/data-model) — buckets, objects, and batches
# Iconik
Source: https://docs.mixpeek.com/docs/integrations/object-storage/iconik
Connect Iconik DAM to Mixpeek for automated media asset processing and search
## Overview
[Iconik](https://iconik.io) is a cloud-native media asset management platform. By connecting Iconik to Mixpeek, you can automatically sync your media library — enabling search, classification, and analysis across video, image, and audio assets.
Iconik uses the same **connection sync** pattern as all other Mixpeek storage integrations. Mixpeek polls Iconik for assets, downloads proxy files, and creates bucket objects with full metadata. It can also parse editorial project files into **footage ↔ ad relationships** (see [Project-file linkage](#project-file-linkage)). Optionally, you can configure webhooks for real-time delete and update events.
## Prerequisites
* An Iconik account with assets
* An Iconik **App ID** and **Auth Token** with read access to assets, proxies, and files
* A Mixpeek account with an active namespace
## Configuration
### Connection-level fields
| Field | Required | Description |
| ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `auth_token` | Yes | Iconik authentication token (JWT or API token) |
| `app_id` | Yes | Iconik application ID (UUID format) |
| `webhook_secret` | No | Shared signing secret for inbound Iconik webhooks. Required only if you want Mixpeek to cascade-delete objects when assets are deleted in Iconik (see [Webhooks: delete and update](#webhooks-delete-and-update)). |
### Sync-level fields
| Field | Required | Default | Description |
| -------------------------- | -------- | ------------ | ------------------------------------------------------------ |
| `source_path` | No | `iconik://` | Sync all assets, or filter by collection |
| `sync_mode` | No | `continuous` | `initial_only` or `continuous` |
| `polling_interval_seconds` | No | 300 | How often to check for new/modified assets (continuous mode) |
| `skip_duplicates` | No | `true` | Skip assets already synced with the same modification time |
| `reconcile_on_sync` | No | `false` | Remove objects whose source assets no longer exist in Iconik |
| `provider_filters` | No | — | Filter by collection IDs, asset status, or media type |
## Setup
1. In the Iconik admin panel, go to **Admin > Integrations > Applications**
2. Click **Create Application**
3. Copy the **App ID** and **Auth Token** — you'll need both for the Mixpeek connection
Use a dedicated application for the Mixpeek integration rather than a personal token. Application tokens don't expire when a user leaves the organization.
In the Mixpeek Studio, go to **Settings > Storage Connections > Add Connection** and select **Iconik**.
Or via the API:
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="YOUR_API_KEY")
connection = client.organizations.connections.create(
name="Iconik Production",
provider_type="iconik",
provider_config={
"credentials": {
"auth_token": "YOUR_ICONIK_AUTH_TOKEN",
"app_id": "YOUR_ICONIK_APP_ID"
}
},
test_before_save=True
)
print(connection.connection_id) # e.g. conn_3010b0af974c8d83
```
```javascript JavaScript theme={null}
import Mixpeek from "mixpeek";
const client = new Mixpeek({ apiKey: "YOUR_API_KEY" });
const connection = await client.organizations.connections.create({
name: "Iconik Production",
provider_type: "iconik",
provider_config: {
credentials: {
auth_token: "YOUR_ICONIK_AUTH_TOKEN",
app_id: "YOUR_ICONIK_APP_ID",
},
},
test_before_save: true,
});
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Iconik Production",
"provider_type": "iconik",
"provider_config": {
"credentials": {
"auth_token": "YOUR_ICONIK_AUTH_TOKEN",
"app_id": "YOUR_ICONIK_APP_ID"
}
},
"test_before_save": true
}'
```
Save the returned `connection_id`.
Create a bucket and configure it to sync from your Iconik connection:
```python Python theme={null}
bucket = client.buckets.create(
namespace_id="your-namespace-id",
name="iconik-assets",
description="Media assets from Iconik",
blob_type="video"
)
sync = client.buckets.syncs.create(
namespace_id="your-namespace-id",
bucket_id=bucket["bucket_id"],
connection_id=connection.connection_id,
source_path="iconik://",
sync_mode="continuous",
polling_interval_seconds=300,
skip_duplicates=True
)
```
```bash cURL theme={null}
# Create bucket
curl -X POST https://api.mixpeek.com/v1/buckets \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "iconik-assets",
"description": "Media assets from Iconik"
}'
# Create sync config
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_YOUR_CONNECTION_ID",
"source_path": "iconik://",
"sync_mode": "continuous",
"polling_interval_seconds": 300,
"skip_duplicates": true
}'
```
Monitor the sync status in Studio or via the API. Assets will be automatically downloaded (proxy files) and processed through your collection's feature extractors.
```bash theme={null}
curl "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE"
```
## Collection traversal and directory mapping
When you specify `collection_ids` in `provider_filters`, the connector recursively traverses the Iconik collection hierarchy (up to 10 levels deep) and maps the collection tree to a `directory_path` on each synced object.
For example, an Iconik collection structure like:
```
Brand Campaign (root collection)
├── 01 Product Shots
│ ├── 01 Ingredients
│ └── 08 Outros
├── 02 UGC Footage
│ ├── 01 Talent (Before)
│ │ ├── 00 Young
│ │ └── 01 Mature
│ └── 02 Talent (After)
└── 03 High-End Footage
```
produces objects with `directory_path` values like:
* `Brand Campaign/01 Product Shots/08 Outros`
* `Brand Campaign/02 UGC Footage/01 Talent (Before)/01 Mature`
* `Brand Campaign/03 High-End Footage`
This path is available in `source_metadata.directory_path` and can be used for filtering in retrievers.
### Glob filtering with `path_patterns`
Use `path_patterns` in `provider_filters` to include only assets whose `directory_path` matches one or more glob patterns. Patterns use standard `fnmatch` syntax and are case-insensitive.
```json theme={null}
{
"provider_filters": {
"collection_ids": ["15202dd4-9b00-11ef-8481-fedcde789d87"],
"media_type": "video",
"path_patterns": ["*UGC*", "*High-End*"]
}
}
```
| Pattern | Matches |
| ---------------------- | ------------------------------------------ |
| `*UGC*` | Any path containing "UGC" |
| `Brand Campaign/01*` | All paths under "01 Product Shots" |
| `*Talent*` | Before and After talent folders |
| `Brand Campaign/*/01*` | First subfolder in each top-level category |
| `*Outros*` | Only the Outros folder |
## Iconik metadata on objects and retriever results
When the connector syncs an Iconik asset into a bucket, it captures metadata from two sources and exposes them as **`source_metadata`** on the resulting bucket object.
### Asset-level fields
These come from the Iconik asset record directly:
| Field | Description |
| ----------------- | ------------------------------------------ |
| `iconik_asset_id` | Iconik asset UUID |
| `title` | Asset title |
| `status` | Asset status (e.g., `ACTIVE`) |
| `media_type` | Media type (`video`, `image`, `audio`) |
| `date_created` | Asset creation timestamp |
| `date_modified` | Last modification timestamp |
| `date_imported` | When the asset was imported into Iconik |
| `is_online` | Whether the asset is currently online |
| `archive_status` | Archive state of the asset |
| `analyze_status` | Iconik analysis status |
| `category` | Asset category |
| `type` | Iconik object type (e.g., `ASSET`) |
| `external_id` | External reference ID |
| `created_by_user` | Iconik user ID who created the asset |
| `updated_by_user` | Iconik user ID who last modified the asset |
| `directory_path` | Collection hierarchy path (see above) |
| `thumbnail_url` | Keyframe/thumbnail URL (if available) |
| `proxy_url` | Pre-signed proxy download URL |
### Custom metadata from views
The connector automatically discovers all metadata views configured in your Iconik workspace and fetches custom metadata for each asset across all views. Fields are flattened from Iconik's nested `field_values` format into simple key-value pairs.
Example fields (depending on your Iconik configuration):
| Field | Description |
| ----------------- | --------------------------- |
| `Keywords` | Tag cloud / keywords |
| `ProjectName` | Project or campaign name |
| `Category` | Asset category |
| `Description` | Asset description |
| `FolderNamePaths` | Iconik folder path metadata |
| `CreatorName` | Creator / uploader name |
All populated custom metadata fields are included automatically — no configuration needed. If you want to limit metadata to a specific view, set `metadata_view_id` in the connection's `provider_config`:
```json theme={null}
{
"provider_config": {
"credentials": { "auth_token": "...", "app_id": "..." },
"metadata_view_id": "97324f86-55ba-11ef-877b-c2a940a89d9c"
}
}
```
## Project-file linkage
Beyond per-asset metadata, the Iconik connector captures the **relationships between assets** that come from editorial (NLE) projects. For Iconik **`NLE_PROJECT` assets** (e.g. an Adobe Premiere project — one per produced ad), the connector reads Iconik's **relations** — the media assets the project uses, already parsed by Iconik — and attaches typed, directed **[edges](/docs/retrieval/stages/traverse-edge)** to the object — Mixpeek's first-class model for customer-owned relationships between objects.
When the project file itself is downloadable (native `.prproj`, `.fcpxml`, or an `xmeml` `.xml` export), the connector additionally parses it to enrich each edge with **clip order and timeline ticks** — matched to the relations by footage file name. This is how an assembled ad is linked to every piece of footage it uses, with its position in the cut — captured as a **saved value** on the object at ingestion, not recomputed at query time.
### What gets captured
For an NLE project asset, the connector emits one edge per related media asset, stored at the **root** of the object under `edges`:
| Edge field | Description |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | `uses_footage` for related video assets; `uses_media` for audio/image/other (with `attributes.media_type`) |
| `target_object_id` | The footage object this relation references, resolved from the source asset id. Until that footage object exists in Mixpeek, the edge preserves the source asset reference and is flagged `attributes.target_unresolved` so a later reconcile can rewrite it. |
| `direction` | `out` (ad → footage) |
| `attributes.iconik_asset_id` | The related Iconik asset's id (the durable foreign key) |
| `attributes.clip_order` | The clip's order within the ad sequence (1-based; from the project-file parse) |
| `attributes.start_ticks_in` / `start_ticks_out` | Clip position in Adobe ticks (`254,016,000,000` per second). `attributes.tick_semantics` says what they measure: `"timeline"` (position in the ad — native `.prproj`) or `"source"` (in/out within the source footage — `xmeml` exports) |
The connector also sets two root-level signals on the ad object:
| Field | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `is_final_ad` | `true` when the asset is an assembled/produced ad (a project file with clips) — distinguishing it from raw pre-production footage |
| `ad_clip_count` | Number of footage clips composited into the ad |
Plain media assets (raw footage) get no edges and pay no extra API calls — linkage capture is gated to NLE project assets. Everything is fully **fail-open**: a relations error or a missing/malformed project file never blocks asset ingestion (edges without a parseable project file simply carry no tick attributes).
Edges and these signals are **customer-owned data** and live at the object **root**, never under `_internal`. They flow automatically from the object through your collections into the document's search payload, so they are available to retrievers.
### Traversing the relationships
Once footage↔ad edges are on your documents, the [`traverse_edge`](/docs/retrieval/stages/traverse-edge) retriever stage follows them at query time — e.g. starting from an ad, fetch every piece of footage it uses, carrying the clip order and start-ticks of each:
```json theme={null}
{
"stage_id": "traverse_edge",
"parameters": { "edge_type": "uses_footage", "direction": "out" }
}
```
This linkage is a **generalizable** capability of Mixpeek's source-adapter framework — any connector whose source carries a companion/sidecar file (an NLE project, an XMP sidecar, a per-object `.json`) can populate edges the same way. The Adobe Premiere `xmeml` parser is the Iconik-specific piece.
## File resolution
The Iconik connector resolves downloadable files in priority order:
1. **Proxy URL** — pre-signed download URL from `GET /files/v1/assets/{id}/proxies/`
2. **Original file URL** — download URL from `GET /files/v1/assets/{id}/formats/`
Per asset, the connector makes: one asset metadata call, one proxy URL resolution call, and one metadata view call per discovered view (views are cached after the first asset).
## Webhooks: delete and update
Mixpeek handles Iconik webhook events to keep your index in sync in real time, without waiting for the next poll cycle.
### How it works
1. Iconik POSTs events to a Mixpeek endpoint that includes your `connection_id`.
2. Mixpeek verifies the `X-Iconik-Signature` header against the `webhook_secret` stored on that connection.
3. **`assets.asset_deleted`** — Mixpeek finds every object synced from that Iconik asset and deletes them, cascading through `OBJECT_DELETED` to remove derived documents from your collections.
4. **`assets.asset_updated`** — Mixpeek re-evaluates the asset against each sync config's `metadata_filters`. If the asset no longer matches **and `reconcile_on_sync` is enabled**, the corresponding objects are unindexed.
5. **`assets.asset_created`** — Acknowledged; the asset will be picked up on the next sync poll.
### Setup
Pick any high-entropy string (32+ chars) and store it as `webhook_secret` in the Iconik connection's credentials. You can set it at create time or patch an existing connection.
```bash cURL theme={null}
curl -X PATCH https://api.mixpeek.com/v1/organizations/connections/{connection_id} \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"provider_config": {
"credentials": {
"auth_token": "...",
"app_id": "...",
"webhook_secret": "your-32-char-random-string"
}
}
}'
```
In the Iconik admin panel:
1. Go to **Admin > Integrations > Webhooks**
2. Click **Create Webhook**
3. Set the **URL** to `https://api.mixpeek.com/v1/webhooks/iconik/{connection_id}` — replace `{connection_id}` with the Mixpeek connection ID
4. Subscribe to events: **Asset Created**, **Asset Updated**, **Asset Deleted**
5. Save the webhook
Delete a synced asset in Iconik and confirm the corresponding object disappears from your bucket. The webhook returns a JSON body documenting what it did:
```json theme={null}
{
"received": true,
"event_type": "assets.asset_deleted",
"handled": true,
"asset_id": "e0f0ec84-4d63-11f1-99fa-b221c0ff30c8",
"deleted_objects": 1
}
```
### Signature format
Mixpeek verifies inbound Iconik webhooks using HMAC-SHA256:
* Header: `X-Iconik-Signature: `
* HMAC computed as `HMAC_SHA256(webhook_secret, raw_body)` over the **exact raw bytes** of the request body.
* Signatures are compared with `hmac.compare_digest` to avoid timing attacks.
### Response codes
| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------- |
| `200` | Event received. `handled` is `true` for delete/update/create, `false` for unrecognized event types. |
| `400` | Connection has no `webhook_secret` configured, or payload is not valid JSON. |
| `401` | `X-Iconik-Signature` is missing, malformed, or the HMAC doesn't match. |
| `404` | `connection_id` in the URL is not an Iconik connection. |
## Modification detection
When `skip_duplicates` is `true` (default), re-syncing the same Iconik account skips assets whose `date_modified` hasn't changed since the last sync. This makes continuous syncing efficient — only new or modified assets are re-processed.
When `skip_duplicates` is `false`, every asset is re-downloaded and re-processed on each sync cycle, replacing existing objects with fresh ones.
## Reconciliation
When `reconcile_on_sync` is enabled, each sync cycle checks whether previously synced assets still exist in Iconik. Assets that have been deleted from Iconik are automatically removed from your bucket.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_YOUR_CONNECTION_ID",
"source_path": "iconik://",
"reconcile_on_sync": true
}'
```
Reconciliation checks each indexed object against the Iconik API on every sync cycle. For large libraries (10k+ assets), this adds API calls proportional to the number of indexed objects. Use it when stale objects are unacceptable.
## Provider filters
Filter which Iconik assets get synced using `provider_filters`:
```json theme={null}
{
"provider_filters": {
"collection_ids": ["col_abc", "col_def"],
"status": "ACTIVE",
"media_type": "video"
}
}
```
| Filter | Description |
| ---------------- | ------------------------------------------------- |
| `collection_ids` | Only sync assets from specific Iconik collections |
| `status` | Filter by asset status (e.g., `ACTIVE`) |
| `media_type` | Filter by media type (`video`, `image`, `audio`) |
## Sync modes
| Mode | Description |
| -------------- | ------------------------------------------------------- |
| `initial_only` | Sync all current assets once |
| `continuous` | Poll for new/modified assets at the configured interval |
## Troubleshooting
Verify your App ID and Auth Token are correct. The App ID is a UUID (e.g., `e0f0ec84-4d63-...`). Tokens can be regenerated in the Iconik admin panel under **Admin > Integrations > Applications**.
Only assets with proxy files or original files available are synced. Check that your Iconik credentials have read access to the files API (`GET /files/v1/assets/{id}/proxies/`).
Ensure `skip_duplicates` is set to `true` on your sync config. This deduplicates assets by their Iconik asset ID and modification timestamp.
Enable `reconcile_on_sync` on your sync config, or configure webhooks to handle `assets.asset_deleted` events for real-time cleanup.
The `X-Iconik-Signature` header doesn't match. Verify the `webhook_secret` on your Mixpeek connection matches what Iconik is using to sign payloads.
## Related
* [Buckets](/docs/ingestion/buckets) — How bucket sync works
* [Webhooks](/docs/operations/webhooks) — How outbound Mixpeek webhooks work
* [Storage Connections API](/docs/api-reference/organization-connections/create-storage-connection) — Full API reference
# Mux
Source: https://docs.mixpeek.com/docs/integrations/object-storage/mux
Connect Mux video infrastructure to Mixpeek for automated video asset processing
## Overview
[Mux](https://mux.com) is a video infrastructure platform for building video experiences. By connecting Mux to Mixpeek, you can automatically process your video assets through feature extraction pipelines — enabling search, classification, and analysis across your entire video library.
## Prerequisites
* A Mux account with video assets
* A Mux access token with **Mux Video** read permissions
* A Mixpeek account with an active namespace
* **Static renditions or MP4 support enabled on your Mux assets** (see below)
**Static renditions are required.** By default, Mux assets only have HLS streaming — no downloadable MP4 files. Mixpeek needs MP4 renditions to download and process your videos. Assets without static renditions will be **skipped** during sync.
Enable static renditions when creating assets:
```json theme={null}
{
"input": [{"url": "https://example.com/video.mp4"}],
"playback_policy": ["public"],
"static_renditions": [{"resolution": "720p"}]
}
```
Or enable them on existing assets in the [Mux Dashboard](https://dashboard.mux.com) under each asset's settings.
## Configuration
### Connection-level fields
| Field | Required | Description |
| --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `access_token_id` | Yes | Mux access token ID (UUID format) |
| `access_token_secret` | Yes | Mux access token secret |
| `webhook_secret` | No | Shared signing secret for inbound Mux webhooks. Required only if you want Mixpeek to cascade-delete objects when assets are deleted in Mux (see [Cascade delete via webhooks](#cascade-delete-via-webhooks)). |
### Sync-level fields
| Field | Required | Default | Description |
| ------------------------------- | -------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `source_path` | No | `mux://` | Sync all assets, or `mux://{asset_id}` for a specific asset |
| `sync_mode` | No | `continuous` | `initial_only` or `continuous` |
| `polling_interval_seconds` | No | 300 | How often to check for new assets (continuous mode) |
| `file_filters.metadata_filters` | No | — | Array of metadata filter rules applied to Mux asset fields (see [Metadata filtering](#metadata-filtering)) |
| `reconcile_on_sync` | No | `false` | Re-check previously indexed assets on each sync and unindex any that no longer match metadata filters (see [Reconciliation](#reconciliation)) |
## Setup
1. Go to [Mux Dashboard → Settings → Access Tokens](https://dashboard.mux.com/settings/access-tokens)
2. Click **Generate new token**
3. Select **Mux Video** with **Read** permissions
4. Copy the **Token ID** and **Token Secret** immediately — the secret is only shown once
In the Mixpeek Studio, go to **Settings → Storage Connections → Add Connection** and select **Mux**.
Or via the API:
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="YOUR_API_KEY")
connection = client.organizations.connections.create(
name="Mux Production",
provider_type="mux",
provider_config={
"credentials": {
"type": "access_token",
"access_token_id": "a3b02074-dbca-47b6-...",
"access_token_secret": "Am9+RGn2mhmz..."
}
},
test_before_save=True
)
```
```javascript JavaScript theme={null}
import Mixpeek from "mixpeek";
const client = new Mixpeek({ apiKey: "YOUR_API_KEY" });
const connection = await client.organizations.connections.create({
name: "Mux Production",
provider_type: "mux",
provider_config: {
credentials: {
type: "access_token",
access_token_id: "a3b02074-dbca-47b6-...",
access_token_secret: "Am9+RGn2mhmz...",
},
},
test_before_save: true,
});
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Mux Production",
"provider_type": "mux",
"provider_config": {
"credentials": {
"type": "access_token",
"access_token_id": "a3b02074-dbca-47b6-...",
"access_token_secret": "Am9+RGn2mhmz..."
}
},
"test_before_save": true
}'
```
Create a bucket and configure it to sync from your Mux connection:
```python Python theme={null}
bucket = client.buckets.create(
namespace_id="your-namespace-id",
name="mux-videos",
description="Video assets from Mux",
blob_type="video"
)
sync = client.buckets.syncs.create(
namespace_id="your-namespace-id",
bucket_id=bucket["bucket_id"],
connection_id=connection.connection_id,
source_path="mux://",
sync_mode="continuous",
polling_interval_seconds=300
)
```
Monitor the sync status in Studio or via the API. Video assets in `ready` status will be automatically downloaded and processed through your collection's feature extractors.
## Mux metadata on objects and retriever results
When the connector syncs a Mux asset into a bucket, it captures the Mux identifiers and exposes them as **`source_metadata`** on the resulting bucket object. This metadata flows through unchanged to downstream documents and is returned alongside every retriever match — so your application can render results directly from Mux without falling back to the S3 mirror.
| Field | Description |
| ----------------- | --------------------------------------------------------------------------------------------------------- |
| `asset_id` | Mux asset ID — use to look the asset up via the Mux API |
| `playback_id` | Primary public/signed playback ID |
| `playback_ids` | Full list of playback IDs on the asset |
| `playback_url` | HLS manifest URL (`https://stream.mux.com/{playback_id}.m3u8`) ready for `` or any HLS client |
| `thumbnail_url` | Default Mux thumbnail (`https://stream.mux.com/{playback_id}/thumbnail.jpg`) |
| `duration` | Asset duration in seconds |
| `resolution_tier` | Source quality tier (e.g. `1080p`, `1440p`) |
| `status` | Mux asset status at sync time |
| `aspect_ratio` | Asset aspect ratio (e.g. `16:9`) |
Example object payload:
```json theme={null}
{
"object_id": "obj_7fa70fd60a0547152df57cff",
"bucket_id": "bkt_648b4be8",
"source_metadata": {
"asset_id": "5sCF1Yk3rC0102zMVEh74jJE9pT6KFeZ1N8gcQZ3l53FI",
"playback_id": "YGDmuLw7AOiQKwvCZbHAGjdZRZwxMqqg1iBELyJig5E",
"playback_url": "https://stream.mux.com/YGDmuLw7AOiQKwvCZbHAGjdZRZwxMqqg1iBELyJig5E.m3u8",
"thumbnail_url": "https://stream.mux.com/YGDmuLw7AOiQKwvCZbHAGjdZRZwxMqqg1iBELyJig5E/thumbnail.jpg",
"duration": 23.86,
"resolution_tier": "1080p",
"status": "ready",
"aspect_ratio": "16:9"
}
}
```
In retriever results, the same `source_metadata` is attached to each match. A typical UI flow is:
```javascript theme={null}
// Each search result carries the original Mux IDs — no S3 hop needed
results.forEach((match) => {
const { playback_id, playback_url, thumbnail_url } = match.source_metadata;
renderMuxPlayer({ playbackId: playback_id, poster: thumbnail_url });
});
```
## Webhooks: delete and update
Mixpeek handles two Mux webhook event types to keep your index in sync with Mux automatically.
### How it works
1. Mux POSTs events to a Mixpeek endpoint that includes your `connection_id`.
2. Mixpeek verifies the `Mux-Signature` header against the `webhook_secret` stored on that connection.
3. **`video.asset.deleted`** — Mixpeek finds every object synced from that Mux asset and deletes them, cascading through `OBJECT_DELETED` to remove derived documents from your collections.
4. **`video.asset.updated`** — Mixpeek re-fetches the asset's current metadata from Mux and re-evaluates it against each sync config's `metadata_filters`. If a previously non-matching asset now matches, it becomes eligible for indexing on the next sync. If the asset no longer matches **and `reconcile_on_sync` is enabled**, the corresponding objects are unindexed. Without `reconcile_on_sync`, the event is acknowledged but no objects are removed.
### Setup
Pick any high-entropy string (32+ chars) and store it as `webhook_secret` in the Mux connection's credentials. You can set it at create time or patch an existing connection.
```bash cURL theme={null}
curl -X PATCH https://api.mixpeek.com/v1/organizations/connections/{connection_id} \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"provider_config": {
"credentials": {
"type": "access_token",
"access_token_id": "...",
"access_token_secret": "...",
"webhook_secret": "your-32-char-random-string"
}
}
}'
```
In the [Mux Dashboard → Settings → Webhooks](https://dashboard.mux.com/settings/webhooks), add a new endpoint:
* **URL:** `https://api.mixpeek.com/v1/webhooks/mux/{connection_id}` — replace `{connection_id}` with the Mixpeek connection ID returned when you created the connection.
* **Signing secret:** the **same** value you put in `webhook_secret`.
* **Events:** at minimum `video.asset.deleted` and `video.asset.updated`. You can subscribe to additional events; Mixpeek will acknowledge them and ignore any it doesn't act on.
Delete a synced asset in Mux (or use the dashboard's "Send test event" button) and confirm the corresponding object disappears from your bucket. The webhook returns a JSON body documenting what it did:
```json theme={null}
{
"received": true,
"event_type": "video.asset.deleted",
"handled": true,
"asset_id": "5sCF1Yk3rC0102zMVEh74jJE9pT6KFeZ1N8gcQZ3l53FI",
"deleted_objects": 1
}
```
### Signature format
Mixpeek follows the same scheme Mux documents for outgoing webhooks:
* Header: `Mux-Signature: t=,v1=`
* HMAC computed as `HMAC_SHA256(webhook_secret, f"{t}.{raw_body}")` over the **exact raw bytes** of the request body.
* Signatures are compared with `hmac.compare_digest` to avoid timing attacks.
### Response codes
| Status | Meaning |
| ------ | ----------------------------------------------------------------------------------------------- |
| `200` | Event received. `handled` is `true` for `video.asset.deleted`, `false` for ignored event types. |
| `400` | Connection has no `webhook_secret` configured, or payload is not valid JSON. |
| `401` | `Mux-Signature` is missing, malformed, or the HMAC doesn't match. |
| `404` | `connection_id` in the URL is not a Mux connection. |
## Source path format
| Path | Behavior |
| ------------------ | ------------------------------------- |
| `mux://` | Sync all video assets in ready status |
| `mux://{asset_id}` | Sync a specific asset by ID |
## Asset filtering
Only assets in `ready` status are synced. Assets that are `preparing` or `errored` are skipped automatically. You can also use file filters to narrow sync scope:
```python theme={null}
sync = client.buckets.syncs.create(
namespace_id="your-namespace-id",
bucket_id=bucket["bucket_id"],
connection_id=connection.connection_id,
source_path="mux://",
file_filters={
"min_size": 1000000, # Skip assets under ~1MB (estimated)
}
)
```
## Metadata filtering
You can filter which Mux assets get synced based on **asset-level metadata** — `passthrough`, `meta.external_id`, or any field captured from the Mux asset. This is useful when you want a single Mux environment connected to Mixpeek but only certain assets indexed.
Metadata filters are evaluated against the asset's metadata dict (which includes `passthrough`, `duration`, `resolution_tier`, `status`, etc.). Each filter specifies a `field`, `operator`, and `value`.
### Supported operators
| Operator | Description | Example |
| ------------------------ | ------------------------------ | ------------------------------------------ |
| `contains` | Field value contains substring | `passthrough` contains `vi:1` |
| `equals` | Exact match | `status` equals `ready` |
| `not_equals` | Does not match | `resolution_tier` not\_equals `audio_only` |
| `not_contains` | Does not contain substring | `passthrough` not\_contains `skip` |
| `gt`, `lt`, `gte`, `lte` | Numeric comparison | `duration` gt `10` |
| `exists` | Field is present and non-empty | `passthrough` exists `true` |
### Example: sync only flagged assets
A common pattern is to use the Mux `passthrough` field as a sync selector. Your application sets a flag on assets that should be indexed, and Mixpeek only syncs matching assets.
```python Python theme={null}
sync = client.buckets.syncs.create(
namespace_id="your-namespace-id",
bucket_id=bucket["bucket_id"],
connection_id=connection.connection_id,
source_path="mux://",
file_filters={
"metadata_filters": [
{
"field": "passthrough",
"operator": "contains",
"value": "vi:1"
}
]
}
)
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_abc123",
"source_path": "mux://",
"file_filters": {
"metadata_filters": [
{
"field": "passthrough",
"operator": "contains",
"value": "vi:1"
}
]
}
}'
```
With this configuration, an asset with `passthrough: "myapp|vi:1|v:abc"` would be synced, while `passthrough: "myapp|vi:0|v:abc"` would be skipped.
**Metadata filters work with any sync provider**, not just Mux. The same `metadata_filters` syntax applies to S3, GCS, and all other storage connections — the filters are evaluated against whatever metadata the provider exposes for each file.
## Reconciliation
When `reconcile_on_sync` is enabled, each sync cycle re-checks all previously indexed assets against the current metadata filters. Assets that no longer match are automatically unindexed.
This is useful when your application changes asset metadata after initial sync — for example, removing the visual-index flag from a Mux asset's `passthrough`. Without reconciliation, the already-indexed object would remain in the bucket. With reconciliation, it gets cleaned up on the next sync.
```bash theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{"reconcile_on_sync": true}'
```
Reconciliation queries the source provider for each indexed object on every sync cycle. For large libraries (10k+ assets), this adds API calls proportional to the number of indexed objects. Use it when metadata changes are common and stale objects are unacceptable.
## Sync modes
| Mode | Description |
| -------------- | ---------------------------------------------- |
| `initial_only` | Sync all current assets once |
| `continuous` | Poll for new assets at the configured interval |
## Troubleshooting
Verify your access token ID and secret are correct. The token ID is a UUID (e.g., `a3b02074-dbca-47b6-...`), not the environment ID. Tokens can be regenerated in the [Mux Dashboard](https://dashboard.mux.com/settings/access-tokens).
Only assets with `ready` status and at least one playback ID are synced. Check your asset status in the Mux Dashboard. Assets that are still encoding (`preparing`) will be picked up on the next sync cycle.
Assets need a public or signed playback ID **and** either static renditions or MP4 support enabled. By default, Mux assets only have HLS streaming — MP4 downloads require explicit configuration.
To enable static renditions when creating an asset via the Mux API:
```json theme={null}
{
"input": [{"url": "https://example.com/video.mp4"}],
"playback_policy": ["public"],
"static_renditions": [{"resolution": "720p"}]
}
```
You can also enable static renditions on existing assets in the Mux Dashboard under the asset's settings. Mixpeek will try multiple rendition qualities (low, medium, high, highest) and use the first available one.
## Related
* [Buckets](/docs/ingestion/buckets) — How bucket sync works
* [Webhooks](/docs/operations/webhooks) — How outbound Mixpeek webhooks work
* [Storage Connections API](/docs/api-reference/organization-connections/create-storage-connection) — Full API reference
# Overview
Source: https://docs.mixpeek.com/docs/integrations/object-storage/overview
Connect Mixpeek to your cloud object storage to ingest multimodal data in place.
Integrate Mixpeek directly with your existing object storage solutions like AWS S3, Google Cloud Storage (GCS), and Azure Blob Storage to process and analyze your multimodal data where it lives.
Object storage is the warehouse's canonical store, where all extracted features persist across tiers and serve as the durable source of truth regardless of which serving layer is active. Object storage systems are a fundamental component for storing large amounts of unstructured data, making them ideal upstream sources for Mixpeek. By connecting your buckets, you enable Mixpeek to automatically discover, process, and index your files, making images, videos, audio, and PDFs searchable in place.
## Why Connect Object Storage?
Process diverse file types stored in your buckets without needing to move data. Mixpeek accesses files directly from your provider.
Use the scalability of cloud object storage. Mixpeek can handle growing volumes of data as your needs evolve.
Set up automated pipelines. New files added to connected buckets can be automatically indexed and enriched by Mixpeek.
Utilize secure authentication methods (like IAM roles or access keys) to grant Mixpeek the necessary permissions to access your data.
## Supported Providers
Mixpeek supports direct integration with the major cloud object storage providers:
Connect your Amazon Simple Storage Service (S3) buckets.
Integrate with Google Cloud Storage (GCS) buckets.
Connect your Azure Blob Storage containers.
S3-compatible egress-free storage on Cloudflare's edge.
Low-cost S3-compatible object storage.
S3-compatible hot object storage.
Globally distributed S3-compatible storage.
Connect Box workspaces with OAuth or CCG.
Sync video assets from the Mux video infrastructure platform — playback metadata flows through to retriever results.
Sync media assets from Iconik DAM — proxy files, metadata, and real-time webhook sync.
Choose your provider above to find specific setup instructions.
## Best Practices for Data Structuring
While Mixpeek can process complex, nested data structures within a single bucket connection, a more scalable strategy often involves structuring your data upfront in your object storage, then relying on taxonomy joins to put them together.
We rely on the post processing joins to intelligently combine related content after ingestion. This approach offers several advantages over trying to group files during the initial upload:
By organizing content into separate buckets or prefixes, you make the relationships between files clear and programmatically accessible. This is more reliable than trying to infer relationships from file names or metadata.
Mixpeek's taxonomy and clustering features allow you to join related content based on multiple criteria (IDs, metadata, semantic similarity) after ingestion, giving you more control over how content is combined.
Processing pipelines become simpler and more focused when handling one type of content structure at a time, making it easier to scale and maintain.
When new content is added to a structured bucket, Mixpeek can process it independently and then join it with related content, avoiding issues with partial uploads or timing dependencies.
**Recommended Approach: Pre-Structured Pipelines**
1. **Separate Buckets or Prefixes:** Organize related but distinct types of content into separate S3 buckets or dedicated prefixes (folders) within a single bucket.
* *Example:* For analyzing video content, you might store raw videos in `s3://my-videos/raw/`, extracted transcripts in `s3://my-videos/transcripts/`, and associated metadata JSON files in `s3://my-videos/metadata/`.
2. **Multiple Mixpeek Connections:** Set up distinct Mixpeek buckets & collections pointing to each specific bucket or prefix.
3. **Join in Mixpeek:** After ingestion, use Mixpeek's enrichment features like **Clustering** or **Taxonomies** (joining based on matching IDs or other rules) to link the related pieces of content (e.g., connecting a transcript to its corresponding video and metadata).
This approach is more reliable than alternatives like:
* Relying on file naming conventions or metadata (which can be inconsistent)
* Using "agent mode" with LLMs to infer relationships (which is experimental and less predictable)
* Trying to group files during upload (which can be fragile due to timing issues)
Consider this approach if you are dealing with complex multimodal data where different components (like video, audio, text transcripts, metadata) need to be linked and analyzed together.
## Getting Started
1. **Choose your Provider:** Select the object storage provider you use (AWS S3, GCS, Azure Blob Storage).
2. **Configure Access:** Follow the provider-specific guide to grant Mixpeek secure access to your desired bucket(s). This typically involves setting up appropriate permissions (e.g., read access).
3. **Add Connection in Mixpeek:** Use the Mixpeek dashboard or API to add the connection details for your object storage bucket.
4. **Start Processing:** Once connected, Mixpeek can begin discovering and processing files according to your configured pipelines.
Ready to connect your data? Select a provider guide above to begin.
# Cloudflare R2
Source: https://docs.mixpeek.com/docs/integrations/object-storage/r2
Connect Mixpeek to your Cloudflare R2 buckets to ingest and process your data.
This guide explains how to connect your Cloudflare R2 storage buckets to Mixpeek, enabling automated data ingestion and processing. R2 is S3-compatible, so the connection process is very similar.
## Prerequisites
* An active Cloudflare account.
* An R2 bucket containing the data you want Mixpeek to process.
* Permissions to create API Tokens in your Cloudflare account with R2 access.
## Configuration Steps
Connecting Mixpeek to R2 requires granting Mixpeek read access to your bucket.
First, create an API Token that grants the necessary permissions for Mixpeek to list and read objects from your R2 bucket.
1. Navigate to your Cloudflare Dashboard.
2. Go to **My Profile** > **API Tokens**.
3. Click **Create Token**.
4. You can use the "Read all R2 Buckets" template or create a custom token.
5. For a custom token, ensure you grant at least `Object Read` and `Bucket Read` permissions for the specific bucket(s) Mixpeek will access.
* **Permissions**:
* Account > R2 > Read
* **Account Resources**:
* Include > Specific account > Your Account
* **Bucket Resources** (if you want to limit access to specific buckets):
* Include > Specific bucket(s) > Your R2 Bucket Name(s)
6. Continue to summary and click **Create Token**.
7. **Important:** Copy the **Access Key ID** and **Secret Access Key** displayed. You will also need your **Account ID**. You will need these to configure the connection in Mixpeek. Store them securely. Cloudflare will only show you the Secret Access Key once.
1. Navigate to the **Integrations** or **Data Sources** section in your Mixpeek dashboard (or Mixpeek Studio).
2. Click **Add Connection** or **New Source** and select **Cloudflare R2** (or **S3-Compatible Storage** if a dedicated R2 option is not available).
3. Enter the required details:
* **Bucket Name:** The name of your R2 bucket.
* **Endpoint URL:** Your R2 S3 API endpoint. This looks like `https://.r2.cloudflarestorage.com`. Replace `` with your Cloudflare Account ID.
* **Access Key ID:** The Access Key ID from the API Token you created.
* **Secret Access Key:** The Secret Access Key from the API Token you created.
* Optionally, specify a **Prefix** if you only want Mixpeek to process files within a specific folder in your bucket.
4. Click **Test Connection** (if available) to verify the credentials and permissions.
5. Click **Save** or **Connect**.
## Verification
Once connected, Mixpeek should start discovering files in your specified R2 bucket (and prefix, if provided). You can monitor the ingestion status within the Mixpeek Studio. Depending on your pipeline configuration, feature extraction and indexing will begin automatically for supported file types.
If you encounter issues, double-check the API Token permissions and the credentials provided in Mixpeek. Ensure the bucket name and S3 endpoint URL (including your Account ID) are correct.
## Conceptual Diagram: Mixpeek and R2 Integration
```mermaid theme={null}
graph TD
R2Bucket[R2 Bucket] -- Contains Objects (Files) --> R2Bucket;
MixpeekPlatform[Mixpeek Platform] -- "Connects via S3 API using API Token credentials" --> R2Bucket;
MixpeekPlatform -- "Retrieves & Processes Each Object" --> ProcessedData[Processed & Indexed Data];
MixpeekPlatform -- "Exposes" --> SearchAPI[Mixpeek Search API];
YourApp[Your Application / Users] -- "Sends Search Queries" --> SearchAPI;
SearchAPI -- "Returns Results" --> YourApp;
subgraph "Mixpeek Infrastructure"
direction LR
MixpeekPlatform
ProcessedData
SearchAPI
end
style R2Bucket fill:#F6821F,stroke:#333,stroke-width:2px
style MixpeekPlatform fill:#4CAF50,stroke:#333,stroke-width:2px
style ProcessedData fill:#2196F3,stroke:#333,stroke-width:2px
style SearchAPI fill:#7E57C2,stroke:#333,stroke-width:2px
style YourApp fill:#00BCD4,stroke:#333,stroke-width:2px
```
This diagram illustrates the workflow:
1. Your data (objects/files) resides in your **R2 Bucket**.
2. The **Mixpeek Platform** securely connects to your R2 Bucket using the S3 API, authenticating with the API Token credentials you provide (as detailed in the configuration steps).
3. Mixpeek retrieves each object from the bucket, performs its processing and indexing routines, creating **Processed & Indexed Data** within its infrastructure.
4. Mixpeek then exposes a **Mixpeek Search API**.
5. **Your Application or Users** interact with this API by sending search queries.
6. The API processes these queries against the indexed data and **Returns Results** to your application or users.
# AWS S3
Source: https://docs.mixpeek.com/docs/integrations/object-storage/s3
Connect Mixpeek to your Amazon S3 buckets to ingest and process your data.
This guide explains how to connect your AWS Simple Storage Service (S3) buckets to Mixpeek, enabling automated data ingestion and processing.
## Prerequisites
* An active AWS account.
* An S3 bucket containing the data you want Mixpeek to process.
* Permissions to create IAM policies and users/roles in your AWS account.
## Configuration Steps
Connecting Mixpeek to S3 requires granting Mixpeek read access to your bucket. We recommend using an IAM Role for enhanced security, but you can also use an IAM User with access keys.
First, create an IAM policy that grants the necessary permissions for Mixpeek to list and read objects from your S3 bucket.
1. Navigate to the IAM service in your AWS Console.
2. Go to **Policies** and click **Create policy**.
3. Switch to the **JSON** tab and paste the following policy document. Remember to replace `YOUR_BUCKET_NAME` with the actual name of your S3 bucket.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
]
}
]
}
```
4. Click **Next: Tags**, then **Next: Review**.
5. Give the policy a descriptive name (e.g., `MixpeekS3ReadAccessPolicy`) and click **Create policy**.
Choose one of the following authentication methods:
**Option A: IAM User (Simpler Setup)**
1. In the IAM service, go to **Users** and click **Add users**.
2. Enter a user name (e.g., `mixpeek-s3-user`).
3. Select **Access key - Programmatic access** as the AWS credential type.
4. Click **Next: Permissions**.
5. Choose **Attach existing policies directly**.
6. Search for and select the policy you created in the previous step (e.g., `MixpeekS3ReadAccessPolicy`).
7. Click **Next: Tags**, then **Next: Review**, then **Create user**.
8. **Important:** Copy the **Access key ID** and **Secret access key**. You will need these to configure the connection in Mixpeek. Store them securely.
**Option B: IAM Role (Recommended for Security)**
* Follow AWS best practices to create an IAM role that Mixpeek can assume. This typically involves setting up a trust policy allowing Mixpeek's AWS account (or a specific identifier provided by Mixpeek) to assume the role.
* Attach the `MixpeekS3ReadAccessPolicy` created in the previous step to this role.
* You will use the **Role ARN** when configuring the connection in Mixpeek instead of access keys. (Consult Mixpeek's specific requirements for Role ARN setup if available).
1. Navigate to the **Integrations** or **Data Sources** section in your Mixpeek dashboard (or Mixpeek Studio).
2. Click **Add Connection** or **New Source** and select **AWS S3**.
3. Enter the required details:
* **Bucket Name:** The name of your S3 bucket (e.g., `YOUR_BUCKET_NAME`).
* **Region:** The AWS region where your bucket is located (e.g., `us-east-1`).
* **Authentication:**
* If using an IAM User: Provide the **Access Key ID** and **Secret Access Key** obtained previously.
* If using an IAM Role: Provide the **Role ARN** obtained previously.
* Optionally, specify a **Prefix** if you only want Mixpeek to process files within a specific folder in your bucket.
4. Click **Test Connection** (if available) to verify the credentials and permissions.
5. Click **Save** or **Connect**.
## Verification
Once connected, Mixpeek should start discovering files in your specified S3 bucket (and prefix, if provided). You can monitor the ingestion status within the Mixpeek Studio. Depending on your pipeline configuration, feature extraction and indexing will begin automatically for supported file types.
If you encounter issues, double-check the IAM policy permissions and the credentials provided in Mixpeek. Ensure the bucket name and region are correct.
# Supabase Storage
Source: https://docs.mixpeek.com/docs/integrations/object-storage/supabase
Sync files from Supabase Storage buckets into Mixpeek using the S3-compatible API.
Supabase Storage exposes an S3-compatible API that Mixpeek talks to directly. You can authenticate with dedicated S3 access keys (recommended) or with a project's `anon` + `service_role` JWTs in session-token mode.
## Overview
The Supabase integration lets Mixpeek pull objects from any Supabase Storage bucket into your Mixpeek buckets for processing. Each file becomes a Mixpeek bucket object, automatically batched into the collections that source the bucket.
Mixpeek derives the endpoint from your project reference — the short URL hash in your project's Dashboard URL (`https://[.supabase.co`). You do not need to manage endpoints by hand.
## Prerequisites
* A Supabase project with at least one Storage bucket.
* Either:
* **S3 access keys** (recommended): a dedicated key pair generated in **Project Settings → Storage → S3 Access Keys**, or
* **Session-token credentials**: the project's `anon` key and `service_role` key from **Project Settings → API**.
## Configuration
### Connection-Level Fields
| Field | Required | Description |
| ------------------------------- | ------------------- | -------------------------------------------------------------------------- |
| `project_ref` | Yes | Supabase project reference (the short hash in `https://][.supabase.co`) |
| `region` | Yes | Region that hosts the project (e.g., `us-east-2`, `eu-west-1`) |
| `credentials.type` | Yes | `access_key` or `session_token` |
| `credentials.access_key_id` | for `access_key` | S3 access key ID generated in Supabase |
| `credentials.secret_access_key` | for `access_key` | S3 secret access key — encrypted at rest |
| `credentials.anon_key` | for `session_token` | Project `anon` JWT |
| `credentials.service_role_key` | for `session_token` | Project `service_role` JWT — encrypted at rest |
### Sync-Level Fields
| Field | Required | Description |
| -------------------------- | -------- | ------------------------------------------------------------------- |
| `source_path` | Yes | `/` — e.g. `user-uploads/photos/` |
| `sync_mode` | No | `continuous`, `one_time`, or `scheduled` |
| `polling_interval_seconds` | No | Seconds between scheduled runs |
| `include_patterns` | No | Glob patterns to include (e.g. `["*.mp4", "*.jpg"]`) |
| `exclude_patterns` | No | Glob patterns to exclude |
| `modified_since` | No | ISO 8601 timestamp; only sync files modified after this date |
## Setup
**Option A — Dedicated S3 keys (recommended):**
1. Open your project in the [Supabase Dashboard](https://supabase.com/dashboard).
2. Go to **Project Settings → Storage → S3 Access Keys**.
3. Click **New access key**, give it a name (e.g. `mixpeek-sync`) and copy the `access_key_id` + `secret_access_key` immediately — the secret is shown only once.
**Option B — Session-token credentials:**
1. Go to **Project Settings → API**.
2. Copy the `anon` key and the `service_role` key. Treat `service_role` like a password — it has full access to Storage.
Session-token mode is convenient when you don't have permission to mint S3 keys, but the `service_role` JWT is a superuser credential. Prefer dedicated S3 keys scoped to the buckets you actually need.
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-mixpeek-api-key")
connection = client.organizations.connections.create(
name="Supabase Uploads",
provider_type="supabase",
provider_config={
"project_ref": "abcdefghijklmnopqrst",
"region": "us-east-2",
"credentials": {
"type": "access_key",
"access_key_id": "supabase_ak_...",
"secret_access_key": "supabase_sk_...",
},
},
)
print(f"Connection: {connection['connection_id']}")
```
```javascript JavaScript theme={null}
import { Mixpeek } from 'mixpeek-sdk'
const client = new Mixpeek({ apiKey: 'your-mixpeek-api-key' })
const connection = await client.organizations.connections.create({
name: 'Supabase Uploads',
provider_type: 'supabase',
provider_config: {
project_ref: 'abcdefghijklmnopqrst',
region: 'us-east-2',
credentials: {
type: 'access_key',
access_key_id: 'supabase_ak_...',
secret_access_key: 'supabase_sk_...',
},
},
})
console.log('Connection:', connection.connection_id)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Supabase Uploads",
"provider_type": "supabase",
"provider_config": {
"project_ref": "abcdefghijklmnopqrst",
"region": "us-east-2",
"credentials": {
"type": "access_key",
"access_key_id": "supabase_ak_...",
"secret_access_key": "supabase_sk_..."
}
}
}'
```
To use session-token mode instead, swap the `credentials` block:
```json theme={null}
{
"type": "session_token",
"anon_key": "eyJ...anon",
"service_role_key": "eyJ...service_role"
}
```
```python Python theme={null}
sync = client.buckets.syncs.create(
bucket_id="bkt_your_bucket_id",
connection_id=connection["connection_id"],
source_path="user-uploads/photos/",
sync_mode="scheduled",
polling_interval_seconds=3600,
)
print(f"Sync: {sync['sync_config_id']}")
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/bkt_your_bucket_id/syncs \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "X-Namespace: ns_your_namespace_id" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_your_connection_id",
"source_path": "user-uploads/photos/",
"sync_mode": "scheduled",
"polling_interval_seconds": 3600
}'
```
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/bkt_your_bucket_id/syncs/SYNC_CONFIG_ID/trigger \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "X-Namespace: ns_your_namespace_id"
```
Mixpeek lists objects under your Supabase prefix, downloads each one, stores it in the Mixpeek-side S3 bucket, and creates a bucket object. Any collection that sources this bucket picks the new objects up on the next batch.
## Advanced Configuration
### File Filtering
```python theme={null}
sync = client.buckets.syncs.create(
bucket_id="bkt_your_bucket_id",
connection_id=connection["connection_id"],
source_path="user-uploads/",
sync_mode="scheduled",
polling_interval_seconds=3600,
include_patterns=["*.mp4", "*.mov", "*.jpg"],
exclude_patterns=["*_thumbnail.*", "*.tmp"],
)
```
### Incremental Sync
Only sync files added or modified after a specific date:
```python theme={null}
sync = client.buckets.syncs.create(
bucket_id="bkt_your_bucket_id",
connection_id=connection["connection_id"],
source_path="user-uploads/",
sync_mode="continuous",
polling_interval_seconds=300,
modified_since="2024-06-01T00:00:00Z",
)
```
## Source Path Format
| Format | Example | Description |
| -------------------- | --------------------------- | ------------------------------------ |
| `bucket/prefix` | `user-uploads/photos/` | Preferred — Supabase bucket + prefix |
| `s3://bucket/prefix` | `s3://user-uploads/photos/` | S3-style URI also accepted |
| `bucket` | `user-uploads` | Entire Supabase bucket |
## Sync Modes
| Mode | Description | When to Use |
| ------------ | -------------------------------------- | ---------------------------------------- |
| `continuous` | Polls every `polling_interval_seconds` | Active uploads, near-real-time ingestion |
| `one_time` | Single import, then completes | Historical backfills, migrations |
| `scheduled` | Runs on a fixed interval | Predictable batch cadence |
## Troubleshooting
* With **access\_key** mode: the keys must come from **Project Settings → Storage → S3 Access Keys**, not the general Supabase API keys.
* With **session\_token** mode: verify `anon_key` and `service_role_key` are both JWTs from the same project and that `project_ref` matches the project they belong to.
* Confirm `region` matches the region shown in the Supabase Dashboard.
The Supabase bucket in `source_path` does not exist or your credentials cannot see it:
* Double-check the bucket name (case-sensitive).
* For access keys with bucket scope, confirm the key grants access to the bucket you're syncing.
* The `source_path` prefix is too narrow — widen it, or drop the trailing prefix.
* `include_patterns` may exclude everything — remove them temporarily to see all files.
* Check the sync run's `matched` counter in the job log to confirm discovery.
Some Supabase projects rotate `anon`/`service_role` keys during upgrades. Refresh both JWTs from **Project Settings → API** and update the connection.
## Related
* [Bucket Syncs](/docs/api-reference/bucket-syncs/create-sync-configuration)
* [Storage Connections](/docs/api-reference/organization-connections/create-storage-connection)
* [Backblaze B2 Integration](/docs/integrations/object-storage/backblaze)
* [Tigris Integration](/docs/integrations/object-storage/tigris)
# Tigris
Source: https://docs.mixpeek.com/docs/integrations/object-storage/tigris
Connect Mixpeek to your Tigris buckets to ingest and process your data.
This guide explains how to connect your Tigris buckets to Mixpeek, enabling automated data ingestion and processing. Tigris is a globally distributed, multi-cloud object storage service with built-in support for the S3 API.
## Prerequisites
* An active Tigris account
* A Tigris bucket containing the data you want Mixpeek to process
* Access key and secret key from your Tigris account
## Configuration Steps
Connecting Mixpeek to Tigris requires granting Mixpeek read access to your bucket. Tigris uses S3-compatible authentication, making it straightforward to set up.
To access your Tigris buckets, you'll need your access key and secret key:
1. Log in to your Tigris account
2. Navigate to your account settings
3. Generate or copy your access key and secret key
4. Store these credentials securely - you'll need them to configure the connection in Mixpeek
If you want to test your Tigris connection locally first, you can configure the AWS CLI:
```bash theme={null}
# Configure AWS CLI
aws configure set aws_access_key_id
aws configure set aws_secret_access_key
aws configure set region auto
# List buckets
aws s3 ls --endpoint-url https://t3.storage.dev
# Create a bucket
aws s3api create-bucket --bucket --endpoint-url https://t3.storage.dev
```
This step is optional but can help verify your credentials are working correctly.
1. Navigate to the **Integrations** or **Data Sources** section in your Mixpeek dashboard
2. Click **Add Connection** or **New Source** and select **Tigris**
3. Enter the required details:
* **Bucket Name:** The name of your Tigris bucket
* **Endpoint URL:** `https://t3.storage.dev`
* **Access Key ID:** Your Tigris access key
* **Secret Access Key:** Your Tigris secret key
* Optionally, specify a **Prefix** if you only want Mixpeek to process files within a specific folder in your bucket
4. Click **Test Connection** to verify the credentials and permissions
5. Click **Save** or **Connect**
## Verification
Once connected, Mixpeek should start discovering files in your specified Tigris bucket (and prefix, if provided). You can monitor the ingestion status within the Mixpeek Studio. Depending on your pipeline configuration, feature extraction and indexing will begin automatically for supported file types.
If you encounter issues, double-check your Tigris credentials and ensure the bucket name is correct. Since Tigris is S3-compatible, you can use standard S3 tools and libraries to verify your bucket access.
## Features
* **Global Distribution:** Tigris automatically places your data close to users for low-latency access worldwide
* **Zero Egress Fees:** No per-GB charge for reading your data out
* **Strong Consistency:** Ensures you always get the correct version of your data
* **S3 Compatibility:** Works with all your familiar S3 tools and libraries
# Wasabi Hot Cloud Storage
Source: https://docs.mixpeek.com/docs/integrations/object-storage/wasabi
Connect Mixpeek to your Wasabi Hot Cloud Storage buckets to ingest and process your data.
This guide explains how to connect your Wasabi Hot Cloud Storage buckets to Mixpeek, enabling automated data ingestion and processing. Wasabi is S3-compatible, making the integration process straightforward.
## Prerequisites
* An active Wasabi account.
* A Wasabi bucket containing the data you want Mixpeek to process.
* Permissions to create IAM users and policies within your Wasabi account.
## Configuration Steps
Connecting Mixpeek to Wasabi requires granting Mixpeek read access to your bucket. This is done by creating an IAM user with appropriate permissions.
First, create an IAM policy in your Wasabi console that grants Mixpeek the necessary permissions to list and read objects from your bucket.
1. Log in to the Wasabi Console.
2. Navigate to **IAM** from the left-hand menu.
3. Select **Policies** and click **Create Policy**.
4. Switch to the **JSON** tab and paste the following policy document. Remember to replace `YOUR_BUCKET_NAME` with the actual name of your Wasabi bucket.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
]
}
]
}
```
5. Click **Next: Tags** (optional), then **Next: Review**.
6. Give the policy a descriptive name (e.g., `MixpeekWasabiReadAccessPolicy`) and an optional description. Click **Create policy**.
Next, create an IAM user and attach the policy you just created.
1. In the Wasabi IAM console, go to **Users** and click **Add User**.
2. Enter a user name (e.g., `mixpeek-wasabi-user`).
3. Select **Programmatic access (generates an access key ID and secret access key)** for the Access type.
4. Click **Next: Permissions**.
5. Choose **Attach existing policies directly**.
6. Search for and select the policy you created in the previous step (e.g., `MixpeekWasabiReadAccessPolicy`).
7. Click **Next: Tags** (optional), then **Next: Review**, then **Create user**.
8. **Important:** Copy the **Access key ID** and **Secret access key**. You will need these to configure the connection in Mixpeek. Store them securely, as the Secret Access Key will not be shown again.
1. Navigate to the **Integrations** or **Data Sources** section in your Mixpeek dashboard (or Mixpeek Studio).
2. Click **Add Connection** or **New Source** and select **Wasabi** or **S3-Compatible Storage**.
3. Enter the required details:
* **Bucket Name:** The name of your Wasabi bucket (e.g., `YOUR_BUCKET_NAME`).
* **Region:** The Wasabi service region where your bucket is located (e.g., `us-east-1`, `eu-central-1`). Refer to Wasabi documentation for the correct service URL endpoint to determine your region or use the specific endpoint.
* **Endpoint URL (Optional but Recommended):** The S3 endpoint for your Wasabi region (e.g., `s3.wasabisys.com` or `s3.us-east-2.wasabisys.com`). Using the specific regional endpoint is best practice.
* **Access Key ID:** The Access Key ID obtained from the Wasabi IAM user.
* **Secret Access Key:** The Secret Access Key obtained from the Wasabi IAM user.
* Optionally, specify a **Prefix** if you only want Mixpeek to process files within a specific folder in your bucket.
4. Click **Test Connection** (if available) to verify the credentials and permissions.
5. Click **Save** or **Connect**.
## Verification
Once connected, Mixpeek should start discovering files in your specified Wasabi bucket (and prefix, if provided). You can monitor the ingestion status within the Mixpeek Studio. Depending on your pipeline configuration, feature extraction and indexing will begin automatically for supported file types.
If you encounter issues, double-check the IAM policy permissions, the credentials provided in Mixpeek, and ensure the bucket name and region/endpoint are correct.
## Conceptual Diagram: Mixpeek and Wasabi Integration
```mermaid theme={null}
graph TD
WasabiBucket[Wasabi Bucket] -- Contains Objects (Files) --> WasabiBucket;
MixpeekPlatform[Mixpeek Platform] -- "Connects via S3 API using IAM User credentials" --> WasabiBucket;
MixpeekPlatform -- "Retrieves & Processes Each Object" --> ProcessedData[Processed & Indexed Data];
MixpeekPlatform -- "Exposes" --> SearchAPI[Mixpeek Search API];
YourApp[Your Application / Users] -- "Sends Search Queries" --> SearchAPI;
SearchAPI -- "Returns Results" --> YourApp;
subgraph "Mixpeek Infrastructure"
direction LR
MixpeekPlatform
ProcessedData
SearchAPI
end
style WasabiBucket fill:#FF9900,stroke:#333,stroke-width:2px
style MixpeekPlatform fill:#4CAF50,stroke:#333,stroke-width:2px
style ProcessedData fill:#2196F3,stroke:#333,stroke-width:2px
style SearchAPI fill:#7E57C2,stroke:#333,stroke-width:2px
style YourApp fill:#00BCD4,stroke:#333,stroke-width:2px
```
This diagram illustrates the workflow:
1. Your data (objects/files) resides in your **Wasabi Bucket**.
2. The **Mixpeek Platform** securely connects to your Wasabi Bucket using the S3 API, authenticating with the IAM user credentials you provide (as detailed in the configuration steps).
3. Mixpeek retrieves each object from the bucket, performs its processing and indexing routines, creating **Processed & Indexed Data** within its infrastructure.
4. Mixpeek then exposes a **Mixpeek Search API**.
5. **Your Application or Users** interact with this API by sending search queries.
6. The API processes these queries against the indexed data and **Returns Results** to your application or users.
# Mixpeek + Snowflake
Source: https://docs.mixpeek.com/docs/integrations/snowflake-warehouse
Use Mixpeek for unstructured multimodal data, Snowflake for structured analytics
## Overview
Mixpeek and Snowflake serve complementary roles in the modern data stack. Mixpeek decomposes unstructured files (images, video, audio, PDFs) into structured features and searchable documents. Snowflake stores, governs, and analyzes structured data at scale. Together, they close the gap between raw multimodal content and business-ready analytics.
Ingests unstructured files, extracts features (embeddings, transcripts, classifications, metadata), and powers multimodal retrieval.
Stores structured outputs, enforces governance, and drives dashboards, ML pipelines, and cross-functional analytics.
## Architecture
```
Mixpeek Snowflake
+-----------------------+ +------------------------+
| | | |
Files -----> | Buckets & Collections| | Structured Tables |
(images, | | | |
video, | Decompose files into | export | - classifications |
audio, | features: | ---------> | - extracted metadata |
PDFs) | - embeddings | | - taxonomy labels |
| - transcripts | | - document payloads |
| - classifications | | |
| - metadata | enrich | Dashboards, BI, ML |
| | <--------- | (feed back into |
| Retrieval & Search | | Mixpeek retrievers) |
+-----------------------+ +------------------------+
```
## Use Cases
### Export taxonomy classifications to Snowflake tables
After Mixpeek classifies your content with [taxonomies](/docs/enrichment/taxonomies), export the labels into Snowflake for reporting and governance.
### Feed extracted metadata into Snowflake dashboards
Mixpeek extracts rich metadata from every file it processes -- transcripts, detected objects, face identities, brand logos, audio fingerprints. Load these structured outputs into Snowflake and build dashboards in Tableau, Sigma, or Snowsight.
### Use Snowflake data to enrich Mixpeek retrievers
Pull structured attributes from Snowflake (pricing, inventory, customer segments) and attach them to Mixpeek documents via the [sql-lookup](/docs/retrieval/stages/sql-lookup) or [api-call](/docs/retrieval/stages/api-call) retriever stages. This lets your multimodal search results carry business context.
## Quick Start
Export Mixpeek document metadata to a Snowflake table using the Mixpeek Python SDK and the Snowflake Connector.
```bash theme={null}
pip install mixpeek snowflake-connector-python
```
```python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
# List documents from a collection
documents = client.collections.documents.list(
collection_id="your-collection-id",
page_size=100
)
```
```python theme={null}
import snowflake.connector
import json
conn = snowflake.connector.connect(
user="YOUR_USER",
password="YOUR_PASSWORD",
account="YOUR_ACCOUNT",
warehouse="YOUR_WAREHOUSE",
database="MIXPEEK_DATA",
schema="PUBLIC"
)
cursor = conn.cursor()
# Create table if it does not exist
cursor.execute("""
CREATE TABLE IF NOT EXISTS mixpeek_documents (
document_id VARCHAR,
source_url VARCHAR,
content_type VARCHAR,
metadata VARIANT,
created_at TIMESTAMP_NTZ
)
""")
# Insert each document
for doc in documents:
cursor.execute(
"""
INSERT INTO mixpeek_documents
(document_id, source_url, content_type, metadata, created_at)
VALUES (%s, %s, %s, PARSE_JSON(%s), %s)
""",
(
doc.get("document_id"),
doc.get("source", {}).get("url"),
doc.get("content_type"),
json.dumps(doc.get("metadata", {})),
doc.get("created_at"),
)
)
conn.commit()
cursor.close()
conn.close()
```
For production workloads, use Snowflake's `COPY INTO` with staged files or Snowpipe for continuous loading instead of row-by-row inserts.
## When to Use Each
| Capability | Mixpeek | Snowflake |
| ----------------------------------------------------------- | ------------------ | ------------------------------ |
| Ingest unstructured files (video, images, audio, PDFs) | Yes | No |
| Extract features (embeddings, transcripts, classifications) | Yes | No |
| Multimodal semantic search | Yes | No |
| Structured SQL analytics | No | Yes |
| Data governance and access control | Document-level ACL | Role-based, column-level |
| Dashboard and BI integration | No | Yes (Snowsight, Tableau, etc.) |
| ML feature store | Embedding vectors | Tabular features |
Mixpeek handles everything before the data is structured. Snowflake handles everything after. Use both to get a complete pipeline from raw files to business insights.
## Related
* [Taxonomies](/docs/enrichment/taxonomies) -- classify content and export labels
* [SQL Lookup Stage](/docs/retrieval/stages/sql-lookup) -- query external databases from retriever pipelines
* [API Call Stage](/docs/retrieval/stages/api-call) -- call external APIs during retrieval
* [Webhooks](/docs/operations/webhooks) -- trigger Snowflake loads when Mixpeek processing completes
# Instagram
Source: https://docs.mixpeek.com/docs/integrations/social-media/instagram
Monitor public Instagram Business and Creator accounts using Mixpeek's Instagram integration.
The Instagram integration uses the **Business Discovery API** to fetch public media from other Instagram Business or Creator accounts. You authenticate with your own account, then monitor any number of public accounts.
## Prerequisites
* A **Facebook account** with a **Facebook Page** that has a linked **Instagram Business or Creator account**. This is your "viewer" account — you don't need to own the accounts you're monitoring.
* The Facebook/Instagram app must have the `instagram_basic`, `pages_show_list`, and `business_management` permissions.
You don't need any relationship with the accounts you monitor. Business Discovery works with any public Instagram Business or Creator account.
## How It Works
The Instagram integration uses Meta's Business Discovery API, which allows an authenticated Instagram Business Account to query public profile data and media from other Business or Creator accounts.
**Key concepts:**
* **Viewer Account** — Your authenticated Instagram Business Account. This is the account that makes API calls on your behalf.
* **Target Accounts** — The public Instagram accounts you want to monitor. Each target is configured as a separate sync config with the account's username as the `source_path`.
* **One Connection, Many Targets** — A single OAuth connection provides the viewer account. You can create unlimited sync configs to monitor different target accounts.
### Data Flow
```
Instagram OAuth → Connection (viewer: @your_business_account)
│
├── Sync Config (source_path: "nike")
│ → Business Discovery API → Media items
│ → Download to S3 → Bucket Objects
│ → Collection Pipeline → MVS Documents
│
├── Sync Config (source_path: "adidas")
│ → ...
│
└── Sync Config (source_path: "spotify")
→ ...
```
### What Gets Synced
For each media item on the target account, Mixpeek captures:
| Field | Description |
| ------------------ | ------------------------------------------------------------- |
| **Media Content** | The image or video file, downloaded and stored in your bucket |
| **Media Type** | `IMAGE`, `VIDEO`, `CAROUSEL_ALBUM`, or `REEL` |
| **Caption** | The post's caption text |
| **Timestamp** | When the post was published |
| **Like Count** | Number of likes at sync time |
| **Comments Count** | Number of comments at sync time |
| **Permalink** | Direct link to the post on Instagram |
## Setup
Navigate to **Connections** in Mixpeek Studio and click **Add Connection**. Select **Instagram** and complete the OAuth flow.
During OAuth, you'll be asked to grant permissions and select which Facebook Pages to share. Make sure to select a Page that has a linked Instagram Business Account.
If none of your Facebook Pages have a linked Instagram Business Account, the connection will fall back to your Facebook user profile and Business Discovery will not work. Ensure at least one Page has a linked IG Business Account in **Page Settings → Instagram**.
Create a bucket to store the synced Instagram media.
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
bucket = client.buckets.create(
bucket_name="instagram-brand-monitor"
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "instagram-brand-monitor"
}'
```
Create a sync config for each Instagram account you want to monitor. The `source_path` is the target account's username.
```python Python theme={null}
# Monitor multiple brands
brands = ["nike", "adidas", "apple", "spotify", "netflix"]
for brand in brands:
sync = client.buckets.syncs.create(
bucket_id=bucket.bucket_id,
connection_id="your-connection-id",
source_path=brand,
sync_mode="initial_only",
batch_size=100
)
print(f"Created sync for @{brand}: {sync.sync_config_id}")
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/BUCKET_ID/syncs \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "your-connection-id",
"source_path": "nike",
"sync_mode": "initial_only",
"batch_size": 100
}'
```
**Sync modes:**
* `initial_only` — Fetches all available media once.
* `continuous` — Periodically checks for new posts and syncs them incrementally.
Trigger each sync config to start fetching media.
```python Python theme={null}
for sync_config_id in sync_config_ids:
result = client.buckets.syncs.trigger(
bucket_id=bucket.bucket_id,
sync_config_id=sync_config_id
)
print(f"Triggered: {result.sync_job_id}")
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/BUCKET_ID/syncs/SYNC_CONFIG_ID/trigger \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE"
```
Create a collection with a feature extractor to process the synced media. The `multimodal_extractor` generates 1408-dimensional embeddings for both images and videos.
```python Python theme={null}
collection = client.collections.create(
collection_name="instagram-multimodal",
source={
"type": "bucket",
"bucket_ids": [bucket.bucket_id]
},
feature_extractor={
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"parameters": {
"run_multimodal_embedding": True
}
}
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/collections \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: YOUR_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "instagram-multimodal",
"source": {
"type": "bucket",
"bucket_ids": ["BUCKET_ID"]
},
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"parameters": {
"run_multimodal_embedding": true
}
}
}'
```
Batch processing runs automatically when syncs complete. Documents are created in [MVS](https://mixpeek.com/mvs) with multimodal embeddings, video segments, thumbnails, and full lineage back to the original bucket object.
## Resilience
The Instagram sync provider includes built-in resilience for handling the Facebook Graph API at scale:
| Feature | Behavior |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Adaptive Page Size** | Starts at 25 items per request. Automatically halves on server errors (minimum 5), preventing failures on large accounts. |
| **Exponential Backoff** | Retries up to 3 times with exponential backoff and jitter on 5xx errors and network timeouts. |
| **Rate Limit Handling** | Respects `429 Retry-After` headers from the Graph API. The Instagram API allows \~200 calls per hour per account. |
| **Graph API Error Parsing** | Parses 400 error responses from the Graph API. Retries "reduce data" errors with smaller page sizes. Fails fast on non-retryable errors (invalid username, permission denied). |
| **Graceful Degradation** | If a page of results fails after all retries, items already synced from previous pages are preserved. |
| **CDN Download Retries** | Media downloads from Instagram's CDN retry independently with their own backoff logic. |
## Token Lifecycle
Instagram access tokens have a limited lifespan. The integration handles token management automatically:
1. **Short-lived token** (1 hour) — Obtained during the OAuth callback.
2. **Long-lived token** (60 days) — Exchanged automatically during the OAuth flow.
3. **Auto-refresh** — When a token is within 7 days of expiry, it's refreshed automatically before each sync execution.
If a token expires without being refreshed (e.g., no syncs run for 60+ days), you'll need to re-authenticate by creating a new connection.
## Limitations
* **Business/Creator accounts only** — Business Discovery only works with public Instagram Business or Creator accounts. Personal accounts cannot be discovered.
* **Public media only** — Only publicly visible posts are accessible. Stories, DMs, and private account content are not available.
* **Rate limits** — The Graph API allows approximately 200 calls per hour per authenticated account. With a default page size of 25, this supports syncing \~5,000 media items per hour.
* **No real-time updates** — Media is fetched on-demand when a sync is triggered. Use `continuous` sync mode for periodic polling.
## Use Cases
Track competitor visual strategies across Instagram. Analyze creative trends, posting frequency, and content themes using multimodal search.
Build a searchable database of influencer content. Find visually similar posts, track engagement patterns, and identify content themes.
Process Instagram media through feature extractors to detect objects, extract text from images, transcribe video audio, and generate semantic embeddings for search.
Compare visual content across brands. Use multimodal retrieval to find similar creative executions and track how visual trends evolve over time.
# Overview
Source: https://docs.mixpeek.com/docs/integrations/social-media/overview
Connect Mixpeek to social media platforms to ingest and analyze public media at scale.
Social media integrations allow you to monitor and analyze public media from other accounts using platform APIs. Mixpeek handles authentication, media download, and processing automatically.
Social media platforms are rich sources of multimodal content — images, videos, carousels, and metadata like captions, engagement metrics, and timestamps. By connecting Mixpeek to social media APIs, you can build brand monitoring, competitive analysis, and content intelligence pipelines without manual data collection.
## Why Connect Social Media?
Track visual content from competitor and partner accounts. Analyze trends in imagery, messaging, and creative strategy across your industry.
Process images and videos through Mixpeek's feature extractors to generate embeddings, detect scenes, extract text, and classify content automatically.
Sync media from multiple accounts with built-in pagination, rate limit handling, and adaptive resilience for large accounts.
Social media content flows through the same bucket, collection, and retriever pipeline as your other data sources — no special handling required.
## How It Works
Social media integrations use a **connection + sync config** architecture:
1. **Connection** — Authenticates with the platform via OAuth. One connection provides API access through your authenticated account.
2. **Sync Configs** — Each sync config targets a specific account to monitor. One connection can have many sync configs, each pointing at a different account.
3. **Sync Execution** — When triggered, Mixpeek fetches media from the target account, downloads content to your bucket (S3/LocalStack), and creates bucket objects with metadata.
4. **Processing** — Bucket objects flow through your collection pipeline for feature extraction, embedding generation, and indexing — just like any other data source.
```
Connection (OAuth)
├── Sync Config: @nike → Bucket Objects → Collection → Documents
├── Sync Config: @adidas → Bucket Objects → Collection → Documents
└── Sync Config: @spotify → Bucket Objects → Collection → Documents
```
## Supported Platforms
Monitor public Instagram Business and Creator accounts via the Business Discovery API.
## Getting Started
1. **Authenticate** — Create a connection by completing the OAuth flow for your chosen platform.
2. **Add Accounts** — Create sync configs for each account you want to monitor.
3. **Trigger Sync** — Run an initial sync to pull media into your bucket.
4. **Process** — Set up a collection with a feature extractor to generate embeddings and index the content.
5. **Retrieve** — Build retrievers to search across your synced social media content.
# BrightData
Source: https://docs.mixpeek.com/docs/integrations/web-data/brightdata
Collect web datasets and scraper results from BrightData and ingest them into Mixpeek.
BrightData is a web data platform with pre-built datasets (LinkedIn, Amazon, Google Maps, etc.) and custom web scrapers. Each sync triggers a new dataset snapshot, waits for it to be ready, and ingests every row as a bucket object.
## Overview
The BrightData integration connects Mixpeek to BrightData's Datasets API. When a sync runs, Mixpeek:
1. Triggers a new dataset snapshot for the configured dataset ID.
2. Polls the snapshot until status is `ready`.
3. Downloads the JSONL results.
4. Creates one bucket object per row, with the full JSON record as the object blob.
Each row's fields are stored as bucket object metadata, making them filterable and searchable alongside the extracted features from your collection pipeline.
## Prerequisites
* An active [BrightData](https://brightdata.com) account.
* A BrightData API token (found in Dashboard → Account → API Token).
* Access to the dataset(s) you want to sync (subscription required for most datasets).
## Configuration
### Connection-Level Fields
| Field | Required | Description |
| ----------------------- | -------- | ------------------------------------------------------ |
| `api_token` | Yes | BrightData API token — encrypted at rest |
| `customer_id` | No | BrightData customer ID for zone-level auth |
| `default_output_format` | No | `jsonl` (default) or `json` |
| `country` | No | ISO 3166-1 alpha-2 code for geo-targeting (e.g., `us`) |
### Sync-Level Fields
| Field | Required | Description |
| -------------------------- | -------- | ----------------------------------------------------- |
| `source_path` | Yes | BrightData dataset ID (e.g., `gd_l1vikfnt1wgvvqz95w`) |
| `sync_mode` | No | `continuous`, `one_time`, or `scheduled` |
| `polling_interval_seconds` | No | Seconds between scheduled runs |
You can find a dataset's ID in the BrightData Marketplace under the dataset detail page. It starts with `gd_`.
## Setup
1. Log in to your [BrightData Dashboard](https://brightdata.com/cp/setting).
2. Go to **Account Settings → API Tokens**.
3. Create a new token or copy an existing one.
Keep your API token secret — it has full access to your BrightData account.
1. Open the [BrightData Marketplace](https://brightdata.com/products/datasets).
2. Select the dataset you want to sync (e.g., LinkedIn Company Profiles, Amazon Products).
3. Copy the dataset ID from the URL or dataset detail page.
Common dataset IDs:
* LinkedIn Company Profiles: `gd_l1vikfnt1wgvvqz95w`
* Amazon Product Data: `gd_l7q7dkf244hwjntr0`
* Google Maps Business Data: `gd_l7q7dkf244hwjntr1`
In [Studio](https://studio.mixpeek.com), open **Connections → Add connection** and pick **BrightData** from the storage provider list.
]![Mixpeek Studio 'Select Storage Provider' modal with BrightData highlighted]()
Or create it programmatically:
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-mixpeek-api-key")
connection = client.organizations.connections.create(
name="BrightData Production",
provider_type="brightdata",
provider_config={
"credentials": {
"type": "api_token",
"api_token": "your-brightdata-api-token",
},
"default_output_format": "jsonl",
},
)
print(f"Created connection: {connection['connection_id']}")
```
```javascript JavaScript theme={null}
import { Mixpeek } from 'mixpeek-sdk'
const client = new Mixpeek({ apiKey: 'your-mixpeek-api-key' })
const connection = await client.organizations.connections.create({
name: 'BrightData Production',
provider_type: 'brightdata',
provider_config: {
credentials: {
type: 'api_token',
api_token: 'your-brightdata-api-token',
},
default_output_format: 'jsonl',
},
})
console.log('Created connection:', connection.connection_id)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/organizations/connections \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "BrightData Production",
"provider_type": "brightdata",
"provider_config": {
"credentials": {
"type": "api_token",
"api_token": "your-brightdata-api-token"
},
"default_output_format": "jsonl"
}
}'
```
```python Python theme={null}
sync = client.buckets.syncs.create(
bucket_id="bkt_your_bucket_id",
connection_id=connection["connection_id"],
# source_path is the BrightData dataset ID
source_path="gd_l1vikfnt1wgvvqz95w",
sync_mode="scheduled",
polling_interval_seconds=86400, # Daily
batch_size=500,
)
print(f"Sync created: {sync['sync_config_id']}")
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/bkt_your_bucket_id/syncs \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "X-Namespace: ns_your_namespace_id" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_your_connection_id",
"source_path": "gd_l1vikfnt1wgvvqz95w",
"sync_mode": "scheduled",
"polling_interval_seconds": 86400,
"batch_size": 500
}'
```
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/buckets/bkt_your_bucket_id/syncs/SYNC_CONFIG_ID/trigger \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "X-Namespace: ns_your_namespace_id"
```
BrightData dataset snapshots can take from a few minutes to an hour depending on dataset size and your subscription tier. Mixpeek polls the snapshot status automatically and processes records as soon as they're ready.
## Advanced Configuration
### Geo-Targeting
Restrict data collection to a specific country:
```python theme={null}
connection = client.organizations.connections.create(
name="BrightData US Only",
provider_type="brightdata",
provider_config={
"credentials": {"type": "api_token", "api_token": "your-token"},
"default_output_format": "jsonl",
"country": "us", # ISO 3166-1 alpha-2
},
)
```
### Schema Mapping
Map BrightData record fields to Mixpeek document fields using the sync's `schema_mapping`:
```json theme={null}
{
"mappings": {
"content": {
"target_type": "blob",
"source": {"type": "file"},
"blob_type": "auto"
},
"company_name": {
"target_type": "field",
"source": {"type": "tag", "key": "name"}
},
"industry": {
"target_type": "field",
"source": {"type": "tag", "key": "industry"}
}
}
}
```
### File Filters
Filter which records are ingested using standard Mixpeek file filter fields:
```json theme={null}
{
"include_patterns": ["*.json"],
"modified_after": "2024-01-01T00:00:00Z"
}
```
## Data Model
Each BrightData record becomes a Mixpeek bucket object with:
* **Blob:** Full JSON record body (stored as `application/json`)
* **Metadata:** Top-level string, integer, float, and boolean fields from the record
* **`source_provider`:** `brightdata`
* **`source_object_id`:** `:` (deduplicated across syncs)
## Sync Modes
| Mode | Description | When to Use |
| ------------ | -------------------------------------- | ------------------------------------------------- |
| `continuous` | Polls every `polling_interval_seconds` | Real-time monitoring, frequently updated datasets |
| `one_time` | Single import, then completes | One-off data migrations, historical backfills |
| `scheduled` | Runs on a fixed interval | Daily/weekly dataset refreshes |
For most BrightData datasets (LinkedIn, Amazon, etc.), `scheduled` with a daily or weekly interval is the best choice since the underlying data changes at that cadence.
## Troubleshooting
BrightData snapshots expire after 1 hour by default. If your dataset is large:
* Reduce the number of records by adding geo-targeting (`country` field)
* Use a higher-tier BrightData subscription with faster processing
* Contact BrightData support to increase your snapshot limits
Your API token may be invalid or revoked:
1. Go to BrightData Dashboard → Account → API Tokens
2. Verify the token is active
3. Create a new token if needed and update the connection
* Verify the dataset ID in `source_path` is correct
* Check that your BrightData subscription includes this dataset
* Inspect the sync job logs in Mixpeek Studio for detailed error messages
BrightData API has rate limits per subscription tier. Increase `polling_interval_seconds`
or upgrade your BrightData plan for higher throughput.
## Related
* [Bucket Syncs](/docs/api-reference/bucket-syncs/create-sync-configuration)
* [Storage Connections](/docs/api-reference/organization-connections/create-storage-connection)
* [Bucket Uploads](/docs/ingestion/uploads)
* [Schema Mapping](/docs/ingestion/buckets)
# Agents
Source: https://docs.mixpeek.com/docs/overview/agents
Give AI agents tools to search, ingest, classify, and monitor multimodal content
![Agent integration: AI agents connect to Mixpeek via MCP, SDK, or REST to access ingest, search, classify, and monitor capabilities]()
Mixpeek exposes its entire platform as agent-callable tools. Connect via MCP for zero-code setup, use the built-in Agent Runtime for stateful conversations, or wire retrievers into LangChain, OpenAI, or any framework via REST.
Create a namespace to get your API key and MCP endpoint, then give your agent search, ingest, and classify tools over your own content.
## MCP (Model Context Protocol)
The fastest way to connect an AI agent to Mixpeek. Four hosted servers expose different tool scopes:
| Scope | URL | Tools |
| ------------- | --------------------------------------- | ----- |
| **Full** | `https://mcp.mixpeek.com/mcp` | 48 |
| **Ingestion** | `https://mcp.mixpeek.com/ingestion/mcp` | 20 |
| **Retrieval** | `https://mcp.mixpeek.com/retrieval/mcp` | 11 |
| **Admin** | `https://mcp.mixpeek.com/admin/mcp` | 17 |
```json theme={null}
{
"mcpServers": {
"mixpeek": {
"url": "https://mcp.mixpeek.com/mcp",
"headers": { "Authorization": "Bearer YOUR_API_KEY" }
}
}
}
```
```bash theme={null}
claude mcp add mixpeek \
--transport streamable-http \
--url https://mcp.mixpeek.com/mcp \
--header "Authorization: Bearer YOUR_API_KEY"
```
```json theme={null}
{
"mcpServers": {
"mixpeek": {
"url": "https://mcp.mixpeek.com/mcp",
"headers": { "Authorization": "Bearer YOUR_API_KEY" }
}
}
}
```
### Per-Retriever Server
For a focused search agent, scope the MCP server to a single retriever. It reads your retriever's `input_schema` and generates a typed `search` tool:
```bash theme={null}
pip install mixpeek-mcp-retriever
mixpeek-mcp-retriever \
--retriever-id ret_xxx \
--namespace-id ns_xxx \
--api-key YOUR_API_KEY
```
Exposes three tools: `search` (typed to your schema), `describe` (retriever metadata), and `explain` (pipeline walkthrough).
[Full MCP reference →](/docs/agent-integrations/mcp)
## Agent Sessions
Mixpeek's built-in agent runtime gives you stateful, multi-turn conversations backed by your data. Each session runs as a dedicated process with tool access, conversation memory, and SSE streaming.
```bash theme={null}
# Create a session
curl -X POST "https://api.mixpeek.com/v1/agents/sessions" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"agent_config": {
"system_prompt": "You help users search and analyze video content.",
"available_tools": ["execute_retriever", "list_collections", "get_taxonomy"]
}
}'
```
```bash theme={null}
# Send a message (SSE streaming)
curl -N -X POST "https://api.mixpeek.com/v1/agents/sessions/$SESSION_ID/messages" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{ "content": "Find videos about machine learning", "stream": true }'
```
The agent reasons through a `analyze → plan → execute → synthesize` workflow, calling tools as needed and streaming events back:
| Event | Description |
| ------------- | ------------------------------ |
| `thinking` | Agent is analyzing or planning |
| `tool_call` | Agent is calling a tool |
| `tool_result` | Tool execution result |
| `message` | Response text chunk |
| `done` | Processing complete |
### Available Tools
| Tool | Description |
| ------------------- | ----------------------------------------- |
| `execute_retriever` | Search documents via a retriever pipeline |
| `search_retrievers` | Find available retrievers |
| `get_retriever` | Get retriever configuration |
| `list_collections` | List collections in the namespace |
| `get_collection` | Get collection details |
| `list_taxonomies` | List taxonomies |
| `get_taxonomy` | Get taxonomy details |
| `list_clusters` | List cluster configurations |
| `get_object` | Get object metadata |
Sessions persist for 7 days and automatically rehydrate after idle periods.
[Agent Sessions API →](/docs/api-reference/agent-sessions/create-session)
## LangChain
The `langchain-mixpeek` package provides a retriever, individual tools, and a full toolkit:
```bash theme={null}
pip install langchain-mixpeek
```
```python theme={null}
from langchain_mixpeek import MixpeekToolkit
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
toolkit = MixpeekToolkit(
api_key="mxp_...",
namespace="my-namespace",
bucket_id="bkt_...",
collection_id="col_...",
retriever_id="ret_...",
)
agent = create_react_agent(
ChatAnthropic(model="claude-sonnet-4-20250514"),
toolkit.get_tools(),
)
result = agent.invoke({
"messages": [("user", "Find product demos and summarize what's shown")]
})
```
### Toolkit Tools
| Tool | What it does |
| ------------------ | ------------------------------------------------- |
| `mixpeek_search` | Search video, images, audio, documents |
| `mixpeek_ingest` | Upload content (text, images, video, audio, PDFs) |
| `mixpeek_process` | Trigger feature extraction |
| `mixpeek_classify` | Run taxonomy classification |
| `mixpeek_cluster` | Group similar documents |
| `mixpeek_alert` | Set up monitoring (webhook, Slack, email) |
Scope tools to what your agent needs with `toolkit.get_tools(actions=["search", "ingest"])`.
[Full LangChain guide →](/docs/agent-integrations/langchain)
## OpenAI Function Calling
Define a Mixpeek retriever as an OpenAI function schema:
```python theme={null}
from openai import OpenAI
from mixpeek import Mixpeek
openai_client = OpenAI()
mixpeek_client = Mixpeek(api_key="mxp_...")
tools = [{
"type": "function",
"function": {
"name": "search_mixpeek",
"description": "Search video, image, and audio content",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results"}
},
"required": ["query"]
}
}
}]
response = openai_client.chat.completions.create(
model="gpt-4o", messages=messages, tools=tools
)
# Handle tool_calls by calling mixpeek_client.retrievers.execute()
```
Works with both the Chat Completions API and Assistants API.
[Full OpenAI guide →](/docs/agent-integrations/openai-function-calling)
## Any Framework (REST)
The same pattern works with CrewAI, LlamaIndex, Haystack, Autogen, or plain HTTP — wrap the retriever execute endpoint as a tool:
```python theme={null}
import requests
def search(query: str, limit: int = 10) -> list:
resp = requests.post(
f"https://api.mixpeek.com/v1/retrievers/{RETRIEVER_ID}/execute",
headers={
"Authorization": f"Bearer {API_KEY}",
"X-Namespace": NAMESPACE_ID,
"Content-Type": "application/json",
},
json={"inputs": {"query_text": query}, "limit": limit},
)
return resp.json()["documents"]
```
[Retriever Execute API →](/docs/api-reference/retrievers/execute-retriever-auto-optimized)
## Choosing an Integration
| I want to... | Use |
| ------------------------------------- | --------------------------------------------------- |
| Connect Claude or Cursor with no code | [MCP](#mcp-model-context-protocol) |
| Build a stateful conversational agent | [Agent Sessions](#agent-sessions) |
| Build a LangChain/LangGraph agent | [LangChain](#langchain) |
| Add tools to GPT models | [OpenAI Function Calling](#openai-function-calling) |
| Use any other framework | [REST](#any-framework-rest) |
# Core Concepts
Source: https://docs.mixpeek.com/docs/overview/concepts
Understand the building blocks that power Mixpeek
Mixpeek is the **multimodal data warehouse** — a system that decomposes unstructured objects into queryable features, stores them across cost tiers, and reassembles them through multi-stage retrieval pipelines. This page introduces the warehouse primitives: the resources that make this work.
Create a managed namespace to get a bucket, collection, and retriever of your own, then run the flow above on your files.
## The six objects, in one sentence each
If you remember nothing else, remember these:
| Object | In plain English |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Namespace** | Your project/environment boundary — everything lives inside one (think "database"). |
| **Bucket** | Where your raw files land before processing (think "inbox" / S3 folder). |
| **Object** | One raw item in a bucket — a video, image, PDF, or JSON record. |
| **Collection** | A recipe that turns objects into searchable **documents** by running the pipeline for the features you pick. |
| **Feature** | What you want extracted per modality (e.g. `video_search`, `faces`) — the plain-language capability you ask for. See [Features](/docs/processing/features). |
| **Extractor** | The model pipeline a feature runs under the hood (a Gemini embedder, Whisper, CLIP, …). You rarely name one directly — features resolve to extractors — but it's what a `feature_uri` points at when you search. See [Extractors](/docs/processing/feature-extractors). |
| **Retriever** | Your query pipeline — composable stages (search → filter → rerank → …) that return results. |
The flow: you put **objects** in a **bucket**, a **collection** extracts the **features** you picked to produce searchable **documents**, and a **retriever** queries those documents — all inside a **namespace**.
![Mixpeek flow: objects in a bucket are processed by a collection's feature pipeline into documents, queried by a retriever, all within a namespace]()
## Entities & Relationships
The full resource model, including the operational and enrichment layers:
| Layer | Entity | What it Represents | Related APIs |
| ---------- | -------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Isolation | **Organization / API Key** | Authentication boundary (`Authorization: Bearer …`) | [API Keys](/docs/api-reference/organization-api-keys/list-api-keys) |
| Isolation | **Namespace** | Tenant or environment boundary (`X-Namespace`) | [Namespaces](/docs/vector-store/namespaces) |
| Storage | **Bucket** | Schema-validated container for objects | [Ingest Data](/docs/platform/data-model) |
| Storage | **Object** | Logical record referencing blobs (files/JSON) | [Ingest Data](/docs/platform/data-model) |
| Processing | **Batch** | Submission that feeds objects into collection processing | [Ingest Data](/docs/platform/data-model) |
| Processing | **Collection** | Document store + feature extraction recipe | [Extract Features](/docs/platform/processing) |
| Processing | **Feature** | What you want extracted per modality — resolved to a versioned pipeline internally | [Features](/docs/processing/features) (advanced pipeline config: [Feature Extractors](/docs/processing/feature-extractors)) |
| Retrieval | **Retriever** | Stage-based search pipeline | [Retrievers](/docs/retrieval/retrievers) |
| Enrichment | **Taxonomy** | Retrieval-backed enrichment recipe (flat or hierarchical) | [Taxonomies](/docs/enrichment/taxonomies) |
| Enrichment | **Cluster** | Vector-based grouping and enrichment artifacts | [Clusters](/docs/enrichment/clusters) |
| Operations | **Task** | Status wrapper for asynchronous jobs | [Tasks](/docs/processing/tasks) |
| Operations | **Webhook** | Event notification subscription | [Webhooks](/docs/platform/operations#webhooks) |
## Dual-ID Multi-Tenancy
Mixpeek separates authentication from authorization by using two IDs per organization:
* **`organization_id`** – Short, user-facing identifier returned in API responses
* **`internal_id`** – 24-character key used inside services, task payloads, and database documents
Namespaces are the primary isolation boundary. Every API request must include `X-Namespace` unless your organization has a single shared namespace. Enforced rules:
* All MongoDB collections index on `namespace_id`
* Each namespace maps to a dedicated [MVS](https://mixpeek.com/mvs) namespace (`ns_`)
* Redis keys and Ray jobs include namespace prefixes
* Cross-namespace queries are not permitted by design
## Object → Document Lineage
Ingestion separates raw objects from processed documents so you can run multiple extraction tiers without duplicating data. Every document tracks:
```json theme={null}
{
"root_object_id": "obj_video_123",
"root_bucket_id": "bkt_marketing",
"source_type": "collection",
"source_collection_id": "col_frames",
"source_document_id": "doc_frame_050",
"lineage_path": "bkt_marketing/col_frames/col_scenes/col_highlights",
"processing_tier": 3
}
```
* **Tier 0** – Raw object in the bucket
* **Tier N** – Document produced by another collection (`source_type = "collection"`)
* The `lineage_path` is a denormalized materialized path for fast queries
* Collections respect dependency tiers during extraction so downstream collections only execute when inputs are ready
Use the [Object Decomposition Tree](/docs/api-reference/document-lineage/get-decomposition-tree-visualization) endpoint to inspect the entire lineage for a given object.
## Feature URIs
Every feature a collection produces is addressed with a version-pinned URI that references the internal pipeline that produced it:
```
mixpeek://{extractor_name}@{version}/{output_name}
```
Examples:
* `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1`
* `mixpeek://image_extractor@v1/google_siglip_base_v1`
* `mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding`
You don't construct these names yourself — discover the URIs a collection emits with `GET /v1/collections/{collection_id}/features`. Feature URIs are referenced by retriever stages (`feature_uri`), taxonomies, clustering jobs, and analytics. They guarantee query-time model compatibility with the ingestion pipeline.
## TaskStatusEnum Standard
All asynchronous operations—batches, clustering jobs, taxonomy materialization, namespace migrations—report status using the shared `TaskStatusEnum`:
```
PENDING → PROCESSING → COMPLETED
↘ ↘
FAILED COMPLETED_WITH_ERRORS
```
Terminal statuses are `COMPLETED`, `COMPLETED_WITH_ERRORS`, `FAILED`, and `CANCELED` — poll until any of these (see [Tasks](/docs/processing/tasks)). Additional lifecycle values include `IN_PROGRESS`, `SKIPPED`, `UNKNOWN`, `DRAFT`, `ACTIVE`, `ARCHIVED`, and `SUSPENDED`. Use the [Tasks API](/docs/processing/tasks) for short-term polling and fall back to the resource (e.g., batch or cluster) for long-running workflows.
## Caching Signatures
Mixpeek uses deterministic signatures to avoid stale results:
* Collection index signatures hash document count, vector dimensions, and schema state
* Retriever caches incorporate the collection signature to invalidate automatically
* Stage-level caches speed up pipelines that reuse expensive stages (KNN → rerank)
* Inference cache shortcuts repeated embedding requests for identical inputs
Learn more in [Caching](/docs/overview/caching).
## Putting It Together
```
Namespace
└── Bucket
├── Object (Tier 0)
└── Batch → Collection (Tier 1)
└── Collection (Tier 2)
└── ...
```
* Documents retain lineage to the original object (`root_object_id`)
* Enrichment layers (taxonomies, clustering) augment documents in place
* Retrievers run on namespace-scoped data, returning results with presigned URLs, metrics, and cache hints
With these concepts in mind you can navigate deeper sections of the docs—whether you’re planning ingestion schemas, designing retriever pipelines, or wiring observability for production deployments.
# Data Model & Lineage
Source: https://docs.mixpeek.com/docs/overview/data-model
Understand how objects transform into documents and how lineage tracks provenance
Mixpeek's data model is built around a clear transformation pipeline: **Objects → Documents → Features**. Each entity has a specific purpose, and lineage metadata ensures you can always trace results back to their source.
## Core Entities
Raw inputs registered in buckets. Objects hold blobs (video, image, text, audio) and metadata but are **not processed** until added to a batch.
Processed representations produced by collection pipelines (the [features](/docs/processing/features) you enabled). Documents live in collections and include vectors, metadata, and lineage references.
Extracted representations (embeddings, classifications, segments) stored as [MVS](https://mixpeek.com/mvs) vectors and referenced by feature URIs.
## Transformation Flow
![Mixpeek Transformation Flow]()
### 1. Object Registration
Objects are created via the Buckets API and validated against the bucket's JSON schema:
```bash theme={null}
POST /v1/buckets/{bucket_id}/objects
```
**Object Structure:**
* `object_id` – unique identifier
* `bucket_id` – parent bucket reference
* `key_prefix` – logical path/grouping
* `blobs[]` – array of file references or inline data
* `metadata` – custom JSON validated against bucket schema
* `created_at` / `updated_at` – audit timestamps
Objects remain **inert** until processed. They exist as metadata records and S3 blob references only.
### 2. Batch Processing
Batches group objects for efficient parallel processing:
```bash theme={null}
POST /v1/buckets/{bucket_id}/batches
POST /v1/buckets/{bucket_id}/batches/{batch_id}/submit
```
When submitted, the API:
1. Resolves all collections that consume the bucket
2. Generates per-extractor artifacts (manifests) and uploads to S3
3. Dispatches Ray tasks to the Engine with S3 artifact URIs
### 3. Document Creation
The Engine runs each collection's processing pipeline in parallel. For each object and each collection:
1. **Download manifest** – fetch pipeline config and input mappings
2. **Run the pipeline** – run model inference (embeddings, classifications, etc.)
3. **Write to MVS** – upsert vectors and payload with `internal_id` tenant filter
4. **Update metadata** – set `__fully_enriched`, `__pipeline_version`, `source_object_id`
**Document Structure:**
* `document_id` – MVS point ID
* `collection_id` – parent collection
* `source_object_id` – lineage back to originating object
* `root_object_id` – if object was derived, traces to original input
* `feature_refs[]` – array of feature URIs (e.g., `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1`)
* `metadata` – passthrough fields + enrichments from taxonomies/clusters
* `__fully_enriched` – boolean flag indicating the full pipeline succeeded
* `__missing_features` – array of feature addresses that failed
* `__pipeline_version` – integer tracking collection schema version
### 4. Feature Storage
Features are stored as [MVS](https://mixpeek.com/mvs) vectors with named indexes:
**Feature URI Format:**
```
mixpeek://@/
```
Example:
```
mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1
mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding
mixpeek://image_extractor@v1/google_siglip_base_v1
```
These URIs are used in:
* Retriever stage configurations (`feature_uri`)
* Taxonomy input mappings
* Join enrichment strategies
## Lineage Tracking
Every document maintains lineage metadata for auditability and debugging:
| Field | Purpose |
| -------------------- | ------------------------------------------------ |
| `source_object_id` | The immediate parent object |
| `root_object_id` | The original root object (for derived documents) |
| `collection_id` | Which collection produced this document |
| `__pipeline_version` | Collection schema version at processing time |
### Querying Lineage
Use the Document Lineage API to trace provenance:
```bash theme={null}
# Get full lineage tree for a document
GET /v1/collections/{collection_id}/documents/{document_id}/lineage
# Find all documents derived from an object
GET /v1/documents/by-object/{object_id}
# Visualize decomposition tree (for hierarchical processing)
GET /v1/documents/{document_id}/lineage/tree
```
## Multi-Level Decomposition
Some pipelines produce **multiple documents per object** (e.g., video → scenes, PDF → pages):
```
Object: video_file.mp4
└─ Document: full_video_summary
├─ Document: scene_001 (0:00-0:15)
├─ Document: scene_002 (0:15-0:32)
└─ Document: scene_003 (0:32-1:00)
```
Each scene document includes:
* `source_object_id` → points to `video_file.mp4` object
* `root_object_id` → same as `source_object_id` (unless video itself was derived)
* `parent_document_id` → points to `full_video_summary` document (if hierarchical)
* `segment_metadata` → `{ start_time: 0.0, end_time: 15.0 }`
## Tenant Isolation
All entities are scoped by:
* **`internal_id`** (organization) – injected at API layer from `Authorization` header
* **`namespace_id`** – resolved from `X-Namespace` header
MVS namespaces map 1:1 to namespaces, and all queries/writes automatically include `internal_id` filters. This ensures hard multi-tenancy without application-level filtering.
## Schema Evolution
Collections track `__pipeline_version` to handle schema changes:
1. Update collection definition (add/remove features, change mappings)
2. Increment `pipeline_version` automatically
3. New batches write documents with updated version
4. Old documents remain queryable with legacy schema
5. Optional: Trigger reprocessing to backfill with the new pipeline
## Feature Reuse Across Collections
Processing pipelines are **reusable**. Collections that enable the same feature emit the same feature URI, even when wired to different input fields (advanced per-pipeline config shown; the extractor names are deprecated aliases):
```json theme={null}
// Collection A: Product descriptions
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"input_mappings": { "text": "product_description" }
}
}
// Collection B: Customer reviews
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"input_mappings": { "text": "review_text" }
}
}
```
Both produce `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1` features, enabling **cross-collection search** via retrievers that span multiple collection IDs.
## Document Lifecycle States
| State | Meaning |
| ------------ | ------------------------------------------------------ |
| `pending` | Batch submitted but Engine hasn't started processing |
| `processing` | Feature extraction in progress |
| `completed` | The full pipeline succeeded (`__fully_enriched: true`) |
| `partial` | Some features failed (`__missing_features` populated) |
| `enriching` | Materialized taxonomy/cluster enrichment running |
Query `__fully_enriched` in retriever filters to exclude incomplete documents.
## Best Practices
Bucket schemas enforce input shape but don't constrain downstream processing. Keep them minimal and use collection `field_passthrough` to propagate only what's needed.
Organize objects with `key_prefix` (e.g., `/catalog/electronics`, `/users/avatars`) to enable bulk operations and filtering without custom metadata.
Group 100-1000 objects per batch for optimal throughput. Smaller batches add orchestration overhead; larger batches delay feedback and complicate retries.
When retrieval results are unexpected, use lineage APIs to trace documents back to source objects and inspect raw input data and processing history.
When changing features or mappings, create a new collection rather than mutating the existing one. This preserves reproducibility and simplifies rollback.
## Example: E-Commerce Product Pipeline
```json theme={null}
1. Register object in "products" bucket:
- blobs: [{ property: "image", type: "image", url: "s3://..." }]
- metadata: { sku: "HD-9000", category: "headphones", price: 299 }
2. Process through "product-embeddings" collection (features: ["image_search", "text_search"]):
- image_search → image embeddings
- text_search → description embeddings
- Passthrough: sku, category, price
3. Resulting document in MVS:
- document_id: "doc_abc123"
- source_object_id: "obj_prod_001"
- feature_refs: [
"mixpeek://image_extractor@v1/google_siglip_base_v1",
"mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
]
- metadata: { sku: "HD-9000", category: "headphones", price: 299 }
- __fully_enriched: true
4. Query with hybrid retriever:
- Stage 1: feature_search on mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1
- Stage 2: attribute_filter where category = "headphones"
- Stage 3: sort_attribute by price ascending
```
## Next Steps
* Dive into advanced pipeline configuration with [Feature Extractors](/docs/processing/feature-extractors)
* Explore [Collections](/docs/ingestion/collections) configuration and output schemas
* Review [Document Lineage API](/docs/api-reference/document-lineage/get-document-lineage) for tracing provenance
* Understand [Namespaces](/docs/ingestion/namespaces) for tenant isolation
# Migrate from Elasticsearch
Source: https://docs.mixpeek.com/docs/overview/from-elasticsearch
From keyword search to multimodal warehouse retrieval
Elasticsearch is a search engine built on keyword matching and BM25. Over time, you may have bolted on vector search (`dense_vector`), external embedding generation, and custom pipelines to handle multimodal content. Mixpeek unifies all of this: feature extraction, tiered storage, and multi-stage retrieval in a single API.
This guide walks you through migrating your search workload from Elasticsearch to Mixpeek.
## Why Migrate
Elasticsearch started as a keyword search engine. Vector search, embedding generation, and multimodal processing are additions you configure and maintain yourself. Mixpeek was built from the ground up as a multimodal data warehouse where feature extraction, storage tiering, and multi-stage retrieval are native primitives, not plugins.
| Elasticsearch | Mixpeek |
| ------------------------------------------------------- | ------------------------------------------------------------------------ |
| Keyword search (BM25) with bolted-on vector search | Native semantic, keyword, and hybrid search in one pipeline |
| You generate and manage embeddings externally | Feature extractors handle embedding generation automatically |
| Single storage tier (hot-warm-cold requires manual ILM) | Automatic tiered storage: active, cold, archive, up to 90% savings |
| Complex DSL for combining query types | Multi-stage retriever pipelines: chain stages declaratively |
| Ingest pipelines for basic transforms | Collections with ML-powered feature extractors (CLIP, Whisper, LayoutLM) |
## Concept Mapping
| Elasticsearch | Mixpeek | Notes |
| --------------------------------- | ------------------------------- | ---------------------------------------------------------------------------- |
| Index | Namespace | Top-level container for your data |
| Document | Document | Mixpeek documents contain extracted features, metadata, and source lineage |
| Mapping / Schema | Collection + Feature Extractor | Collections define what features to extract; schema is derived automatically |
| DSL query | Retriever (with stages) | Stages are like query clauses, composable and ordered |
| `bool` query (must/should/filter) | Multi-stage pipeline | Each clause becomes a stage: search, filter, boost, rerank |
| Aggregation | Reduce stages / Taxonomies | Group, classify, and summarize results |
| Ingest pipeline | Collection + Feature Extractors | Mixpeek pipelines extract ML features, not just field transforms |
| Analyzer (tokenizer + filters) | Feature Extractor configuration | Extractors handle tokenization, embedding, and structured extraction |
| ILM (Index Lifecycle Management) | Automatic storage tiering | Hot, cold, archive managed by the platform |
## Migration Steps
Replace your Elasticsearch index with a Mixpeek namespace.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/namespaces \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespace_name": "knowledge-base"
}'
```
Instead of defining field types and analyzers, create a collection with a feature extractor that matches your content.
```json Elasticsearch (before) theme={null}
PUT /knowledge-base
{
"mappings": {
"properties": {
"title": { "type": "text", "analyzer": "english" },
"body": { "type": "text", "analyzer": "english" },
"embedding": { "type": "dense_vector", "dims": 768 },
"category": { "type": "keyword" },
"published_at": { "type": "date" }
}
}
}
```
```python Mixpeek (after) theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="YOUR_API_KEY")
# Collection handles schema, embedding, and indexing
collection = client.collections.create(
collection_name="articles",
feature_extractor={
"feature_extractor_name": "multimodal",
"version": "v1"
},
namespace="knowledge-base"
)
```
You do not need to define field types or manage embedding dimensions. The feature extractor handles all of this based on your content.
Elasticsearch ingest pipelines handle basic field transforms. Mixpeek collections run ML models on your content: generating embeddings, extracting entities, transcribing audio, and more.
```python theme={null}
# Upload source files - the collection handles everything
client.objects.create(
bucket_id="your-bucket-id",
key_prefix="/articles",
blobs=[
{"property": "content", "data": "s3://your-bucket/article-001.pdf"}
],
namespace="knowledge-base"
)
```
Do not try to bulk-import your Elasticsearch documents or vectors. Re-ingest your source files so the pipeline can extract multi-layered features and build proper lineage.
Elasticsearch's query DSL maps naturally to Mixpeek retriever stages. Each DSL clause becomes a stage in the pipeline.
```json Elasticsearch (before) theme={null}
POST /knowledge-base/_search
{
"query": {
"bool": {
"must": [
{
"knn": {
"field": "embedding",
"query_vector": [0.1, 0.2, ...],
"num_candidates": 50,
"k": 20
}
}
],
"filter": [
{ "term": { "category": "engineering" } },
{ "range": { "published_at": { "gte": "2025-01-01" } } }
]
}
},
"size": 10
}
```
```python Mixpeek (after) theme={null}
# One call - embedding, search, and filtering handled
results = client.retrievers.execute(
retriever_id="article-search",
inputs={
"query": "distributed systems architecture",
"category": "engineering"
},
limit=10,
namespace="knowledge-base"
)
```
Define a retriever that chains stages together. This replaces complex DSL queries with a declarative pipeline.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: knowledge-base" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "article-search",
"stages": [
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"fusion": "rrf",
"final_top_k": 50,
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
},
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"lexical": true,
"top_k": 50
}
]
}
}
},
{
"stage_name": "filter_category",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "category",
"operator": "eq",
"value": "{{INPUT.category}}"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"top_k": 10
}
}
}
]
}'
```
Notice the hybrid approach: semantic search and keyword search run as separate stages, then results are combined through reranking. No need to manually tune BM25 weights against vector scores.
Execute your retriever and compare results against your Elasticsearch baseline.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: knowledge-base" \
-H "Content-Type: application/json" \
-d '{
"inputs": {
"query": "distributed systems architecture",
"category": "engineering"
},
"limit": 10
}'
```
## What You Gain
| Capability | Elasticsearch | Mixpeek |
| ------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------- |
| **Hybrid search** | Manual BM25 + kNN score tuning | Semantic and keyword stages with automatic reranking |
| **Feature extraction** | External embedding generation, custom ingest pipelines | Built-in ML extractors: CLIP, Whisper, LayoutLM, and more |
| **Multimodal** | Text-native; images/video require custom plugins | Native support for video, audio, images, and documents in the same namespace |
| **Storage tiering** | ILM policies you configure and maintain | Automatic tiering: active, cold, archive, managed by the platform |
| **No infrastructure** | Clusters, shards, replicas, JVM tuning | Fully managed API, no cluster operations |
| **Lineage** | Documents disconnected from source files | Trace any result back through document, object, and source file |
| **Multi-stage pipelines** | Complex nested DSL | Declarative stage pipelines: search, filter, rerank, enrich |
## Next Steps
Get Mixpeek running in 10 minutes
Learn about automatic feature extraction
Build multi-stage retrieval pipelines
Understand the data model
# Migrate from Pinecone
Source: https://docs.mixpeek.com/docs/overview/from-pinecone
Move from single-vector search to multi-stage warehouse retrieval
Pinecone is a vector database built for single-embedding KNN search. Mixpeek is a multimodal data warehouse that decomposes files into searchable features, stores them across cost tiers, and reassembles answers through multi-stage retrieval pipelines.
This guide walks you through migrating your search workload from Pinecone to Mixpeek.
## Why Migrate
Pinecone stores and queries individual vectors. Mixpeek processes raw files end-to-end: extracting features, storing documents across tiered storage, and executing multi-stage retrieval pipelines. You stop managing embeddings and start working with content.
| Pinecone | Mixpeek |
| ------------------------------------ | -------------------------------------------------------------------------- |
| You generate embeddings externally | Feature extractors generate embeddings automatically |
| Single-vector KNN per query | Multi-stage pipelines: search, filter, rerank, enrich |
| Flat storage pricing | Tiered storage: hot (active), warm (cold), archive, up to 90% savings |
| Metadata filtering on vector results | Attribute filters, boolean logic, and cross-modal joins as pipeline stages |
| One index per embedding model | One namespace handles multiple modalities and models simultaneously |
## Concept Mapping
| Pinecone | Mixpeek | Notes |
| ----------------- | ------------------------------------- | ----------------------------------------------------------------------------------- |
| Index | Namespace | Top-level container for your data |
| Namespace | Namespace (via `X-Namespace` header) | Tenant or environment isolation within a namespace |
| Vector | Document (with features) | Documents contain extracted features, metadata, and lineage back to the source file |
| Upsert | Object upload + Collection processing | Data flows through the pipeline: upload to bucket, collection triggers extraction |
| Query (top-k KNN) | Retriever execution (multi-stage) | Retrievers chain stages: semantic search, filters, reranking, enrichment |
| Metadata filter | Attribute filter stage | Filters are composable stages in a retrieval pipeline |
## Migration Steps
Set up a namespace to hold your data. This replaces your Pinecone index.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/namespaces \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespace_name": "product-catalog"
}'
```
Define what features to extract from your data. This replaces the external embedding step you had with Pinecone.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/collections \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: product-catalog" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "products",
"feature_extractor": {
"feature_extractor_name": "multimodal",
"version": "v1"
}
}'
```
Upload your source files to a bucket and let the collection process them. Do not try to import your existing Pinecone vectors directly. Mixpeek extracts richer, multi-modal features from your raw content.
Never insert vectors directly into the storage layer. All data must flow through the ingestion pipeline: bucket upload, collection trigger, feature extraction. This ensures proper lineage, validation, and multi-modal indexing.
```bash theme={null}
# Upload objects to a bucket
curl -X POST https://api.mixpeek.com/v1/buckets/{bucket_id}/objects \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: product-catalog" \
-H "Content-Type: application/json" \
-d '{
"key_prefix": "/products",
"blobs": [
{ "property": "image", "data": "s3://your-bucket/product-001.jpg" },
{ "property": "description", "data": "s3://your-bucket/product-001.json" }
]
}'
```
Build a retriever that goes beyond single-vector KNN. Chain semantic search with filters, reranking, and enrichment.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: product-catalog" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "product-search",
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"final_top_k": 50,
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
}
]
}
}
},
{
"stage_name": "filter_category",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "category",
"operator": "eq",
"value": "{{INPUT.category}}"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"top_k": 10
}
}
}
]
}'
```
Execute your retriever and compare results against your Pinecone baseline.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: product-catalog" \
-H "Content-Type: application/json" \
-d '{
"inputs": {
"query": "red running shoes",
"category": "footwear"
},
"limit": 10
}'
```
## Side-by-Side Comparison
The Mixpeek retriever does in one API call what requires multiple steps with Pinecone: embedding generation, vector search, and post-processing.
```python Pinecone theme={null}
import pinecone
from sentence_transformers import SentenceTransformer
# You manage the embedding model
model = SentenceTransformer("all-MiniLM-L6-v2")
# You generate the embedding
query_embedding = model.encode("red running shoes").tolist()
# Single-vector KNN search
pinecone.init(api_key="PINECONE_KEY", environment="us-east-1")
index = pinecone.Index("product-catalog")
results = index.query(
vector=query_embedding,
top_k=10,
filter={"category": "footwear"},
include_metadata=True
)
for match in results["matches"]:
print(match["id"], match["score"], match["metadata"])
```
```python Mixpeek theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="YOUR_API_KEY")
# One call: embedding, search, filter, rerank - all handled
results = client.retrievers.execute(
retriever_id="product-search",
inputs={
"query": "red running shoes",
"category": "footwear"
},
limit=10,
namespace="product-catalog"
)
for doc in results:
print(doc["document_id"], doc["score"], doc["metadata"])
```
## What You Gain
| Capability | Pinecone | Mixpeek |
| -------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------- |
| **Multi-stage retrieval** | Single KNN query; post-processing is your problem | Chain search, filter, rerank, and enrich stages in one pipeline |
| **Automatic feature extraction** | You build and maintain embedding pipelines | Feature extractors handle it: CLIP, Whisper, LayoutLM, and more |
| **Tiered storage** | All vectors at one price tier | Hot, cold, and archive tiers, up to 90% savings on infrequently accessed data |
| **Multimodal search** | One embedding model per index | Search across text, images, video, and audio in the same namespace |
| **No per-query vector fees** | Per-read pricing on every query | Flat API pricing, no per-vector-read charges |
| **Complete lineage** | Vectors disconnected from source files | Trace any result back through document, object, and source file |
## Next Steps
Get Mixpeek running in 10 minutes
Learn about automatic feature extraction
Build multi-stage retrieval pipelines
Understand the data model
# Migrate from Weaviate
Source: https://docs.mixpeek.com/docs/overview/from-weaviate
Upgrade from vector database to multimodal data warehouse
Weaviate is a vector database with built-in vectorization modules. Mixpeek is a multimodal data warehouse that goes further: decomposing files into layered features, storing them across cost tiers, and reassembling answers through multi-stage retrieval pipelines.
This guide walks you through migrating your search workload from Weaviate to Mixpeek.
## Why Migrate
Weaviate introduced multimodal capabilities with modules like `multi2vec-clip` and `img2vec-neural`. Mixpeek builds on this direction but takes a fundamentally different approach.
Where Weaviate adds vector search to a database, Mixpeek starts from the file itself. A single video becomes transcripts, visual embeddings, scene descriptions, and detected entities, each independently searchable, all stored across cost tiers, and reassembled through configurable pipelines.
| Weaviate | Mixpeek |
| --------------------------------------- | --------------------------------------------------------------------- |
| Vectorization modules bolt onto storage | Feature extraction is the core of the pipeline |
| GraphQL queries with vector search | Multi-stage retrieval pipelines: search, filter, rerank, enrich |
| Single storage tier | Tiered storage: hot, cold, archive, up to 90% savings |
| Schema-per-class design | Namespace-level organization with collections per processing pipeline |
| Module-based multimodal support | Native decomposition: one file becomes many searchable layers |
## Concept Mapping
| Weaviate | Mixpeek | Notes |
| -------------------------------- | ------------------------ | ------------------------------------------------------------------------------- |
| Class | Collection | Defines processing pipeline and feature extraction for a data type |
| Property | Document field | Documents contain extracted features, metadata, and source lineage |
| Object (with vector) | Document (with features) | Documents hold multiple feature types, not just one vector |
| Module (e.g., `text2vec-openai`) | Feature Extractor | Built-in extractors: multimodal, image, text, face-identity, document, and more |
| GraphQL query | Retriever execution | Retrievers are multi-stage pipelines, not single queries |
| `nearText` / `nearVector` | Semantic search stage | One stage in a larger pipeline |
| `where` filter | Attribute filter stage | Filters compose as pipeline stages alongside search and ranking |
| Cross-reference | Semantic JOIN / Taxonomy | Connect documents across collections using vector similarity |
## Migration Steps
Replace your Weaviate instance with a Mixpeek namespace.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/namespaces \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespace_name": "media-library"
}'
```
Each Weaviate class becomes a Mixpeek collection with a feature extractor. Instead of choosing a vectorization module, you choose an extractor that matches your content type.
```python Weaviate (before) theme={null}
# Weaviate class definition
client.schema.create_class({
"class": "Article",
"vectorizer": "text2vec-openai",
"properties": [
{"name": "title", "dataType": ["text"]},
{"name": "body", "dataType": ["text"]},
{"name": "category", "dataType": ["text"]}
]
})
```
```python Mixpeek (after) theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="YOUR_API_KEY")
# Collection with automatic feature extraction
collection = client.collections.create(
collection_name="articles",
feature_extractor={
"feature_extractor_name": "multimodal",
"version": "v1"
},
namespace="media-library"
)
```
Upload source files through the Mixpeek pipeline instead of importing Weaviate objects. The pipeline extracts richer features than a single vectorization module.
Do not export vectors from Weaviate and import them into Mixpeek. Re-ingest your source files so the pipeline can extract multi-layered features, build lineage, and index across modalities.
```python theme={null}
# Upload objects to a bucket for processing
client.objects.create(
bucket_id="your-bucket-id",
key_prefix="/articles",
blobs=[
{"property": "content", "data": "s3://your-bucket/article-001.pdf"}
],
namespace="media-library"
)
```
Weaviate's GraphQL queries map to Mixpeek retrievers. The difference: retrievers chain multiple stages together.
```graphql Weaviate (before) theme={null}
{
Get {
Article(
nearText: { concepts: ["machine learning trends"] }
where: {
path: ["category"]
operator: Equal
valueText: "technology"
}
limit: 10
) {
title
body
_additional { score }
}
}
}
```
```python Mixpeek (after) theme={null}
results = client.retrievers.execute(
retriever_id="article-search",
inputs={
"query": "machine learning trends",
"category": "technology"
},
limit=10,
namespace="media-library"
)
```
Go beyond what Weaviate's query language supports. Chain semantic search with attribute filters, reranking, and enrichment in a single retriever.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: media-library" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "article-search",
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"final_top_k": 50,
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
}
]
}
}
},
{
"stage_name": "filter_category",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "category",
"operator": "eq",
"value": "{{INPUT.category}}"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"top_k": 10
}
}
}
]
}'
```
Execute retrievers and validate results against your Weaviate baseline.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: media-library" \
-H "Content-Type: application/json" \
-d '{
"inputs": {
"query": "machine learning trends",
"category": "technology"
},
"limit": 10
}'
```
## What You Gain
| Capability | Weaviate | Mixpeek |
| ------------------------- | -------------------------------------- | --------------------------------------------------------------------------------- |
| **File decomposition** | Vectorize one property at a time | Decompose a file into multiple searchable layers automatically |
| **Multi-stage retrieval** | Single query with optional filters | Chain search, filter, rerank, and enrich stages in one pipeline |
| **Tiered storage** | All data at one storage tier | Hot, cold, and archive tiers, up to 90% savings |
| **Cross-modal search** | Modules per class, limited cross-modal | Native cross-modal: text query finds video moments, audio segments, image regions |
| **No infrastructure** | Self-hosted or managed cluster | Fully managed API, no clusters, no module configuration |
| **Complete lineage** | Objects in a class | Trace results back through document, object, and source file |
If you are using Weaviate's `multi2vec-clip` for image-text search, Mixpeek's multimodal extractor handles the same use case and adds video, audio, and document support in the same namespace.
## Next Steps
Get Mixpeek running in 10 minutes
Learn about automatic feature extraction
Build multi-stage retrieval pipelines
Understand the data model
# Introduction
Source: https://docs.mixpeek.com/docs/overview/introduction
Mixpeek gives AI agents the ability to see, hear, and understand multimodal content
Your AI agent can read text. It cannot watch a video, scan a photo for faces, or search audio by what was said. Mixpeek is the infrastructure layer that gives agents access to video, images, audio, and documents through a single API.
Create a workspace and run your first multimodal search — index your own video, images, audio, or documents in minutes.
## How It Works
Upload video, images, audio, and documents to [Buckets](/docs/platform/data-model#create-a-bucket). Mixpeek runs feature extraction automatically — faces, objects, transcripts, embeddings, and structured metadata all get indexed into searchable [Collections](/docs/overview/concepts).
![Indexing multimodal content into semantic layers]()
See this running on real content — 7 PDFs that fan out into 59 searchable knowledge-graph nodes — in the [sample data](/docs/overview/sample-data#diagram-1-how-content-decomposes).
Build retrieval pipelines that your agent calls. Semantic search, face search, object search, transcript search — chain them together into multi-stage [Retrievers](/docs/retrieval/retrievers) and expose them as a single endpoint.
![Multi-stage retrieval pipeline]()
Run this exact pipeline against the [sample data](/docs/overview/sample-data#diagram-2-how-retrieval-reassembles) right now — no account or API key needed.
Wire Mixpeek into your agent as a [LangChain tool](/docs/integrations/agents#langchain), an [MCP server](/docs/integrations/agents#mcp-model-context-protocol), or a direct REST call. Your agent sends a query, gets structured results back, and acts on them.
**Already have embeddings?** Skip extraction entirely — bring your own vectors and search instantly with the [Mixpeek Vector Store](/docs/vector-store/overview). 1M vectors free, 60 seconds to first query.
## Quickstart
Index multimodal content and search it in under 10 minutes — LangChain, MCP, or REST
## What Can You Build?
Pick an outcome and follow the guide — each one is end-to-end and copy-pasteable.
Find moments by what's shown or said — visual + speech embeddings
Use an image or clip as the query to find visually similar media
Combine dense vectors with keyword/BM25 and attribute filters
Push the most relevant results to the top with a cross-encoder
Group, label, and visualize a collection automatically
Classify content against your own hierarchy with a multimodal join
Enforce access control with built-in ACLs or external OpenFGA
Use Mixpeek as a standalone vector store — upsert and query instantly
Improve ranking automatically from click and reward signals
Re-extract, validate, and cut over to a new model with no downtime
## What Gets Extracted
| File Type | Extracted Features |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **[Video](/docs/processing/extractors/multimodal)** | Face embeddings (ArcFace 512D), scene descriptions (Gemini), visual embeddings (Vertex AI 1408D), transcripts (Whisper), transcript embeddings (E5-Large 1024D), keyframes |
| **[Images](/docs/processing/extractors/image)** | Visual embeddings (SigLIP 768D or Vertex AI 1408D), face embeddings (ArcFace 512D), OCR text, descriptions, structured extraction |
| **[Audio](/docs/processing/extractors/multimodal)** | Transcripts (Whisper), transcript embeddings (E5-Large 1024D), multimodal audio embeddings (Vertex AI 1408D) |
| **[Documents](/docs/processing/extractors/document)** | Text chunks, text embeddings (E5-Large 1024D), OCR for scanned PDFs, structured extraction |
Each extracted feature becomes an independently searchable document. A single video can produce hundreds of documents — one per face, one per transcript segment, one per scene.
## Key Concepts
* **Namespaces** isolate data between tenants, environments, or projects. Every API request includes a namespace header.
* **Buckets** hold your raw files. Upload once, process many ways.
* **Collections** define what gets extracted. Each collection runs a feature extractor (CLIP, Whisper, LayoutLM, etc.) against objects in a bucket.
* **Retrievers** are search pipelines you configure in JSON. Chain stages together — vector search, face matching, filters, re-ranking — and expose the result as one endpoint your agent calls.
## Next Steps
Understand namespaces, buckets, collections, and retrievers in depth
See how namespaces, buckets, collections, objects, and features fit together
Learn what each extractor does and how to configure it
Step-by-step guides for common use cases
# Quickstart
Source: https://docs.mixpeek.com/docs/overview/quickstart
Start searching in under 60 seconds with BYO vectors, or index multimodal content with the managed platform
Get a Mixpeek API key from [mixpeek.com/start](https://mixpeek.com/start?utm_source=docs\&utm_medium=docs_quickstart\&utm_campaign=mvs_onramp), then pick your path — BYO vectors for instant search, or the managed platform: point it at your files, pick what you want to search by, and extraction is handled for you.
**API-first?** Your first key is minted at signup, but you don't need the dashboard after that — create, scope, and rotate additional keys entirely from the API: [Create](/docs/api-reference/organization-api-keys/create-api-key) · [Rotate](/docs/api-reference/organization-api-keys/rotate-api-key) · [List](/docs/api-reference/organization-api-keys/list-api-keys).
**New to Mixpeek?** Follow the first tab — **MVS Standalone** — end to end. It's the shortest path to a working search (under 60 seconds) and needs nothing but an API key. The other tabs wire that same search into an agent (LangChain, MCP) or call it over REST once you've created a retriever.
Already have embeddings? Skip extraction — search in under 60 seconds.
```bash theme={null}
pip install mixpeek
export MIXPEEK_API_KEY="mxp_sk_replace_me"
```
```python theme={null}
import os
from mixpeek import Mixpeek
client = Mixpeek(api_key=os.environ["MIXPEEK_API_KEY"])
ns = client.namespaces.create(
namespace_name="my-search", # required, 3–64 chars; the id is auto-generated
mode="standalone", # bring your own vectors — no extractors
)
```
```python theme={null}
client.namespaces.documents.upsert(
namespace_id=ns["namespace_id"],
documents=[{
"document_id": "doc-001",
"vectors": {"embedding": [0.12, -0.34, 0.56]}, # your embedding
"payload": {"title": "First document", "category": "demo"},
}],
)
```
Vectors are searchable within **10–30 seconds** of upsert (the index flushes in the background). For immediate verification, add a short poll:
```python theme={null}
import time
# Wait for indexing
time.sleep(15)
results = client.search(
namespace_id=ns["namespace_id"],
queries=[{
"vector_name": "embedding",
"vector": [0.15, -0.28, 0.44], # query embedding
"top_k": 10,
}],
)
for hit in results["documents"]:
print(f"{hit['score']:.3f} | {hit['payload']['title']}")
```
MVS infers vector dimensions on first write — no schema needed. When you're ready for automatic extraction, [promote your namespace to Managed](/docs/vector-store/promote).
Build a LangChain agent with a `video_search` tool backed by Mixpeek.
```bash theme={null}
pip install mixpeek langchain langchain-openai
export MIXPEEK_API_KEY="mxp_sk_replace_me"
export OPENAI_API_KEY="sk-replace_me"
```
```python theme={null}
import os
from mixpeek import Mixpeek
client = Mixpeek(api_key=os.environ["MIXPEEK_API_KEY"])
# Managed namespace — `namespace_name` is the only required field. You don't
# register extractors up front; the collection's `features` (below) auto-provision
# whatever pipeline the namespace needs.
ns = client.namespaces.create(namespace_name="agent-video-demo")
# Scope the client to the namespace you just created — every call below sends it
# as the X-Namespace header, which bucket/collection create and search require.
client.namespace = ns["namespace_id"]
bucket = client.buckets.create(
bucket_name="demo-videos",
# "video" is the standard source property name per modality —
# collections created with `features` wire to it automatically.
bucket_schema={"properties": {"video": {"type": "video"}}},
)
# Create the collection by picking a feature — Mixpeek resolves the
# pipeline and default input wiring (see /processing/features).
col = client.collections.create(
collection_name="video-scenes",
source={"type": "bucket", "bucket_ids": [bucket["bucket_id"]]},
features=["video_search"],
)
```
```python theme={null}
# Upload the video to the bucket, then trigger the collection to process it.
client.buckets.upload(
bucket["bucket_id"],
url="https://storage.googleapis.com/mixpeek-public-demo/videos/sample-product-demo.mp4",
)
result = client.collections.trigger(col["collection_id"])
```
Poll `client.tasks.get(task_id=result["task_id"])` until the status is terminal — `COMPLETED` or `COMPLETED_WITH_ERRORS` (1-5 min). Don't wait only for `COMPLETED`, or a batch that finished with some failed items will hang your loop.
```python theme={null}
from langchain.tools import Tool
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
ret = client.retrievers.create(
retriever_name="agent-video-search",
input_schema={"query_text": {"type": "text", "required": True}},
collection_identifiers=[col["collection_id"]],
stages=[{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://universal_extractor@v1/gemini-embedding-2",
"query": {"input_mode": "text", "value": "{{INPUT.query_text}}"},
"top_k": 50
}],
"final_top_k": 20
}
}
}]
)
def search_video(query: str) -> str:
results = client.retrievers.execute(
retriever_id=ret["retriever_id"],
inputs={"query_text": query},
)
return "\n".join(
f"[{r.get('start_time_s','?')}s] (score: {r['score']:.3f}) {r.get('description','')}"
for r in results["documents"]
) or "No results found."
video_tool = Tool(name="video_search", description="Search indexed video by natural language", func=search_video)
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "You answer questions about video content. Use video_search to find relevant moments."),
("human", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad"),
])
agent = create_openai_tools_agent(llm, [video_tool], prompt)
executor = AgentExecutor(agent=agent, tools=[video_tool], verbose=True)
print(executor.invoke({"input": "What product features are shown?"})["output"])
```
Add Mixpeek as a tool in Claude Desktop or Claude Code — no code required.
Open `claude_desktop_config.json` and add:
```json theme={null}
{
"mcpServers": {
"mixpeek-retrieval": {
"url": "https://mcp.mixpeek.com/retrieval/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
Add `.mcp.json` to your project root:
```json theme={null}
{
"mcpServers": {
"mixpeek-retrieval": {
"url": "https://mcp.mixpeek.com/retrieval/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
Restart Claude and ask: *"What Mixpeek tools do you have access to?"*
| Scope | URL | Tools |
| ------------- | --------------------------------------- | ----- |
| **Full** | `https://mcp.mixpeek.com/mcp` | 48 |
| **Ingestion** | `https://mcp.mixpeek.com/ingestion/mcp` | 20 |
| **Retrieval** | `https://mcp.mixpeek.com/retrieval/mcp` | 11 |
| **Admin** | `https://mcp.mixpeek.com/admin/mcp` | 17 |
See the [full MCP reference](/docs/agent-integrations/mcp) for per-retriever servers and troubleshooting.
Call `POST /v1/retrievers/{id}/execute` from any language.
```python theme={null}
import os
import requests
MIXPEEK_API_KEY = os.environ["MIXPEEK_API_KEY"]
MIXPEEK_NAMESPACE = os.environ["MIXPEEK_NAMESPACE"] # the namespace_id (ns_...) returned at creation
RETRIEVER_ID = os.environ["MIXPEEK_RETRIEVER_ID"] # the retriever_id (ret_...) from retrievers.create
HEADERS = {
"Authorization": f"Bearer {MIXPEEK_API_KEY}",
"X-Namespace": MIXPEEK_NAMESPACE,
"Content-Type": "application/json",
}
resp = requests.post(
f"https://api.mixpeek.com/v1/retrievers/{RETRIEVER_ID}/execute",
headers=HEADERS,
json={"inputs": {"query_text": "safety regulations"}},
)
results = resp.json()["documents"]
```
Each result is a document with a top-level `score` plus the document's own fields (e.g. `content`, `start_time_s`) flattened to the top level. Wrap this call as a tool in any agent framework — OpenAI function calling, CrewAI, LlamaIndex, or plain HTTP.
See the [API reference](/docs/api-reference/retrievers/execute-retriever-auto-optimized) for full request and response details.
# Explore the sample data
Source: https://docs.mixpeek.com/docs/overview/sample-data
Run real multimodal queries against a live sample namespace — no account, no API key, no setup
The fastest way to understand Mixpeek is to run it. **`sample-data`** is a live
namespace loaded with real content — video, ad creatives, PDFs, and a product
catalog, already extracted and searchable — and the retrievers below are published
publicly, so **every example on this page runs with no account and no API key**.
Copy any `curl` here into a terminal right now. It will return real results.
It is also the working version of the two diagrams on the
[Introduction](/docs/overview/introduction) page: how content decomposes into layers,
and how a retriever reassembles those layers at query time.
## Run your first query
No `Authorization` header. No `X-Namespace`. Just a query.
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/public/retrievers/sample-document-knowledge/execute" \
-H "Content-Type: application/json" \
-d '{
"inputs": { "query": "chevron quarterly revenue" },
"pagination": { "method": "offset", "page_size": 10, "page_number": 1 }
}'
```
```python Python theme={null}
import requests
resp = requests.post(
"https://api.mixpeek.com/v1/public/retrievers/sample-document-knowledge/execute",
json={
"inputs": {"query": "chevron quarterly revenue"},
"pagination": {"method": "offset", "page_size": 10, "page_number": 1},
},
)
for hit in resp.json()["results"]:
print(hit["score"], hit["title"], "—", hit["description"][:60])
```
```javascript JavaScript theme={null}
const resp = await fetch(
'https://api.mixpeek.com/v1/public/retrievers/sample-document-knowledge/execute',
{
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
inputs: { query: 'chevron quarterly revenue' },
pagination: { method: 'offset', page_size: 10, page_number: 1 },
}),
}
);
const { results } = await resp.json();
```
On this **public** path, hits come back under **`results`**. This differs from the
authenticated path — `POST /v1/retrievers/{retriever_id}/execute` returns hits
under `documents`, where `results` is a deprecated field that is no longer
populated. Read the field that matches the path you called, or you will parse an
empty list from a working query.
## Diagram 1 — how content decomposes
The first diagram on the Introduction page (source → temporal segments → detected
entities) is a real two-tier pipeline here. Each bucket is extracted into a
**tier-0** collection, and tier-0 is extracted *again* into a **tier-1**
collection — the layer you actually search.
| Bucket | Tier 0 — segments | Tier 1 — entities |
| ------------------ | ---------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `video-library` | `video-scenes` — scene segments with transcript + on-screen text | `scene-entities` — faces per scene (512-d ArcFace, with crops and bounding boxes) |
| `ad-creatives` | `ad-visual` — creative-level embeddings | `ad-entities` — faces per creative |
| `document-archive` | `document-text` — PDF text + embeddings | `document-graph` — a knowledge graph: **7 PDFs fan out into 59 nodes** |
| `product-catalog` | `product-images` (SigLIP) and `product-text` (E5) | `product-attributes` — structured attributes per product |
The `document-archive` row is the clearest illustration: decomposition is a real
**one-to-many fan-out**, not a reformat. Seven PDFs become 59 independently
searchable nodes — and each node still knows which PDF it came from, which is what
makes the next section work.
## Diagram 2 — how retrieval reassembles
The second diagram (query → vector search → entity filter → similarity rank →
enrich) is exactly what **Scene Moment Reassembly** does. The other three apply the
same reassembly idea to different content. Every one is runnable as-is.
`sample-scene-reassembly` — all four stages in the diagram's order:
`feature_search` (vector) → `attribute_filter` (genre) → `rerank`
(cross-encoder) → `document_enrich` (join the tier-1 faces back in).
```json theme={null}
{ "inputs": { "query": "functions in C programming" } }
```
Returns 8 scene moments — `title`, `description`, `start_time`/`end_time`,
`genre` — each carrying a `face_crop_url` for the person detected **in that
scene**. That join is the whole point: the tier-0 scene and its tier-1 entities
come back reassembled into one result, so you get "who is on screen, and when."
`sample-person-across-media` — `feature_search` over ArcFace embeddings spanning
**two buckets' tier-1 collections** (`scene-entities` + `ad-entities`) →
`document_enrich` to join back to the source.
```json theme={null}
{
"inputs": {
"query_image": "https://mixpeek-public-demo.s3.us-east-2.amazonaws.com/demo-faces/mixpeek-brand-reel-face.jpg"
}
}
```
Returns **2 hits with different `title`s** — *Pipeline is product reel* and
*Vector DB scam reel*. That is the lesson: the same person found across two
**different** documents, not a single self-match.
Swap in any face image URL. A face that does not appear in the sample content
correctly returns nothing — that is reverse-face search working, not failing.
This retriever needs an image; a text query has no face to embed.
`sample-document-knowledge` — `feature_search` over the graph **nodes** →
`document_enrich` to join back to the parent PDF.
```json theme={null}
{ "inputs": { "query": "chevron quarterly revenue" } }
```
Each hit's `description` is the matched **node** text and `object_type` tells you
what it is (e.g. `paragraph`) — but `title` is the **parent document**
(*Chevron Financial Filing*). You search fine-grained pieces and still get the
source back. Several hits often share one parent: that is the 7-PDFs-to-59-nodes
fan-out from Diagram 1, seen from the query side.
`sample-hybrid-catalog` — a single `feature_search` running **two modalities at
once** (SigLIP text-to-image over `product-images` + E5 text over
`product-text`), fused with [RRF](/docs/relevance/fusion-strategies).
```json theme={null}
{ "inputs": { "query": "adjustable LED lamp" } }
```
Returns products with `product_name`, `price`, `category`, and `description`.
A product can appear from **both** its imagery and its text — the same item
reached down two independent paths and merged into one ranking.
RRF scores are rank-based (all hits land near `0.017`), so read the **order**,
not the magnitude — the score is not a similarity. The sample catalog is small
(8 products), so a broad query returns most of it; ranking is what changes.
Media fields (`thumbnail_url`, `video_segment_url`, `face_crop_url`) come back
from a retriever as signed URLs minted fresh on every request, and they expire
after 24 hours. Do not expect a URL you copied yesterday to still resolve.
Browsing the same documents through `POST /v1/documents/list` returns raw
`gs://` values instead, unless you ask for signing. See
[Media URLs](/docs/vector-store/documents#media-urls).
## Make it yours
These pipelines use the same stages you would configure yourself. Fork the
namespace in Studio for an editable copy, or read how the pieces work:
Compose search stages into a pipeline
How content is decomposed into searchable layers
# Ingest Data
Source: https://docs.mixpeek.com/docs/platform/data-model
Get your files into Mixpeek — namespaces, buckets, objects, uploads, and batching
Create a namespace, connect a bucket, and run your first batch in minutes — no API key setup required.
## Set Up a Namespace
Every project starts with a namespace — the isolation boundary for all your resources. Use one per environment (dev, staging, prod) or per tenant.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "namespace_name": "production" }'
```
That's the whole managed create — `namespace_name` is the only required field. You don't register extractors up front: creating a collection with `features: [...]` auto-provisions the pipelines the namespace needs (see [Features](/docs/processing/features)). The old `feature_extractors` field is deprecated.
### Standalone (bring your own vectors)
To upsert your own vectors instead of Mixpeek-managed extraction, declare the index shapes with `vector_configs`:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespace_name": "byo-vectors",
"vector_configs": [
{ "name": "clip", "dimension": 768, "metric": "cosine" }
]
}'
```
`vector_configs` is a **list** of `{ name, dimension, metric }` objects (not a single object). `name` and `dimension` are required; `metric` defaults to `cosine` (also `euclidean`, `dot_product`). Passing `vector_configs` with no `features`/`feature_extractors` **infers standalone mode** — no `mode` field needed. Omit `vector_configs` entirely and indexes auto-create on first upsert.
Every subsequent request needs two headers: `Authorization: Bearer mxp_sk_...` and `X-Namespace: ns_...`.
[Namespace API →](/docs/api-reference/namespaces/create-namespace)
## Create a Bucket
Buckets are schema-validated containers for raw files. Define what blob types you accept (text, image, audio, video, json, binary).
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "product-catalog",
"bucket_schema": {
"properties": {
"product_text": { "type": "text", "required": true },
"hero_image": { "type": "image" }
}
}
}'
```
[Bucket API →](/docs/api-reference/buckets/create-bucket)
### Storage class
Pass an optional `storage_class` on create/update to pick a cost tier for a bucket's objects. It's provider-agnostic — mapped to your object store:
| `storage_class` | GCS | S3 / MinIO | Best for |
| -------------------- | -------- | ------------ | ---------------------------- |
| `standard` (default) | STANDARD | STANDARD | Hot, frequently-read buckets |
| `nearline` | NEARLINE | STANDARD\_IA | Warm / occasional access |
| `coldline` | COLDLINE | GLACIER\_IR | Cold / rare access |
| `archive` | ARCHIVE | GLACIER | Long-term retention |
**Applied on write for sync-based ingestion; broader rollout in progress.** For buckets fed by a storage **sync** (S3, GCS, Drive, RSS, and other sources — the primary media path), the tier is set on each object at write time. Tiering for **direct uploads** (`POST /objects`) and **presigned client uploads**, plus retroactive re-tiering of **existing** objects, are a separate backend follow-up (in progress). Keep hot, retriever-source buckets on `standard`; reserve cheaper tiers for large write-once/read-occasionally media.
## Connect External Storage
Sync files directly from your existing cloud storage instead of uploading manually. Mixpeek reads from your provider — no migration needed. This is a **two-step** flow: create a reusable **connection** (holds the credentials, lives at the organization level), then attach a **sync** to a bucket that references it.
**Step 1 — Create the connection** (once per provider account):
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/connections" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Production S3",
"provider_type": "s3",
"provider_config": {
"provider_type": "s3",
"bucket_name": "my-source-bucket",
"region": "us-east-1",
"credentials": {
"access_key_id": "AKIA...",
"secret_access_key": "..."
}
}
}'
```
The response includes a `connection_id` (`conn_...`). Credentials are encrypted at rest and reusable across buckets.
**Step 2 — Attach a sync** to your bucket (flat body — no wrapper objects):
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_abc123",
"source_path": "/videos/",
"sync_mode": "continuous",
"polling_interval_seconds": 3600
}'
```
Then trigger the first sync:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/trigger" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
After the initial sync, new files are picked up automatically at the configured polling interval. Use `continuous` mode (vs `initial_only`) to keep picking up new and changed files — only new or modified files since the last sync are processed, so existing files aren't reprocessed.
| Provider | Auth Method | S3-Compatible |
| -------------------- | ----------------------------- | ------------- |
| AWS S3 | IAM User / Role | Native |
| Google Cloud Storage | Service Account Key | No |
| Azure Blob Storage | Access Key / Managed Identity | No |
| Cloudflare R2 | R2 API Token | Yes |
| Backblaze B2 | Application Key | Yes |
| Wasabi | Access Key | Yes |
| Tigris | Access Key | Yes |
| Box | OAuth | No |
| Mux | API Token | No |
| Supabase | Service Key | Yes |
See [Object Storage providers](/docs/integrations/object-storage/overview) for provider-specific setup guides.
[Sync API →](/docs/api-reference/bucket-syncs/create-sync-configuration)
## Register Objects
Objects are raw multimodal assets within a bucket. Two paths:
**URL references** — point to files in your existing storage:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/objects" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"key_prefix": "/products",
"blobs": [
{ "property": "hero_image", "type": "image", "data": "https://example.com/photo.jpg" },
{ "property": "product_text", "type": "text", "data": "Wireless headphones" }
]
}'
```
**Direct uploads** — upload to Mixpeek-managed storage via presigned URLs:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/uploads" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{ "filename": "photo.jpg", "content_type": "image/jpeg" }'
```
Then PUT the file to the returned `presigned_url` and confirm with `POST /uploads/{id}/confirm`.
For bulk imports, use [batch uploads](/docs/api-reference/bucket-uploads/batch-create-uploads) or connect your object storage via [sync configurations](/docs/integrations/object-storage/overview).
[Object API →](/docs/api-reference/bucket-objects/create-object) · [Upload API →](/docs/api-reference/bucket-uploads/create-upload)
## Process with Batches
Batches group objects for extraction. Create a batch, then submit it:
```bash theme={null}
# Create batch
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/batches" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{ "object_ids": ["obj_abc", "obj_def"] }'
# Submit for processing
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/batches/$BATCH_ID/submit" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
### Batch Lifecycle
```
DRAFT → QUEUED → PROCESSING → COMPLETED
↘ ↘
FAILED COMPLETED_WITH_ERRORS
```
Poll `GET /v1/buckets/{id}/batches/{id}` until the status is terminal — `COMPLETED`, `COMPLETED_WITH_ERRORS`, `FAILED`, or `CANCELED` (a poller that waits only for `COMPLETED` hangs on partial success) — or use [webhooks](/docs/platform/operations#webhooks) to get notified on `batch.completed`.
[Batch API →](/docs/api-reference/bucket-batches/create-batch)
# Extract Features
Source: https://docs.mixpeek.com/docs/platform/processing
Turn raw files into searchable documents by picking what you want to search by per file type — collections run the pipeline for you
## Collections
Collections bind a bucket to a processing pipeline. You choose **[features](/docs/processing/features)** — what you want to search by (visual similarity, faces, on-screen text, ...) — and Mixpeek resolves the pipeline internally. When you submit a batch, the engine processes each object and produces searchable documents.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/collections" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "product-images",
"source": { "type": "bucket", "bucket_ids": ["'$BUCKET_ID'"] },
"features": ["image_search"]
}'
```
Discover the menu with `GET /v1/collections/features` — every modality's unit, base feature, and add-ons (faces, on-screen text, layout, ...), with live rates. Add-on features like `faces` create a companion collection over the same source, so each pipeline's outputs stay independently versioned and queryable.
A single object can feed multiple collections — each extracting different features. Documents retain lineage to the source object via `root_object_id`.
[Features guide →](/docs/processing/features) · [Collection API →](/docs/api-reference/collections/create-collection)
Existing configs using explicit `feature_extractor` objects keep working as a deprecated alias — see the [migration guide](/docs/processing/extractor-migration). For advanced wiring (custom input mappings, field passthrough, parameters), see [Pipeline Configuration](/docs/processing/feature-extractors).
### Know the cost before you run
`POST /v1/organizations/billing/estimate` quotes planned ingestion in dollars — same rating engine that bills you:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/billing/estimate" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"items": [{ "mime_type": "video/mp4", "minutes": 42 }],
"features": ["faces", "onscreen_text"]
}'
```
See [Billing & Pricing](/docs/platform/billing) for units, rates, and tier usage pools.
### Embedding task
Instruction-aware embedding models use a **task hint** to optimize the embedding for a specific downstream use case. Set `embedding_task` at the collection level so it applies to every task-aware model in the pipeline.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/collections" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "product-clusters",
"embedding_task": "clustering",
"source": { "type": "bucket", "bucket_ids": ["'$BUCKET_ID'"] },
"features": ["text_search"]
}'
```
| Task | Use Case | Default |
| --------------------- | -------------------------------------------- | ------- |
| `retrieval_document` | Search: find documents from queries | **Yes** |
| `retrieval_query` | Rare at index time — query-side is automatic | No |
| `semantic_similarity` | Symmetric comparison (dedup, matching) | No |
| `classification` | Document categorization pipelines | No |
| `clustering` | Grouping documents into clusters | No |
You almost never need to set this. The default `retrieval_document` is correct for search, and at query time Mixpeek automatically uses `retrieval_query`. Only override for clustering, classification, or symmetric similarity. Non-instruction-aware models ignore this setting.
## Feature URIs
Every extracted feature is addressed by a URI that pins it to a specific pipeline version:
```
mixpeek://{pipeline_name}@{version}/{output_name}
```
Feature URIs are referenced by retriever stages, taxonomies, and clustering jobs. They guarantee query-time compatibility with the extraction pipeline — swap the URI, re-embed, everything downstream stays consistent. Discover the URIs a collection produces with `GET /v1/collections/{id}/features` rather than constructing them by hand; see [Pipeline Configuration](/docs/processing/feature-extractors#feature-uris-the-stable-contract).
## Tiered pipelines
When a batch is submitted, the engine runs a DAG of pipelines:
1. **Tier 1** collections process raw objects from the bucket
2. **Tier 2** collections consume Tier 1 documents as input
3. Each tier waits for dependencies before executing
```
video → scenes (Tier 1) → faces per scene (Tier 2) → expressions per face (Tier 3)
```
Collections define the pipeline through their `source` and feature configuration. Dependencies are resolved automatically. See [Multi-Tier Feature Extraction](/docs/processing/multi-tier-extractors).
## What Mixpeek extracts
| Modality | Unit | Base feature | Add-on features |
| -------- | ---------------- | -------------------------------------------------- | ------------------------------------------------------------------------- |
| Image | per image | `image_search` — visual embeddings | `faces`, `multimodal_understanding` |
| Video | per minute | `video_search` — scene embeddings | `faces`, `onscreen_text`, `audio_fingerprint`, `multimodal_understanding` |
| Audio | per minute | `audio_search` — acoustic fingerprint + embeddings | `multimodal_understanding` |
| Document | per page | `document_search` — page embeddings | `faces`, `document_layout`, `multimodal_understanding` |
| Text | per token | `text_search` — semantic embeddings | `multimodal_understanding` |
| Web | per crawled page | `web_crawl` — crawl + extract | — |
Clustering and taxonomy enrichment are included on every modality at no additional charge. The live menu (with rates) is always `GET /v1/collections/features` — see the [Features guide](/docs/processing/features).
## Custom extractors
For extraction logic beyond the built-in features, build custom extractors:
```bash theme={null}
pip install mixpeek
mixpeek plugin init my-extractor # Scaffold from template
mixpeek plugin test my-extractor # Validate locally
mixpeek plugin publish my-extractor # Upload and deploy
```
Custom extractors run on managed infrastructure with access to GPU/CPU resources, HuggingFace models, and LLM services. Once published, select yours in any collection as `features: ["custom:my-extractor"]` — it prices per unit from its declared compute profile, exactly like native features.
See the [full custom extractors guide](/docs/processing/custom-extractors) for manifest format, pipeline hooks, security constraints, and deployment lifecycle.
[Custom Extractors →](/docs/processing/custom-extractors)
# Syncs
Source: https://docs.mixpeek.com/docs/platform/syncs
Automatically ingest files from external storage into Mixpeek buckets on a schedule
Syncs pull files from your existing storage providers into Mixpeek buckets on a schedule. No migration required — connect your cloud storage, configure what to sync, and Mixpeek keeps your bucket up to date.
## How Syncs Work
```
┌──────────────┐ poll ┌──────────────┐ create ┌──────────────┐
│ External │◄──────────────│ Sync │───objects────►│ Bucket │
│ Storage │───file list──►│ Worker │ │ │
│ (S3, GCS, │ │ │───submit─────►│ Collection │
│ Mux, etc.) │ │ │ batches │ Pipeline │
└──────────────┘ └──────────────┘ └──────────────┘
│
▼
Resume cursor,
metrics, DLQ
```
1. **Poll** — At each interval, Mixpeek lists files from your storage provider using the configured `source_path` and filters.
2. **Filter** — Files are matched against glob patterns, size limits, MIME types, and provider-specific metadata filters.
3. **Create objects** — Matching files are registered in the target bucket. Duplicates are skipped by default via source tracking.
4. **Submit batches** — Objects are grouped into batches and submitted to the bucket's collection pipeline for processing.
5. **Checkpoint** — A resume cursor is saved so the next poll picks up where the last one left off.
## Sync Modes
| Mode | Behavior | Use Case |
| -------------- | ---------------------------------------------- | --------------------------------------------------------- |
| `continuous` | Polls repeatedly at `polling_interval_seconds` | Ongoing ingestion — new files are picked up automatically |
| `initial_only` | Runs once, then stops | One-time backfill or migration |
## Create a Sync
Create a [storage connection](/docs/integrations/object-storage/overview) with credentials for your provider.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_abc123",
"source_path": "/videos/2025/",
"sync_mode": "continuous",
"polling_interval_seconds": 300,
"skip_duplicates": true,
"file_filters": {
"include_patterns": ["*.mp4", "*.mov"],
"max_size_bytes": 5368709120
}
}'
```
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/trigger" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
After the initial run, continuous syncs poll automatically at the configured interval.
## Configuration Reference
### Core Settings
| Field | Type | Default | Description |
| -------------------------- | ------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connection_id` | string | required | Storage connection to pull from |
| `source_path` | string | required | Path in external storage (format varies by provider) |
| `sync_mode` | string | `continuous` | `continuous` or `initial_only` |
| `polling_interval_seconds` | int | `300` | Poll frequency (30–900 seconds) |
| `batch_size` | int | `50` | Files per batch (1–100) |
| `skip_duplicates` | bool | `true` | Skip files already in the bucket |
| `reconcile` | object | — | Reconcile on source change: `on_delete` (cascade-delete objects when the source asset is removed, default `true`), `on_update` (propagate metadata changes + re-extract, default `true`), `on_filter_drift` (drop objects that no longer match filters, default `true`) |
### File Filters
Narrow which files get synced. All filters combine with AND logic.
```json theme={null}
{
"file_filters": {
"include_patterns": ["*.mp4", "*.mov", "*.webm"],
"exclude_patterns": ["*/drafts/*", "*_temp.*"],
"min_size_bytes": 1024,
"max_size_bytes": 5368709120,
"modified_after": "2025-01-01T00:00:00Z",
"mime_types": ["video/mp4", "video/quicktime"]
}
}
```
### Metadata Filters
Filter on provider-specific metadata fields. Useful for syncing only assets that match certain tags, statuses, or custom fields in your storage system.
```json theme={null}
{
"file_filters": {
"metadata_filters": [
{ "field": "status", "operator": "equals", "value": "approved" },
{ "field": "tags", "operator": "contains", "value": "hero" }
]
}
}
```
Supported operators: `equals`, `not_equals`, `contains`, `not_contains`, `gt`, `lt`, `gte`, `lte`, `exists`.
### Schema Mapping
Map provider metadata to bucket schema fields during sync, so structured data arrives alongside your files.
```json theme={null}
{
"schema_mapping": {
"mappings": {
"product_name": { "target_type": "field", "source": { "type": "metadata", "key": "title" } },
"category": { "target_type": "field", "source": { "type": "tag", "key": "category" } }
}
}
}
```
## Lifecycle Management
### Pause and Resume
Temporarily stop a sync without losing progress:
```bash Pause theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/pause" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
```bash Resume theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/resume" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
### Manual Trigger
Force a sync to run immediately, outside the polling schedule:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/trigger" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
### Monitoring
Check sync status and metrics:
```bash theme={null}
curl "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
Response includes:
| Metric | Description |
| ------------------------ | -------------------------------------------------------- |
| `total_files_discovered` | Cumulative files found in source |
| `total_files_synced` | Successfully synced files |
| `total_files_failed` | Files that failed after retries (sent to DLQ) |
| `total_bytes_synced` | Total data transferred |
| `last_sync_at` | When the last sync completed |
| `next_sync_at` | When the next poll is scheduled |
| `consecutive_failures` | Sequential failure count (auto-suspends after threshold) |
## Robustness
Syncs are designed for unattended, long-running operation:
* **Distributed locking** prevents concurrent runs of the same sync
* **Resume cursors** checkpoint progress so interrupted syncs pick up where they left off
* **Dead letter queue** retries failed files up to 3 times before marking them as failed
* **Auto-suspend** pauses syncs after consecutive failures to prevent runaway errors
* **Idempotent ingestion** uses source tracking to never duplicate objects on retries
* **Reconciliation** (the `reconcile` object) cascades source deletes (`on_delete`), propagates metadata updates (`on_update`), and drops objects that no longer match your filters (`on_filter_drift`) — all default `true`
If a sync gets stuck (e.g., a worker crashed mid-run), use the [force unlock endpoint](/docs/api-reference/bucket-syncs/force-unlock-sync-configuration) to release the distributed lock.
[Sync API reference →](/docs/api-reference/bucket-syncs/create-sync-configuration)
# Custom Extractor API (Dedicated Infrastructure)
Source: https://docs.mixpeek.com/docs/processing/custom-extractor-api
HTTP reference for the custom-extractor upload, deploy, real-time, and version-management endpoints — available on dedicated Enterprise deployments
**Dedicated infrastructure only.** These endpoints are served by dedicated Enterprise deployments — they are **not** exposed on the shared public API at `api.mixpeek.com`, where they return `404`/`405`. See [Custom Extractors → Availability](/docs/processing/custom-extractors#availability). [Contact your account team](https://mixpeek.com/contact) to provision a dedicated deployment.
Because this surface isn't in the shared OpenAPI, it isn't part of the auto-generated API Reference. This page is the hand-maintained contract; the reference client is `server/scripts/api/plugins.py`.
**Naming.** On this surface, custom extractors are addressed as **plugins**: the path segment is `/plugins`, and a deployed extractor's `plugin_id` is `{name}_{version}` with dots replaced by underscores — e.g. `text_embed@1.0.0` → `text_embed_1_0_0`. All endpoints are namespace-scoped and take `Authorization: Bearer $MIXPEEK_API_KEY`.
## Lifecycle: upload → deploy → search
The CLI (`python server/scripts/api/plugins.py push` / `deploy`) wraps these calls. The raw HTTP flow:
```bash theme={null}
NS=ns_... # your namespace id
API=https:///v1
AUTH=(-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "Content-Type: application/json")
# 1. Request a presigned upload URL (auto-bumps the patch version if you omit it)
UP=$(curl -s -X POST "$API/namespaces/$NS/plugins/uploads" "${AUTH[@]}" \
-d '{"name":"text_embed","version":"1.0.0","file_size_bytes":5000}')
UPLOAD_ID=$(echo "$UP" | jq -r '.upload_id')
PRESIGNED=$(echo "$UP" | jq -r '.presigned_url')
# 2. PUT the zipped extractor to S3
curl -s -X PUT "$PRESIGNED" -H "Content-Type: application/zip" --data-binary @text_embed.zip
# 3. Confirm — runs the security scan + manifest validation
curl -s -X POST "$API/namespaces/$NS/plugins/uploads/$UPLOAD_ID/confirm" "${AUTH[@]}" \
-d '{"message":"initial release"}'
# → { "success": true, "plugin_id": "text_embed_1_0_0", "validation_errors": [] }
# 4. Deploy (omit deployment_type for full deploy incl. realtime; batch_only skips the HTTP endpoint)
curl -s -X POST "$API/namespaces/$NS/plugins/text_embed_1_0_0/deploy" "${AUTH[@]}"
# 5. Poll deployment status until DEPLOYED
curl -s "$API/namespaces/$NS/plugins/text_embed_1_0_0/status" "${AUTH[@]}"
```
Once `DEPLOYED`, reference the extractor's `feature_uri` in a collection's `feature_extractor` and in retriever `feature_search` stages exactly as you would a built-in — see the [end-to-end walkthrough](/docs/processing/custom-extractors#end-to-end-extractor-collection-retriever).
| Deployment status | Meaning |
| -------------------- | ------------------------------------------- |
| `QUEUED` / `PENDING` | Waiting in / triggered the deployment queue |
| `IN_PROGRESS` | Blue-green rollout in progress |
| `DEPLOYED` | Ready for real-time inference |
| `NOT_DEPLOYED` | Batch-only mode |
| `FAILED` | Check the `error` field |
## Endpoint reference
All paths are prefixed with `/v1/namespaces/{namespace_id}`.
### Upload & deploy
| Method | Path | Purpose |
| ------ | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `POST` | `/plugins/uploads` | Request a presigned upload URL. Body: `{name, version?, file_size_bytes}` → `{upload_id, presigned_url, version}` |
| `POST` | `/plugins/uploads/{upload_id}/confirm` | Confirm the S3 upload; runs security scan + manifest validation. Body: `{message?}` → `{success, plugin_id, validation_errors}` |
| `POST` | `/plugins` | Direct multipart upload (alternative to the presigned flow) |
| `POST` | `/plugins/{plugin_id}/deploy` | Deploy or redeploy. Query: `deployment_type=batch_only` to skip the realtime endpoint |
| `POST` | `/plugins/{plugin_id}/undeploy` | Tear down the deployment (keeps the version) |
| `GET` | `/plugins/{plugin_id}/status` | Deployment status |
| `POST` | `/plugins/{plugin_id}/realtime/test` | Invoke the deployed `realtime.py` endpoint with a test payload |
### Discover & inspect
| Method | Path | Purpose |
| ------ | ------------------------------- | -------------------------------------------------------- |
| `GET` | `/plugins` | List custom plugins in the namespace |
| `GET` | `/plugins/{plugin_id}` | Plugin details (schemas, deployment + validation status) |
| `GET` | `/plugins/{plugin_id}/source` | Download source files of the active version |
| `GET` | `/plugins/{plugin_id}/download` | Presigned archive download URL |
| `GET` | `/plugins/available` | Org-level plugins available to enable in this namespace |
### Version management
| Method | Path | Purpose |
| ------ | ------------------------------------------------ | ------------------------------------------------- |
| `GET` | `/plugins/by-name/{slug}/versions` | Version history with deploy timestamps + messages |
| `POST` | `/plugins/by-name/{slug}/rollback` | Restore a previous version as active |
| `GET` | `/plugins/by-name/{slug}/diff?v1=1.0.0&v2=1.0.1` | Diff source files between two versions |
### Org plugins & namespace config
| Method | Path | Purpose |
| -------- | ---------------------------------- | --------------------------------------------------------------------- |
| `POST` | `/plugins/org/{plugin_id}/enable` | Enable an org-level plugin in this namespace |
| `POST` | `/plugins/org/{plugin_id}/disable` | Disable an org-level plugin in this namespace |
| `POST` | `/plugins/reconfigure` | Reconfigure the namespace vector-store schema for a plugin's features |
| `DELETE` | `/plugins/{plugin_id}` | Delete a plugin version |
## CLI equivalents
`server/scripts/api/plugins.py` is the maintained client for this surface (reads `MIXPEEK_API_KEY`, `MIXPEEK_NAMESPACE`, `MIXPEEK_API_URL`):
| CLI | Endpoint(s) |
| --------------- | ---------------------------------------------------------------------- |
| `push` | `POST /plugins/uploads` → `PUT` → `POST /plugins/uploads/{id}/confirm` |
| `pull` | `GET /plugins/by-name/{slug}/versions` → `GET /plugins/{id}/source` |
| `status` | `GET /plugins/by-name/{slug}/versions` |
| `log` | `GET /plugins/by-name/{slug}/versions` |
| `rollback` | `POST /plugins/by-name/{slug}/rollback` |
| `diff` | `GET /plugins/by-name/{slug}/diff` |
| `lint` / `test` | offline — no API call (work on any plan) |
See [Custom Extractors](/docs/processing/custom-extractors) for the full authoring guide and [Local Development](/docs/processing/custom-extractors#local-development) for `lint`/`test`.
# Custom Extractors
Source: https://docs.mixpeek.com/docs/processing/custom-extractors
Build, test, and deploy your own feature extractors and real-time inference endpoints on Mixpeek infrastructure
Create a managed namespace to get your API key, then package, deploy, and query your own extractor code.
Runnable reference for every built-in Mixpeek extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry, so it always matches production.
![Custom extractor flow: a single extractor archive powers both the ingestion pipeline (batch Ray Data) and the retrieval realtime endpoint (Ray Serve HTTP), used by retriever stages like feature_search, rerank, and agentic_enrich]()
Custom extractors let you run your own code on Mixpeek infrastructure — inside the same Ray cluster that powers the built-in extractors and retriever stages. You keep full control of the logic, model, and I/O; Mixpeek handles packaging, scheduling, GPU allocation, caching, and observability.
## Availability
**The upload → deploy → real-time lifecycle runs on dedicated Enterprise infrastructure — it is not exposed on the shared public API at `api.mixpeek.com`.** What works where:
| Capability | Shared API (`api.mixpeek.com`) | Dedicated deployment |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------ | -------------------- |
| Discover extractors (`GET /v1/namespaces/{ns}/extractors`, `…/{id}`) | ✅ | ✅ |
| Author + `lint` + `test` locally (no API key) | ✅ | ✅ |
| Contribute via the [Submission workflow](/docs/processing/extractor-marketplace) (reviewed → merged as built-in) | ✅ | ✅ |
| Upload / deploy / undeploy / real-time inference | ❌ | ✅ |
| Version management (`push` / `pull` / `rollback` / `diff`) | ❌ | ✅ |
On the shared API the upload/deploy/realtime endpoints return `404`/`405` by design. [Contact your account team](https://mixpeek.com/contact) to provision a dedicated deployment for self-service custom-extractor uploads, or use Submissions to ship an extractor into the built-in catalog. The dedicated upload/deploy/realtime HTTP contract is documented in the [Custom Extractor API (Dedicated Infrastructure)](/docs/processing/custom-extractor-api) reference.
Want your extractor available to everyone without dedicated infra? Submit it for review to be merged into the built-in catalog — see [Extractor Submissions](/docs/processing/extractor-marketplace).
## What You Can Build
Custom extractors plug into two places in the warehouse:
### 1. Feature Extractors (Decomposition)
Attach custom logic to a collection so every ingested object flows through your pipeline during decomposition. Your extractor is **your vocabulary** — unlike built-in pipelines (which are selected via [features](/docs/processing/features)), custom extractors keep their explicit names and are selected as a custom feature:
```json theme={null}
{
"collection_name": "filings",
"source": { "type": "bucket", "bucket_ids": ["bkt_123"] },
"features": ["custom:my_extractor"]
}
```
The explicit `feature_extractor` config (shown later on this page) also works — use it when you need custom `input_mappings` or parameters. Use custom extractors to:
* Embed domain-specific content with your own model (fine-tuned CLIP, proprietary audio encoder, etc.)
* Extract structured attributes via a VLM you manage (brand compliance, regulated content classification)
* Transcribe, OCR, or segment media with a custom pre/post-processing chain
* Produce multiple named vector indexes from a single pass
Outputs land in MVS and MongoDB with the same feature URI scheme as built-in pipelines (`mixpeek://my_extractor@1.0.0/my_embedding`), so retrievers, taxonomies, and clusters can reference them.
**Pricing**: a `custom:` feature derives its per-unit rate from the compute profile your extractor declares — the same machinery that prices native features. Quote it before running via `POST /v1/organizations/billing/estimate`; see [Billing & Pricing](/docs/platform/billing).
### 2. Retriever Operations (Query Time)
An extractor's `realtime.py` exposes a Ray Serve HTTP endpoint that retriever stages can call during execution. Use this to power:
* **`feature_search`** — embed queries at search time with the same model you used during ingestion, so the query vector lives in the same space as the indexed vectors
* **Inference operations** — on-the-fly classification, scoring, or re-ranking against your model
* **LLM calls** — wrap a hosted or private LLM behind a stable contract, with platform-managed secrets and cost tracking
* **Classifiers** — apply your own classifier to candidate results mid-pipeline
This is what lets a single custom extractor own both halves of a retrieval flow: it encodes documents on the way in, and encodes the query on the way out.
## Delivery Formats
Ship your extractor in either of two formats:
| Format | When to Use |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Zip archive** (`.zip`) | Pure-Python extractors. Mixpeek resolves dependencies against the managed runtime. Scanned by the security linter before deploy. Limit: 500 MB / 1,000 files. |
| **Container image** (OCI) | Extractors that need system packages, custom CUDA builds, compiled binaries, or non-Python runtimes. Base your image on the Mixpeek engine image, push to your org-scoped Artifact Registry repo, and set `container_image` in `manifest.py`. See [BYO Container Image](#byo-container-image). |
Both formats expose the same runtime APIs (batch `__call__`, real-time `run_inference`, platform LLM/secret accessors).
***
## Extractor Structure
Every extractor has the same layout:
```
my_extractor/
├── manifest.py # Schemas, metadata, vector indexes
├── pipeline.py # Batch processing pipeline
├── realtime.py # Real-time HTTP endpoint (optional)
└── processors/
└── core.py # Your processing logic
```
* **`manifest.py`** declares what your extractor accepts, produces, and which vector indexes to create
* **`pipeline.py`** wires your processor into the Ray Data batch pipeline
* **`realtime.py`** exposes a Ray Serve endpoint for query-time inference (e.g., embedding queries for `feature_search`)
* **`processors/`** contains your actual logic — model loading, embedding, classification, etc.
***
## Manifest
The manifest is your extractor's contract with the platform.
```python theme={null}
# manifest.py
from pydantic import BaseModel, Field
from typing import List
class MyInput(BaseModel):
text: str = Field(..., description="Input text to process")
class MyOutput(BaseModel):
embedding: List[float] = Field(..., description="384-dim embedding vector")
class MyParams(BaseModel):
model_size: str = Field(default="base", description="base or large")
metadata = {
"feature_extractor_name": "my_extractor",
"version": "1.0.0",
"description": "Custom text embedding extractor",
"category": "text",
"inference_type": "embedding", # declares real-time inference capability
}
input_schema = MyInput
output_schema = MyOutput
parameter_schema = MyParams
supported_input_types = ["text"]
features = [
{
"feature_type": "embedding",
"feature_name": "my_embedding",
"embedding_dim": 384,
"distance_metric": "cosine",
},
]
```
### Vector Index Keys
Use the **exact key names** below. Wrong keys silently create a collection with no vector indexes — your batch will show `COMPLETED` but produce 0 documents.
| Key | Required | Description |
| ----------------- | -------- | ---------------------------------- |
| `feature_type` | Yes | Must be `"embedding"` |
| `feature_name` | Yes | Name of the vector index |
| `embedding_dim` | Yes | Vector dimensionality |
| `distance_metric` | Yes | `"cosine"`, `"euclid"`, or `"dot"` |
Multiple vectors are supported — add one entry per embedding your extractor produces.
### Inference Type
Declare what kind of real-time inference your extractor provides by setting `inference_type` in `metadata`. This lets retriever stages validate that an extractor is compatible with the stage slot.
| Value | Contract | Compatible Stages |
| ----------- | ---------------------------------------------------------------- | ------------------------------ |
| `embedding` | Returns `{vector: [float]}` | `feature_search` |
| `rerank` | Accepts `{pairs: [[q, d]]}`, returns `{scores: [float]}` | `rerank` |
| `classify` | Accepts `{text: str}`, returns `{labels: [{label, confidence}]}` | `classify` |
| `generate` | Accepts `{prompt: str}`, returns `{text: str}` | `llm_filter`, `llm_enrich` |
| `general` | No specific contract | Raw `/inference` endpoint only |
If `inference_type` is omitted, the extractor can only be called via the raw inference endpoint.
### Compute Profile
Control resource allocation by adding `compute_profile` to your manifest:
```python theme={null}
compute_profile = {
"resource_type": "cpu", # "cpu", "gpu", or "api"
"batch_size": 32, # Rows per __call__ (default: 64)
"max_concurrency": 4, # Parallel Ray actors (default: 2)
}
```
For API-based or hash-based extractors that don't need GPU, set `resource_type: "cpu"` to skip GPU allocation — saves \~3 minutes startup time and costs \~6x less.
### BYO Container Image
If your extractor needs native binaries or system packages, specify a custom container image:
```python theme={null}
# manifest.py
container_image = "us-east1-docker.pkg.dev/mixpeek-inference-463103/extractors-/my-image:1.0.0"
```
Base your image on the Mixpeek engine image to get Ray, FFmpeg, and all SDK helpers:
```dockerfile theme={null}
FROM us-east1-docker.pkg.dev/mixpeek-inference-463103/mixpeek-engine/engine-base:latest
RUN apt-get update && apt-get install -y libcustom-sdk ...
```
Images must be pushed to your org-scoped Artifact Registry repo. GKE Workload Identity handles pull auth. [Contact your account team](https://mixpeek.com/contact) to provision access.
***
## Batch Processor
Your processor receives a pandas DataFrame and returns it with new columns added.
```python theme={null}
# processors/core.py
import pandas as pd
class MyProcessor:
def __init__(self, config, **kwargs):
self.config = config
self._model = None
def _ensure_model_loaded(self):
if self._model is None:
from sentence_transformers import SentenceTransformer
self._model = SentenceTransformer("all-MiniLM-L6-v2")
def __call__(self, batch: pd.DataFrame) -> pd.DataFrame:
self._ensure_model_loaded()
texts = batch["data"].fillna("").tolist()
embeddings = self._model.encode(texts).tolist()
batch["my_embedding"] = embeddings
return batch
```
### DataFrame Columns
Your `__call__` receives these columns:
| Column | Description |
| --------------- | ---------------------------------------------------------------------------- |
| `data` | For text blobs: the raw string. For binary blobs: an S3 URL (not raw bytes). |
| `document_id` | Unique document ID |
| `object_id` | Source object in the bucket |
| `blob_type` | `"image"`, `"video"`, `"audio"`, `"text"` |
| `blob_property` | Property name from your bucket schema |
| `mime_type` | MIME type (e.g. `image/jpeg`) |
Always read from `batch["data"]` — not a column named after your blob property. If your bucket has a `text` property, the content is still in `batch["data"]`. (The local `test` harness feeds the same `data` column, so an extractor that passes `test` behaves the same in production.)
### Batched Processing
Process all rows together — never call a model or API inside a per-row loop:
```python theme={null}
# WRONG — one GPU call per row
for idx, row in batch.iterrows():
embedding = self._model.encode(row["data"])
batch.at[idx, "embedding"] = embedding
# RIGHT — single batched call
texts = batch["data"].fillna("").tolist()
batch["embedding"] = self._model.encode(texts).tolist()
```
### Loading Assets from S3
For binary blobs (images, video, audio), the `data` column contains S3 URLs. Use the Extractor SDK to download them:
```python theme={null}
from shared.extractors import open_asset
from shared.extractors.sdk import parallel_io, download_asset
class ImageProcessor:
def __call__(self, batch: pd.DataFrame) -> pd.DataFrame:
def load_image(url):
from PIL import Image
path, is_temp = download_asset(url, suffix=".jpg")
img = Image.open(path).convert("RGB")
if is_temp:
import os; os.unlink(path)
return img
images = parallel_io(batch["data"].tolist(), load_image, max_workers=8)
# batch-process all images on GPU...
return batch
```
***
## Pipeline
Wire your processor into the Ray Data pipeline:
```python theme={null}
# pipeline.py
from engine.extractors.my_extractor.pipeline import (
PipelineDefinition, ResourceType, StepDefinition, build_pipeline_steps
)
from .manifest import MyParams, metadata
from .processors.core import MyProcessor
def build_steps(extractor_request, container=None, **kwargs):
params = MyParams(**(extractor_request.extractor_config.parameters or {}))
pipeline = PipelineDefinition(
name=metadata["feature_extractor_name"],
version=metadata["version"],
steps=[
StepDefinition(
service_class=MyProcessor,
resource_type=ResourceType.CPU,
config={"model_size": params.model_size},
),
]
)
return {"steps": build_pipeline_steps(pipeline)}
```
### Row Conditions
Filter which rows a step processes:
| Condition | Matches |
| ----------------------- | ------------------- |
| `RowCondition.IS_TEXT` | text/\* MIME types |
| `RowCondition.IS_IMAGE` | image/\* MIME types |
| `RowCondition.IS_VIDEO` | video/\* MIME types |
| `RowCondition.IS_AUDIO` | audio/\* MIME types |
| `RowCondition.IS_PDF` | application/pdf |
| `RowCondition.ALWAYS` | All rows (default) |
### Using Built-in Models
Compose existing Mixpeek services instead of loading models yourself:
```python theme={null}
from shared.inference.registry import get_batch_service
WhisperBatch = get_batch_service("openai/whisper-large-v3-turbo")
E5Batch = get_batch_service("intfloat/multilingual-e5-large-instruct")
# Use in your pipeline steps
StepDefinition(service_class=E5Batch, resource_type=ResourceType.CPU, config=e5_config)
```
| Service | Type | Dimensions |
| ----------------------------------------- | ------------- | ---------- |
| `intfloat/multilingual-e5-large-instruct` | Embedding | 1024 |
| `google/siglip-base-patch16-224` | Embedding | 512 |
| `jinaai/jina-embeddings-v2-base-code` | Embedding | 768 |
| `BAAI/bge-reranker-v2-m3` | Reranker | — |
| `openai/whisper-large-v3-turbo` | Transcription | — |
For HuggingFace and your own fine-tuned weights, see the [Model Registry](/docs/processing/model-registry).
***
## Real-time Endpoint
Add `realtime.py` to expose an HTTP endpoint for query-time inference. This is what lets [retriever `feature_search` stages](/docs/retrieval/stages/feature-search) embed queries with your model.
```python theme={null}
# realtime.py
from shared.extractors.inference.serve import BaseInferenceService
class InferenceService(BaseInferenceService):
def __init__(self):
super().__init__()
self._model = None
async def run_inference(self, inputs: dict, parameters: dict) -> dict:
if self._model is None:
from sentence_transformers import SentenceTransformer
self._model = SentenceTransformer("all-MiniLM-L6-v2")
text = inputs.get("text", "")
embedding = self._model.encode([text])[0].tolist()
return {"embedding": embedding}
```
The return dict must include an `"embedding"` key — this is what `feature_search` uses as the query vector. For multi-vector extractors, include additional keys matching your `feature_name` values.
`realtime.py` handles embedding only, not retrieval. If your pipeline needs retrieval context (comparing against stored references), configure [retriever stages](/docs/retrieval/stages/overview) to handle that logic. The real-time endpoint serves on dedicated infrastructure (see [Availability](#availability)).
***
## Platform Services
### LLM Access
Use `container.llm` to call platform-managed LLMs with built-in cost tracking and caching:
```python theme={null}
# pipeline.py — pass to your processor
config = {"llm_service": container.llm}
# processors/core.py — call concurrently
from shared.extractors.sdk import concurrent_api_calls
async def analyze(text):
return await self._llm.generate(
instruction="Extract entities from this text",
text=text,
provider="google",
model="gemini-2.5-flash",
schema={"type": "object", "properties": {"entities": {"type": "array"}}}
)
results = concurrent_api_calls(texts, analyze, max_concurrent=10)
```
| Provider | Models |
| ----------- | ------------------------------------ |
| `google` | `gemini-2.5-flash`, `gemini-2.5-pro` |
| `openai` | `gpt-4o`, `gpt-4o-mini` |
| `anthropic` | `claude-sonnet-4-20250514` |
### Secrets
Access encrypted org secrets at runtime via `container.secrets`:
```python theme={null}
api_key = await container.secrets.get("EXTERNAL_API_KEY")
```
For platform LLMs, use `container.llm` instead — it handles API keys automatically.
### CLI Tools
Custom extractors can't import `subprocess` directly. Use `run_tool` for whitelisted CLI tools:
```python theme={null}
from shared.extractors.sdk import run_tool
result = run_tool("ffmpeg", ["-y", "-i", input_path, "-c:v", "libx264", output_path], timeout=600)
```
Available tools: `ffmpeg`, `ffprobe`, `convert`, `identify`, `magick`, `exiftool`, `mediainfo`, `sox`, `soxi`, `REDline`, `art-cmd`
### Pre-installed Tools
The engine runtime includes these media tools, available via `run_tool`:
| Tool | Format | Description |
| ---------------------- | -------------------- | ---------------------------------------------------- |
| `ffmpeg` / `ffprobe` | Standard video/audio | Transcode, extract frames, probe metadata |
| `REDline` | RED R3D | Decode RED cinema camera raw files to ProRes/DPX/EXR |
| `art-cmd` | ARRI RAW | Decode ARRI raw (.ari/.arriraw/.arx) to ProRes |
| `exiftool` | All media | Read/write EXIF and XMP metadata |
| `mediainfo` | All media | Detailed format and codec inspection |
| `convert` / `identify` | Images | ImageMagick image processing |
| `sox` / `soxi` | Audio | Audio processing and info |
**RED R3D** and **ARRI RAW** decode examples:
```python theme={null}
run_tool("REDline", ["--i", input_path, "--o", output_path, "--format", "201", "--resize", "2"], timeout=600)
run_tool("art-cmd", ["--input", input_path, "--output", output_dir, "--format", "prores"], timeout=600)
# Chain with FFmpeg for web playback:
run_tool("ffmpeg", ["-y", "-i", prores_path, "-c:v", "libx264", "-crf", "23", output_mp4], timeout=600)
```
***
## Typed SDK
The Extractor SDK provides typed base classes that replace bare Python variables with validated, IDE-friendly types — autocompletion, validation at upload time, and output column checking in the test harness. Import from `shared.extractors.sdk` or `shared.extractors`.
```python theme={null}
# manifest.py
from shared.extractors.sdk import ExtractorManifest, Feature
manifest = ExtractorManifest(
feature_extractor_name="my_embedder",
version="v1",
description="Custom text embedder",
dependencies=["sentence-transformers>=2.2"],
system_packages=["libmagic1"], # apt packages installed in the container
features=[Feature.embedding("my_embedder_v1", dim=384)],
inference_type="embedding",
)
# Backward-compat: AST parser reads these module-level variables
feature_extractor_name = manifest.feature_extractor_name
version = manifest.version
description = manifest.description
dependencies = manifest.dependencies
features = manifest.features_as_dicts()
feature_uri = manifest.feature_uri
```
```python theme={null}
# pipeline.py
from shared.extractors.sdk import BatchProcessor, prefetch_hf_model
class MyProcessor(BatchProcessor):
def setup(self):
prefetch_hf_model("sentence-transformers/all-MiniLM-L6-v2")
from sentence_transformers import SentenceTransformer
self.model = SentenceTransformer("all-MiniLM-L6-v2")
def process(self, batch):
texts = batch["data"].fillna("").tolist()
batch["my_embedder_v1_embedding"] = self.model.encode(texts).tolist()
return batch
```
`setup()` runs once on first batch (lazy model loading); `process()` runs on each batch. `prefetch_hf_model()` pre-downloads the model to the HF cache to reduce cold start.
```python theme={null}
# realtime.py
from shared.extractors.sdk import InferenceService
class MyInference(InferenceService):
def setup(self):
from sentence_transformers import SentenceTransformer
self.model = SentenceTransformer("all-MiniLM-L6-v2")
async def infer(self, inputs, parameters):
text = inputs.get("text", "")
return {"embedding": self.model.encode([text])[0].tolist()}
```
### SDK Reference
| Function / Class | Purpose |
| ------------------------------------------------------------ | ------------------------------------------------------ |
| `ExtractorManifest` | Typed manifest with validation |
| `Feature.embedding(name, dim)` | Create an embedding feature definition |
| `Feature.classification(name, labels)` | Create a classification feature definition |
| `BatchProcessor` | Base class with `setup()` / `process()` lifecycle |
| `InferenceService` | Base class with `setup()` / `infer()` lifecycle |
| `prefetch_hf_model(model_id)` | Pre-download HF model to cache (cold start mitigation) |
| `parallel_io(items, fn, max_workers)` | Parallel file downloads and I/O |
| `concurrent_api_calls(items, async_fn, max_concurrent)` | Concurrent LLM/API calls |
| `open_asset(url, suffix)` | Context manager for S3 downloads |
| `download_asset(url, suffix)` | Manual S3 download with cleanup flag |
| `run_tool(tool, args, timeout)` | Execute whitelisted CLI tools |
| `upload_asset(path, namespace_id, internal_id, resource_id)` | Upload processed files back to S3 |
***
## Security Rules
Custom extractors are scanned before deployment. Code violating these rules is rejected.
* **Allowed:** `numpy`, `pandas`, `torch`, `transformers`, `sentence_transformers`, `onnxruntime`, `PIL`, `cv2`, `requests`, `httpx`, `os` (safe functions only), `json`, `re`, `pydantic`, `logging`, `getattr`, `hasattr`
* **Blocked:** `subprocess`, `os.system`, `os.popen`, `os.exec*`, `eval`, `exec`, `ctypes`, `socket`, `multiprocessing`, `open`, `setattr`, `delattr`
`import os` is allowed — only dangerous functions are blocked. Library-internal file I/O (`torch.load`, `transformers.from_pretrained`, `pd.read_csv`) is fine since the scanner only inspects your extractor's source code.
***
## Local Development
Validate and test your extractor locally with the CLI before uploading. **No API key needed** — these run fully offline and work on any plan.
```bash theme={null}
# Validate manifest and run the security scanner
python server/scripts/api/plugins.py lint path/to/my_extractor
# Run the pipeline through the Ray Data test harness (real map_batches + Arrow)
python server/scripts/api/plugins.py test path/to/my_extractor
```
`lint` catches common mistakes before upload:
* Wrong feature key names (`name` instead of `feature_name`)
* Missing required fields
* Security scanner violations
`test` runs your processor through real Ray Data `map_batches` with Arrow serialization — the same path used in production. Sample rows are fed in the `data` column (matching production), and the harness validates that your output columns match the manifest features.
### Version Management
On a [dedicated deployment](#availability), the same CLI manages deployed versions with a git-like workflow:
| Command | Description |
| ---------- | ---------------------------------------------------------------------------- |
| `pull` | Download the active version's source files to a local directory |
| `push` | Zip, upload, and confirm a new version (auto-bumps patch version if omitted) |
| `log` | Show version history with deploy timestamps and commit messages |
| `status` | Show active version, extractor ID, and deployment status |
| `rollback` | Restore a previous version as active |
| `diff` | Compare source files between two versions |
The CLI reads `MIXPEEK_API_KEY`, `MIXPEEK_NAMESPACE`, and `MIXPEEK_API_URL` from the environment (or `--api-key` / `--namespace`).
### Archive Limits
| Limit | Value |
| ----------- | ------ |
| Upload size | 500 MB |
| Max files | 1,000 |
Don't bundle model weights — download from HuggingFace Hub at init time, or use the [Model Registry](/docs/processing/model-registry) for custom weights.
***
## End-to-End: Extractor → Collection → Retriever
This walkthrough connects all the pieces. Set `MIXPEEK_API_KEY`, `MIXPEEK_NAMESPACE`, and `MIXPEEK_API_URL` first — the same variables the `plugins.py` CLI reads.
Steps 1 and 5 (the extractor upload/deploy) require a [dedicated deployment](#availability). On the shared API, use a built-in extractor (e.g. `text_extractor`) for the `feature_extractor` in step 3 and skip steps 1 and 5.
### 1. Deploy the Extractor (dedicated infra)
The simplest path is the CLI — `python server/scripts/api/plugins.py push` then `deploy`. The equivalent raw HTTP (custom extractors are addressed as `/plugins` on this surface; see the [Custom Extractor API](/docs/processing/custom-extractor-api) reference):
```bash theme={null}
zip -r my_extractor.zip my_extractor/
# Presigned upload → confirm → deploy
UPLOAD=$(curl -s -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/plugins/uploads" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "Content-Type: application/json" \
-d '{"name":"my_extractor","version":"1.0.0","file_size_bytes":5000}')
UPLOAD_ID=$(echo "$UPLOAD" | jq -r '.upload_id')
PRESIGNED_URL=$(echo "$UPLOAD" | jq -r '.presigned_url')
curl -s -X PUT "$PRESIGNED_URL" -H "Content-Type: application/zip" --data-binary @my_extractor.zip
curl -s -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/plugins/uploads/$UPLOAD_ID/confirm" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "Content-Type: application/json" -d '{}'
curl -s -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/plugins/my_extractor_1_0_0/deploy" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
### 2. Create a Bucket and Upload Data
Buckets, collections, retrievers, and batches are **top-level** resources addressed by the **`X-Namespace` header** — not nested under `/namespaces/{ns}/`. (Only `extractors` and `models` are path-scoped.)
```bash theme={null}
BUCKET=$(curl -s -X POST "https://api.mixpeek.com/v1/buckets" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{"bucket_name":"sec-filings","bucket_schema":{"properties":{"filing_text":{"type":"text"}}}}')
BUCKET_ID=$(echo "$BUCKET" | jq -r '.bucket_id')
curl -s -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/objects" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{"blobs":[{"property":"filing_text","type":"text","data":"Revenue increased 22% year-over-year..."}],"metadata":{"ticker":"AAPL","form":"10-K"}}'
```
### 3. Create a Collection with the Extractor
Every object uploaded to the bucket flows through this extractor's batch processor.
```bash theme={null}
COLLECTION=$(curl -s -X POST "https://api.mixpeek.com/v1/collections" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d "{
\"collection_name\":\"sec-filing-chunks\",
\"source\":{\"type\":\"bucket\",\"bucket_ids\":[\"$BUCKET_ID\"]},
\"feature_extractor\":{\"feature_extractor_name\":\"my_extractor\",\"version\":\"1.0.0\",\"input_mappings\":{\"text\":\"filing_text\"}}
}")
COLLECTION_ID=$(echo "$COLLECTION" | jq -r '.collection_id')
```
### 4. Process the Data (two-step batch)
A batch is **created** (with the objects + target collections) and then **submitted**:
```bash theme={null}
# List object IDs, then create the batch
OBJIDS=$(curl -s "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/objects" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
| jq -c '[.results[].object_id]')
BATCH=$(curl -s -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/batches" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d "{\"batch_name\":\"run-1\",\"object_ids\":$OBJIDS,\"collection_ids\":[\"$COLLECTION_ID\"]}")
BATCH_ID=$(echo "$BATCH" | jq -r '.batch_id')
# collection_ids are fixed at batch creation (above) — submit takes no scope
curl -s -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/batches/$BATCH_ID/submit" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d "{}"
# Poll GET /v1/buckets/$BUCKET_ID/batches/$BATCH_ID until status is COMPLETED
```
### 5. Build a Retriever and Search
At query time the retriever calls your extractor's `realtime.py` (dedicated infra) to embed the query, then searches the vectors your batch processor produced.
```bash theme={null}
RETRIEVER=$(curl -s -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d "{
\"retriever_name\":\"filing-search\",
\"collection_identifiers\":[\"$COLLECTION_ID\"],
\"input_schema\":{\"query\":{\"type\":\"text\",\"required\":true}},
\"stages\":[{\"stage_name\":\"semantic\",\"stage_type\":\"filter\",\"config\":{\"stage_id\":\"feature_search\",\"parameters\":{
\"searches\":[{\"feature_uri\":\"mixpeek://my_extractor@1.0.0/my_embedding\",\"query\":{\"input_mode\":\"text\",\"value\":\"{{INPUT.query}}\"},\"top_k\":20}],
\"final_top_k\":20,\"fusion\":\"rrf\"}}}]
}")
RETRIEVER_ID=$(echo "$RETRIEVER" | jq -r '.retriever_id')
curl -s -X POST "https://api.mixpeek.com/v1/retrievers/$RETRIEVER_ID/execute" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{"inputs":{"query":"revenue growth year over year"}}'
```
***
## Troubleshooting
The upload/deploy/realtime lifecycle is only available on a [dedicated deployment](#availability) — the shared `api.mixpeek.com` exposes only `GET` list/details for extractors. Develop + `lint` + `test` locally, then either provision a dedicated deployment or ship via [Submissions](/docs/processing/extractor-marketplace).
Usually wrong `features` key names in `manifest.py`. Run `python server/scripts/api/plugins.py lint path/to/my_extractor` to validate. Check that your collection has non-empty `vector_indexes` via `GET /v1/collections/{id}`. See [Vector Index Keys](#vector-index-keys).
Most common cause: reading from the wrong column. Always use `batch["data"]`, not `batch["text"]` or other property names. Check Ray logs for `[FailureAggregator]` entries.
Check `validation_errors`. Common issues: using `subprocess` (use `run_tool`), using `open()` directly (use library I/O), using `eval`/`exec` (use `json.loads`).
Use `prefetch_hf_model()` in your `setup()` method to pre-download models to the HF cache. On GKE, `HF_HOME` points to a shared PVC so models persist across pod restarts. First cold start downloads (\~1-2 min); subsequent starts use cache.
On a cold engine, the embedding model loads on demand — the first batch can take **several minutes** to leave `PROCESSING`, and the **first retriever `execute` afterward may return 0 results** (`status` `completed` or `degraded`) while the **query-side** model warms. This is a cold-start artifact, not a real no-match: **retry after a few seconds** and results appear. A warm namespace responds immediately. Keep a namespace warm by issuing a periodic lightweight query, or contact your account team about a warm replica floor for latency-sensitive workloads.
If your manifest includes a `feature_uri`, the system expects a corresponding `realtime.py`. Without it, `feature_search` queries against that URI will fail. Omit both if you only need batch processing.
## Next Steps
Build, test, and query a minimal text embedding extractor end-to-end.
Load HuggingFace models or your own fine-tuned weights inside an extractor.
Submit your extractor for review to be merged into the built-in catalog.
Chain collections into a DAG — transcribe, then embed, then classify.
Run a new extractor over an already-ingested corpus — scoped, cost-safe, priced before you run.
# Decomposition
Source: https://docs.mixpeek.com/docs/processing/decomposition
How Mixpeek turns raw objects into searchable documents and queryable features
![Decomposition: raw objects are split into documents, each with extracted features like embeddings, transcripts, and metadata]()
Decomposition is the core transformation in Mixpeek. A raw file (video, PDF, image, audio) goes in as one **object**. It comes out as many **documents**, each with its own **features**. This is what makes sub-file search possible — you search *within* a video at the segment level, not *for* the video as a whole.
## Three Primitives
| Primitive | What It Is | Role |
| ------------ | -------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| **Object** | Raw file or record in a bucket (video, PDF, JSON row, image). | The input boundary. You upload objects. |
| **Document** | One row of output in a collection, produced by decomposition. | The query boundary. You search documents. |
| **Feature** | A named output attached to a document (embedding, transcript, OCR text, label, score). | The composition boundary. Retrievers reference features by URI. |
The pipeline is always:
```
Object (bucket) → Decomposition → Document (collection) → Features (MVS + MongoDB)
```
## What Decomposition Decides
The feature extractor controls *how* an object is decomposed into documents. The strategy depends on the content type:
| Content Type | Decomposition Strategy | Result |
| ------------------- | ------------------------------------------------- | -------------------------------------------------------------------------------- |
| **Video** | Time intervals, scene boundaries, or silence gaps | Each segment = 1 document with visual embedding + transcript + scene description |
| **Audio** | Silence boundaries or fixed intervals | Each segment = 1 document with transcript + transcript embedding |
| **PDF / Document** | Page, paragraph, or sentence boundaries | Each chunk = 1 document with text content + text embedding |
| **Image** | No split (1:1) | 1 image = 1 document with visual embedding + OCR + description |
| **Structured data** | Row-level (1:1) | 1 row = 1 document with field-level features |
## Why It Matters
**Without decomposition**, a 30-minute video is one record. Searching for "the moment the CEO mentions revenue" means scanning the entire video. There's no way to return a specific timestamp.
**With decomposition**, that video becomes \~180 ten-second segments, each with its own transcript embedding, visual embedding, and scene description. A search returns the exact segment at 14:30 where the CEO says "revenue grew 22%."
The same applies to documents: a 200-page PDF becomes 200 searchable chunks instead of one monolithic record.
## Feature URIs
Every feature produced by decomposition gets a URI that uniquely identifies it:
```
mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding
mixpeek://face_identity_extractor@v1/insightface__arcface
mixpeek://my_custom_plugin@1.0.0/domain_embedding
```
Retrievers, taxonomies, and clusters reference features by URI. This is the composition boundary — you can build a retriever that searches `multimodal_embedding` in one stage and `face_embedding` in another, even though they were produced by different extractors.
## Configuring Decomposition
Decomposition is configured via the `feature_extractor` field on a collection:
```json theme={null}
{
"collection_name": "video-library",
"source": { "type": "bucket", "bucket_ids": ["bkt_videos"] },
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"settings": {
"video_segmentation": {
"type": "time",
"interval_sec": 10
},
"run_transcription": true,
"run_scene_description": true
}
}
}
```
The `settings` object controls the decomposition strategy. Each extractor has its own settings — see the extractor-specific pages for details:
Time, scene, and silence segmentation strategies
Visual embeddings, OCR, and structured extraction
Silence-boundary segmentation and transcription
Page, paragraph, and sentence chunking
## Multi-Tier Decomposition
When a single extraction pass isn't enough — e.g., you need to transcribe audio *then* embed the transcription *then* classify each chunk — you chain collections. Each tier reads the output of the previous one, forming a DAG:
```
Tier 1: raw video → segments with transcripts
Tier 2: tier-1 docs → text chunks with embeddings
Tier 3: tier-2 docs → classifications per chunk
```
The engine resolves tiers automatically and executes them in dependency order. See [Multi-Tier Feature Extraction](/docs/processing/multi-tier-extractors) for the full guide.
## Lineage
Every document tracks its lineage back to the original object:
```json theme={null}
{
"root_object_id": "obj_video_123",
"root_bucket_id": "bkt_marketing",
"source_collection_id": "col_segments",
"lineage_path": "bkt_marketing/col_segments/col_chunks"
}
```
This lets you trace any search result back through tiers to the original file. Use the [Lineage API](/docs/api-reference/document-lineage/get-document-lineage) to visualize the decomposition tree.
# Deduplication & Re-processing
Source: https://docs.mixpeek.com/docs/processing/deduplication
Content-hash dedup decides what work is skipped, replaced, or forced when objects are processed into collections — and lineage decides what re-runs when a source changes
Every object ingested into Mixpeek is fingerprinted with a **SHA-256 content hash**, and every document produced from it carries that hash in its system envelope alongside a full [lineage record](/docs/retrieval/lineage-traversal). Together they answer the two questions every pipeline eventually asks:
1. **Have I already processed this?** — the content hash.
2. **What downstream work is stale now that this changed?** — the lineage chain.
## Content hashing
The hash is computed from the source content at ingestion and stored:
* on the **object** (in the bucket), and
* on every **document** derived from it (`content_hash` in the document's system envelope).
Two objects with identical bytes produce the same hash — regardless of filename, upload time, or path. A changed file produces a new hash, which is what marks its derived documents as stale.
"Latest version" of an asset is therefore a content-hash lookup, not a version counter you maintain yourself.
## Dedup strategy
When you submit a batch, `dedup_strategy` controls how objects that were **already processed in a prior batch** are handled. Dedup is scoped to the *(bucket, collection)* pair — an object is a duplicate if the target collection already has documents produced from that same source object.
| Strategy | Behavior | Cost profile |
| ------------------ | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| `skip` *(default)* | Objects that already have documents are not reprocessed | Free for unchanged content — no extraction, no inference |
| `replace` | Existing documents are deleted and the object is fully reprocessed | Full extraction cost; use when the source content or the extractor configuration changed |
| `force` | Process regardless, allowing duplicate documents | Full cost, duplicates allowed — rarely what you want outside testing |
```json theme={null}
{
"bucket_id": "bkt_abc123",
"collection_id": "col_xyz789",
"dedup_strategy": "skip"
}
```
`skip` compares against *prior processing state*, not against your current extractor configuration. If you have changed extractor settings (different embedding model, new transcode profile) and want existing documents rebuilt under the new configuration, use `replace` — `skip` will happily keep serving documents produced under the old configuration, and re-running a batch with `skip` costs nothing precisely because it does nothing.
## The re-processing cascade
Derived assets stay in sync with their source through the combination of both mechanisms:
1. A source object's content changes → its hash changes.
2. Re-submitting with `replace` (or re-syncing the source) reprocesses it — and every derived document downstream in its [lineage](/docs/retrieval/lineage-traversal) (transcode → embed → tag) is rebuilt from the new source.
3. Sources whose hash **didn't** change are skipped — unchanged content is never re-paid for.
This is what keeps N derived artifacts consistent with one upstream asset without a separate file system or job-tracking database: the document *is* the record, the hash *is* the version, and lineage *is* the dependency graph.
## Related
* [Lineage traversal](/docs/retrieval/lineage-traversal) — querying and walking derivation chains
* [Processing pipeline](/docs/platform/processing) — how objects become documents
* [Batch ingestion at scale](/docs/operations/batch-ingestion-at-scale) — batch mechanics, tiers, and monitoring
# Extractor Submissions
Source: https://docs.mixpeek.com/docs/processing/extractor-marketplace
Submit custom extractors for review and inclusion in the Mixpeek extractor catalog
Runnable reference for every built-in Mixpeek extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry, so it always matches production.
**Marketplace Deprecated.** The self-service marketplace (publish/install/browse via `/v1/plugins/...` and `/v1/public/plugins/...`) has been removed. The SDK methods `client.plugins.publish()`, `client.plugins.marketplace.list()`, and `client.plugins.install()` no longer exist.
Extractor sharing now uses the **submission workflow** described below.
## How It Works
Instead of a self-service marketplace, community extractors follow a submission and review process:
1. **Develop** — Build your extractor following the [Custom Extractors guide](/docs/processing/custom-extractors)
2. **Submit** — Upload your extractor archive via `POST /v1/extractors/submissions`
3. **Review** — The Mixpeek team reviews the submission for quality, security, and compatibility
4. **Merge** — Approved extractors are merged into `engine/extractors/` and become available as built-in extractors
```mermaid theme={null}
graph LR
A[Develop Extractor] --> B[Submit Archive]
B --> C[Mixpeek Reviews]
C --> D[Approved & Merged]
D --> E[Available as built-in extractor]
```
## Submitting an Extractor
Package your extractor as a zip archive and submit it for review:
```bash theme={null}
# Package your extractor
zip -r my_text_extractor.zip my_text_extractor/
# Submit for review
curl -X POST "https://api.mixpeek.com/v1/extractors/submissions" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: multipart/form-data" \
-F "archive=@my_text_extractor.zip" \
-F "display_name=My Text Extractor" \
-F "description=Advanced text extraction with custom NLP models" \
-F "category=text-processing"
```
### Submission Requirements
* Extractor must pass validation and security scan (same rules as custom extractor uploads)
* Include a complete `manifest.py` with correct `features` definitions
* Include working `pipeline.py` and optionally `realtime.py`
* Archive must be under 500 MB
## Review Process
After submission:
1. The archive is validated and scanned automatically
2. The Mixpeek team reviews the extractor code for quality and security
3. If approved, the extractor code is merged into `engine/extractors/`
4. The submitting organization is notified and the extractor becomes available to all users
## Trust Tiers
| Tier | Description |
| ----------- | ----------------------------------------------------------------- |
| `community` | Extractors submitted by community members and approved by Mixpeek |
| `verified` | Extractors with additional performance and reliability validation |
| `official` | Extractors developed and maintained by Mixpeek |
## Using Approved Extractors
Once an extractor is approved and merged, it works like any built-in extractor. Reference it by name and version in your collection configuration:
```python theme={null}
# Create a collection using the approved extractor
client.collections.create(
namespace_id="ns_abc123",
collection_name="my-collection",
source={"type": "bucket", "bucket_ids": ["bkt_xyz"]},
feature_extractor={
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {"text": "description"},
}
)
```
# Migrating from Extractor Names
Source: https://docs.mixpeek.com/docs/processing/extractor-migration
Built-in extractor names are deprecated aliases — map your existing feature_extractor configs to features keys
Mixpeek's public vocabulary is **modality + features**: you configure collections by picking [features](/docs/processing/features) (`image_search`, `faces`, `onscreen_text`, ...), and the platform resolves the implementation internally. Built-in extractor names — `image_extractor`, `universal_extractor`, `face_identity_extractor`, and friends — are **internal implementation details** and no longer appear in new API payloads, errors, or docs.
**Nothing breaks.** Existing configs and SDK clients that pass `feature_extractor: {...}` keep working: known built-in extractor names are accepted as deprecated aliases and resolved to their feature through the same catalog that powers the features path. Responses carry a deprecation notice pointing at `features: [...]`.
This page is the one place old names are documented, for migration. Custom extractors you publish yourself are **not** deprecated — your plugin is your vocabulary, selected as [`custom:`](/docs/processing/features#custom-features-bring-your-own).
## Alias map
| Deprecated extractor name | Use instead | Modality |
| ----------------------------- | ------------------------------------------------------------------------------------------ | ---------------------- |
| `image_extractor` | `features: ["image_search"]` | image |
| `image_extractor` (PDF input) | `features: ["document_search"]` | document |
| `universal_extractor` | `features: ["video_search"]` | video |
| `text_extractor` | `features: ["text_search"]` | text |
| `audio_fingerprint_extractor` | `features: ["audio_search"]` (audio) or `features: ["audio_fingerprint"]` (in-video adder) | audio / video |
| `face_identity_extractor` | `features: ["faces"]` | image, video, document |
| `scrolling_text_extractor` | `features: ["onscreen_text"]` | video |
| `document_graph_extractor` | `features: ["document_layout"]` | document |
| `web_scraper` | `features: ["web_crawl"]` | web |
| `multimodal_extractor` | `features: ["multimodal_understanding"]` | any |
| `gemini_multifile_extractor` | `features: ["multimodal_understanding"]` | any |
| `passthrough_extractor` | `features: ["storage_only"]` | any |
The live menu — including keys, display names, and rates — is always `GET /v1/collections/features` ([discovery](/docs/processing/features#discover-features)).
## Before / after
**Before** (still works, deprecated):
```json theme={null}
{
"collection_name": "press-footage",
"source": { "type": "bucket", "bucket_ids": ["bkt_123"] },
"feature_extractor": {
"feature_extractor_name": "face_identity_extractor",
"version": "v1",
"input_mappings": { "video": "video_url" }
}
}
```
**After**:
```json theme={null}
{
"collection_name": "press-footage",
"source": { "type": "bucket", "bucket_ids": ["bkt_123"] },
"features": ["faces"]
}
```
The features path also defaults `input_mappings` for the standard `uploads` bucket properties (`image`, `video`, `audio`, `pdf`, `content`, `url`) — see [default input wiring](/docs/processing/features#default-input-wiring). If you need explicit input mappings, field passthrough, or extractor parameters, the `feature_extractor` config object remains the [advanced path](/docs/processing/feature-extractors).
## What you don't need to migrate
* **Existing collections** — they keep running unchanged. Aliases have no sunset date.
* **Feature URIs** — query-side references like `mixpeek://text_extractor@v1/...` in existing retrievers remain valid contracts; they pin the pipeline version that produced your vectors.
* **Custom extractors** — your own plugins keep their explicit names.
## Why this changed
Pricing and configuration now share one vocabulary: features applied to modality units (images, video minutes, pages, tokens). When Mixpeek swaps or upgrades the models behind a feature, nothing about your config or your rates moves. See [Billing & Pricing](/docs/platform/billing) for how features are priced.
# Audio Fingerprint Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/audio-fingerprint
Audio fingerprinting with CLAP — 512-d embeddings from audio files or video audio tracks for sound-mark matching and audio similarity
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["audio_search"]` for audio files, or `features: ["audio_fingerprint"]` as an in-video add-on. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Create a managed namespace, index your audio and video files, then match them by acoustic fingerprint.
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
The audio fingerprint extractor produces **512-dimensional CLAP embeddings** (Contrastive Language-Audio Pretraining, `laion/clap-htsat-tiny`) from audio files or the audio track of a video. It segments audio into overlapping windows, embeds each segment, and L2-normalizes the vectors for cosine similarity. Use it for sound-mark matching, audio similarity, and retrieving audio by acoustic fingerprint.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/audio\_fingerprint\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/audio_fingerprint_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Resolve input** — apply `input_mappings` to get the audio or video URL.
2. **Audio extraction** — if the source is video, extract the audio track.
3. **Resample** — resample audio to `sample_rate` (48000 Hz CLAP default).
4. **Segment** — split into `segment_duration_sec` windows hopping by `segment_hop_sec` (default 50% overlap); truncate beyond `max_audio_length_sec`.
5. **CLAP embedding** — embed each segment to a 512-d vector.
6. **Normalize** (if `normalize_embeddings`) — L2-normalize to unit vectors.
7. **Output** — one document per segment with timing metadata.
## When to Use
| Use Case | Description |
| ----------------------- | ------------------------------------------------------------------ |
| **Sound-mark matching** | Detect a known jingle, sound logo, or audio cue across a corpus |
| **Audio similarity** | Find acoustically similar clips (music, ambience, effects) |
| **Ad/asset detection** | Match the audio fingerprint of an ad or asset within longer media |
| **Video audio search** | Search the audio track of video assets without separate extraction |
## When NOT to Use
| Scenario | Recommended Alternative |
| -------------------------------------- | ---------------------------------------------- |
| Speech-to-text / transcription | A transcription extractor (e.g. Whisper-based) |
| Text semantic search over spoken words | Transcribe, then `text_extractor` |
| Whole-file multimodal embedding | `universal_extractor` / `multimodal_extractor` |
| Music metadata/tagging only | A classification taxonomy over fingerprints |
## Input Schema
| Field | Type | Required | Description |
| ------- | ------ | -------- | -------------------------------------------------------------- |
| `audio` | string | one of | URL or path to an audio file. Populated from `input_mappings`. |
| `video` | string | one of | URL or path to a video file; the audio track is extracted. |
```json theme={null}
{
"audio": "s3://my-bucket/spots/jingle.wav"
}
```
Supported input types: **AUDIO, VIDEO** (max 1 each per object).
## Output Schema
One document per audio segment:
| Field | Type | Description |
| ------------------------------------------ | ------------- | ------------------------------------------------------ |
| `audio_fingerprint_extractor_v1_embedding` | float\[512] | CLAP embedding (L2-normalized when enabled) |
| `segment_index` | integer | Segment index (0-based) |
| `start_time_sec` / `end_time_sec` | float | Segment start/end time in seconds |
| `duration_sec` | float | Duration of this segment (seconds) |
| `total_duration_sec` | float \| null | Source audio duration |
| `sample_rate` | integer | Sample rate used for processing |
| `audio_source_type` | string | Source type: `audio` or `video` |
| `embedding_model` | string | Embedding model used (default `laion/clap-htsat-tiny`) |
| `processing_time_ms` | float | Per-segment processing time |
```json theme={null}
{
"audio_fingerprint_extractor_v1_embedding": [0.041, -0.018, 0.092, ...],
"segment_index": 0,
"start_time_sec": 0.0,
"end_time_sec": 5.0,
"duration_sec": 5.0,
"sample_rate": 48000,
"audio_source_type": "audio",
"embedding_model": "laion/clap-htsat-tiny"
}
```
## Parameters
| Parameter | Type | Default | Range | Description |
| ---------------------- | ------- | ------- | --------- | ----------------------------------------------------------------------------------------------------- |
| `segment_duration_sec` | float | `5.0` | 1.0–30.0 | Duration of each audio segment (seconds). 5.0 recommended for sound-mark matching |
| `segment_hop_sec` | float | `2.5` | 0.5–15.0 | Hop between segments (seconds). 2.5 = 50% overlap. Set equal to `segment_duration_sec` for no overlap |
| `sample_rate` | integer | `48000` | — | Target sample rate (Hz). 48000 is the CLAP default; audio is resampled before embedding |
| `normalize_embeddings` | boolean | `true` | — | L2-normalize embeddings to unit vectors (recommended for cosine similarity) |
| `max_audio_length_sec` | float | `120.0` | 1.0–600.0 | Maximum audio length to process (seconds). Audio beyond this is truncated |
## Configuration Examples
```json Default (5s windows, 50% overlap) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_fingerprint_extractor",
"version": "v1",
"input_mappings": {
"audio": "audio_url"
},
"parameters": {}
}
}
```
```json Video Audio Track theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_fingerprint_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"parameters": {}
}
}
```
```json Non-Overlapping Segments, Longer Audio theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_fingerprint_extractor",
"version": "v1",
"input_mappings": {
"audio": "audio_url"
},
"parameters": {
"segment_duration_sec": 10.0,
"segment_hop_sec": 10.0,
"max_audio_length_sec": 300.0
}
}
}
```
## Performance & Costs
| Metric | Value |
| -------------------- | -------------------------------------------------------------------------------------- |
| **Cost** | See [Billing & Pricing](/docs/platform/billing) — rates come from `GET /v1/billing/pricing` |
| **Model** | `laion/clap-htsat-tiny` (CLAP) |
| **Default coverage** | First 120 s of audio (configurable to 600 s) |
## Vector Index
| Property | Value |
| ------------------- | ------------------------------------------- |
| **Index name** | `audio_fingerprint_extractor_v1_embedding` |
| **Dimensions** | 512 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `laion/clap-htsat-tiny` |
| **Normalization** | L2 normalized (when `normalize_embeddings`) |
## Limitations
* **Length cap**: Audio beyond `max_audio_length_sec` is truncated (default 120 s).
* **Not for transcription**: Produces acoustic fingerprints, not text — pair with a transcription extractor for spoken-word search.
* **Segment fan-out**: Overlapping windows multiply the document count per source; tune `segment_hop_sec` to control density.
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Universal Extractor](/docs/processing/extractors/universal)
# Audio Sentiment Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/audio-sentiment
Vocal intelligence for financial earnings calls — FinBERT text sentiment, prosodic audio features, and speaker diarization for quantitative alternative data
**Not yet available.** This extractor is a design/roadmap page — it is not in the
platform's extractor registry today, and referencing it in a collection returns a
validation error. For working alternatives see the
[extractor catalog](/docs/processing/feature-extractors).
Configuring collections by built-in extractor name is a **deprecated** path — collections are now created by picking [features](/docs/processing/features). This extractor does not yet have a direct feature-key replacement; existing `feature_extractor` configs keep working. See the [migration guide](/docs/processing/extractor-migration).
Runnable reference for every built-in Mixpeek extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry, so it always matches production.
![Audio sentiment extractor pipeline showing speaker diarization, parallel FinBERT and prosodic feature extraction, and output alpha signals]()
The audio sentiment extractor processes **earnings call recordings, analyst day presentations, Fed press conferences, and financial podcasts** to produce two parallel signal streams: FinBERT financial-domain text sentiment (768D) from Whisper transcription, and a 5-feature prosodic vector (128D) capturing vocal stress, hesitation, and deception markers. Speaker diarization separates management from analysts for role-attributed sentiment.
This extractor addresses the gap identified in SEC 8-K forward guidance NLP studies: text-only sentiment models generate crowded alpha (in-sample IC \~+0.12 but poor walk-forward generalization). The five prosodic features — pitch variability, speech rate, vocal energy, pause ratio, and audio-text alignment — are largely uncorrelated with published text signals and untested at scale, representing a structural alternative data opportunity.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/audio\_sentiment\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/audio_sentiment_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Filter Dataset** (if `collection_id` provided)
* Filter to specified collection
2. **Apply Input Mappings**
* Resolve audio/video field from source (e.g., `payload.audio_url`, `payload.webcast_url`)
3. **Audio Extraction** (conditional: if video input)
* FFmpeg strips audio track from MP4/MOV; supports AAC, MP3, FLAC output
4. **Voice Activity Detection + Segmentation**
* `split_method: time` — fixed-length windows (default 30s)
* `split_method: silence` — split at natural speech pauses (VAD threshold configurable)
* `split_method: speaker` — one segment per speaker turn (requires `run_diarization=true`)
5. **Speaker Diarization** (conditional: if `run_diarization=true`)
* pyannote.audio 3.x pipeline separates speakers (CEO, CFO, Analyst\_1, etc.)
* Assigns `speaker_id` and optionally maps to `speaker_role` via role manifest
6. **Transcription** (conditional: if `run_transcription=true`)
* Whisper large-v3-turbo speech-to-text with financial vocabulary prompt
* Per-segment timestamps aligned to diarization boundaries
7. **FinBERT Text Sentiment** (conditional: if `run_finbert=true`)
* ProsusAI/FinBERT financial-domain sentiment classifier
* Outputs `sentiment_label` (positive/negative/neutral), `sentiment_score` (-1 to +1), `confidence`
* Generates 768D FinBERT CLS embedding for semantic search
8. **Prosodic Feature Extraction** (conditional: if `run_prosodics=true`)
* LibROSA + Parselmouth extract 5 features per segment:
* **Pitch variability** (F0 standard deviation, Hz) — hesitation and stress indicator
* **Speech rate** (words per minute) — confidence and urgency signal
* **Vocal energy** (RMS dB) — assertiveness and emotional weight
* **Pause ratio** (fraction of silence) — cognitive load and evasiveness marker
* **Vocal tremor** (jitter + shimmer) — anxiety and deception indicator
* Normalized into a 128D prosodic embedding for similarity search
9. **Audio-Text Alignment Score** (conditional: if `run_alignment=true`)
* Cosine similarity between FinBERT sentiment direction and prosodic valence
* Low alignment = voice contradicts words (high-value deception/stress signal)
10. **LLM Structured Enrichment** (conditional: if `run_llm_enrichment=true` or `response_shape` set)
* Gemini/GPT-4o processes transcription with custom prompt
* Extracts structured signals: guidance confidence, topic classification, hedging language
11. **Output**
* Per-segment documents with both embedding types, raw prosodic features, sentiment scores, speaker metadata, and computed alpha signals
## When to Use
| Use Case | Description |
| ----------------------------------- | ------------------------------------------------------------------------------ |
| **Earnings call analysis** | Process quarterly earnings calls for CEO/CFO vocal stress relative to guidance |
| **Forward guidance scoring** | Score management's confidence level on forward-looking statements |
| **Analyst day processing** | Build speaker-attributed sentiment timelines across presentations |
| **Fed press conference monitoring** | Track FOMC chair vocal markers around policy language |
| **Alpha signal generation** | Combine text + prosodic features for uncrowded quantitative factors |
| **Sentiment divergence detection** | Flag calls where management tone contradicts word sentiment |
| **Comparative speaker analysis** | Track individual executive vocal patterns across multiple quarters |
| **Podcast / financial media** | Extract sentiment from analyst interviews, TV appearances, podcasts |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------------------- | ---------------------------------------------------------------- |
| Text-only documents (PDFs, filings) | `document_extractor` or `text_extractor` |
| Very short clips (\< 10 seconds) | Processing overhead disproportionate |
| Non-speech audio (music, noise) | `multimodal_extractor` |
| Real-time live streaming | Specialized streaming extractors |
| Non-English earnings calls | Set `transcription_language` explicitly; FinBERT is English-only |
## Supported Input Types
| Input | Type | Description | Processing |
| ------- | ------ | -------------------------------------- | -------------------------- |
| `audio` | string | URL or S3 path to MP3, WAV, FLAC, M4A | Direct processing |
| `video` | string | URL or S3 path to MP4, MOV, MKV | Audio extracted via FFmpeg |
| `url` | string | Direct URL to webcast / podcast stream | Downloaded and processed |
**Supported audio formats:** MP3, WAV, FLAC, M4A, OGG, OPUS
**Supported video formats (audio extracted):** MP4, MOV, MKV, AVI, WebM
## Input Schema
Provide **one** of the following inputs:
```json theme={null}
{
"audio": "s3://bucket/earnings/AAPL_Q4_2024.mp3"
}
```
```json theme={null}
{
"video": "s3://bucket/investor-day/msft-2024-ceo-keynote.mp4"
}
```
```json theme={null}
{
"url": "https://edge.media-server.com/mmc/p/xyz/earnings-call.mp3"
}
```
| Field | Type | Description |
| ------- | ------ | -------------------------------------------------------------- |
| `audio` | string | URL/S3 path to audio file. Recommended: \< 3 hours per file |
| `video` | string | URL/S3 path to video file; audio track extracted automatically |
| `url` | string | Direct stream URL; downloaded before processing |
## Output Schema
Each audio segment produces one document:
| Field | Type | Description |
| ------------------------------------------------- | ----------- | ---------------------------------------------------------------- |
| `start_time` | number | Segment start time in seconds |
| `end_time` | number | Segment end time in seconds |
| `speaker_id` | string | Diarized speaker label (e.g., `SPEAKER_00`) |
| `speaker_role` | string | Mapped role if manifest provided (e.g., `CEO`, `CFO`, `Analyst`) |
| `transcription` | string | Whisper transcription of segment |
| `sentiment_label` | string | `positive`, `negative`, or `neutral` |
| `sentiment_score` | number | FinBERT sentiment score: -1.0 (negative) to +1.0 (positive) |
| `sentiment_confidence` | number | FinBERT confidence 0.0–1.0 |
| `pitch_variability_hz` | number | F0 standard deviation (Hz) — stress/hesitation |
| `speech_rate_wpm` | number | Words per minute — confidence/urgency |
| `vocal_energy_db` | number | RMS energy in dB — assertiveness |
| `pause_ratio` | number | Fraction of silence 0.0–1.0 — cognitive load |
| `vocal_tremor` | number | Jitter + shimmer composite 0.0–1.0 — anxiety |
| `audio_text_alignment` | number | Prosody-sentiment cosine alignment -1.0 to +1.0 |
| `stress_index` | number | Composite vocal stress score 0.0–1.0 |
| `audio_sentiment_extractor_v1_text_embedding` | float\[768] | FinBERT CLS embedding |
| `audio_sentiment_extractor_v1_prosodic_embedding` | float\[128] | Normalized prosodic feature vector |
```json theme={null}
{
"start_time": 245.0,
"end_time": 275.0,
"speaker_id": "SPEAKER_00",
"speaker_role": "CEO",
"transcription": "We're very confident in our Q1 guidance range of twelve to fourteen dollars per share...",
"sentiment_label": "positive",
"sentiment_score": 0.71,
"sentiment_confidence": 0.89,
"pitch_variability_hz": 38.2,
"speech_rate_wpm": 142,
"vocal_energy_db": -18.4,
"pause_ratio": 0.21,
"vocal_tremor": 0.14,
"audio_text_alignment": 0.63,
"stress_index": 0.31,
"audio_sentiment_extractor_v1_text_embedding": [0.041, -0.018, ...],
"audio_sentiment_extractor_v1_prosodic_embedding": [0.72, 0.34, ...]
}
```
## Parameters
### Audio Segmentation
| Parameter | Type | Default | Description |
| -------------- | ------ | ----------- | ------------------------------------------------------ |
| `split_method` | string | `"silence"` | Segmentation strategy: `time`, `silence`, or `speaker` |
**Fixed-interval splitting** — equal-duration segments regardless of speech content.
| Parameter | Type | Default | Description |
| --------------------- | ------- | ------- | --------------------------- |
| `time_split_interval` | integer | `30` | Segment duration in seconds |
**Best for:** Batch processing, predictable segment counts, initial exploration
```json theme={null}
{
"split_method": "time",
"time_split_interval": 30
}
```
**Voice activity detection** — splits at natural speech pauses. Preserves complete sentences and thoughts.
| Parameter | Type | Default | Description |
| ------------------------- | ------- | ------- | --------------------------------------- |
| `silence_db_threshold` | integer | `-40` | dB level below which audio is silence |
| `min_silence_duration_ms` | integer | `500` | Minimum silence length to trigger split |
**Best for:** Earnings calls, presentations, interviews — preserves semantic units
```json theme={null}
{
"split_method": "silence",
"silence_db_threshold": -40,
"min_silence_duration_ms": 500
}
```
**Speaker-turn segmentation** — one segment per speaker turn. Requires diarization. Ideal for Q\&A analysis.
**Characteristics:**
* Variable segment lengths (1s–5 min typical for earnings Q\&A)
* Each segment is a single speaker's continuous turn
* Enables per-speaker sentiment timelines
**Best for:** Q\&A sections, panel discussions, analyst questioning
```json theme={null}
{
"split_method": "speaker",
"run_diarization": true
}
```
### Feature Extraction Parameters
| Parameter | Type | Default | Description |
| ------------------------ | ------- | ------- | ------------------------------------------------------------ |
| `run_transcription` | boolean | `true` | Run Whisper transcription |
| `transcription_language` | string | `"en"` | Language code for transcription |
| `transcription_prompt` | string | `null` | Domain vocabulary hint (e.g., ticker symbols, product names) |
| `run_finbert` | boolean | `true` | Run FinBERT financial sentiment classification |
| `run_prosodics` | boolean | `true` | Extract 5 prosodic features + 128D embedding |
| `run_alignment` | boolean | `true` | Compute audio-text alignment score |
| `run_diarization` | boolean | `false` | Run speaker diarization (adds \~20% processing time) |
| `num_speakers` | integer | `null` | Hint for diarization (null = auto-detect) |
### Speaker Role Manifest
Map diarized speaker IDs to roles (CEO, CFO, Analyst, etc.) using a manifest:
```json theme={null}
{
"speaker_role_manifest": {
"SPEAKER_00": "CEO",
"SPEAKER_01": "CFO",
"SPEAKER_02": "Analyst"
}
}
```
When `speaker_role_manifest` is not provided, roles are labeled `SPEAKER_00`, `SPEAKER_01`, etc.
### LLM Structured Extraction
| Parameter | Type | Default | Description |
| -------------------- | ---------------- | ------- | ----------------------------------- |
| `run_llm_enrichment` | boolean | `false` | Run LLM over transcription segments |
| `response_shape` | string \| object | `null` | Custom structured output schema |
**Natural Language Mode:**
```json theme={null}
{
"response_shape": "Extract: forward guidance confidence level (1-5), number of hedging phrases, primary topic discussed, and any mentioned risk factors"
}
```
**JSON Schema Mode for Quant Signals:**
```json theme={null}
{
"response_shape": {
"type": "object",
"properties": {
"guidance_confidence": { "type": "integer", "minimum": 1, "maximum": 5 },
"hedging_phrase_count": { "type": "integer" },
"topic": { "type": "string", "enum": ["revenue", "margins", "guidance", "macro", "capex", "other"] },
"risk_factors": { "type": "array", "items": { "type": "string" } },
"quantitative_claims": { "type": "array", "items": { "type": "string" } }
}
}
}
```
## Configuration Examples
```json Earnings Call — Full Analysis theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_sentiment_extractor",
"version": "v1",
"input_mappings": {
"audio": "audio_url"
},
"field_passthrough": [
{ "source_path": "metadata.ticker" },
{ "source_path": "metadata.fiscal_quarter" },
{ "source_path": "metadata.call_section" }
],
"parameters": {
"split_method": "speaker",
"run_transcription": true,
"transcription_language": "en",
"run_finbert": true,
"run_prosodics": true,
"run_alignment": true,
"run_diarization": true,
"num_speakers": 6,
"speaker_role_manifest": {
"SPEAKER_00": "CEO",
"SPEAKER_01": "CFO",
"SPEAKER_02": "IR"
}
}
}
}
```
```json CEO Prepared Remarks — Guidance Confidence theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_sentiment_extractor",
"version": "v1",
"input_mappings": {
"audio": "prepared_remarks_url"
},
"field_passthrough": [
{ "source_path": "metadata.ticker" },
{ "source_path": "metadata.event_date" }
],
"parameters": {
"split_method": "silence",
"silence_db_threshold": -38,
"run_transcription": true,
"transcription_prompt": "earnings call forward guidance revenue EPS margin",
"run_finbert": true,
"run_prosodics": true,
"run_alignment": true,
"run_llm_enrichment": true,
"response_shape": {
"type": "object",
"properties": {
"guidance_confidence": { "type": "integer", "minimum": 1, "maximum": 5 },
"hedging_phrase_count": { "type": "integer" },
"topic": { "type": "string" },
"forward_looking": { "type": "boolean" }
}
}
}
}
}
```
```json Fed Press Conference — Policy Tone theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_sentiment_extractor",
"version": "v1",
"input_mappings": {
"video": "press_conf_url"
},
"field_passthrough": [
{ "source_path": "metadata.fomc_date" },
{ "source_path": "metadata.rate_decision" }
],
"parameters": {
"split_method": "silence",
"run_transcription": true,
"run_finbert": true,
"run_prosodics": true,
"run_alignment": true,
"run_diarization": false
}
}
}
```
```json Analyst Day — Multi-Speaker Timeline theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_sentiment_extractor",
"version": "v1",
"input_mappings": {
"video": "analyst_day_recording"
},
"parameters": {
"split_method": "speaker",
"run_transcription": true,
"run_finbert": true,
"run_prosodics": true,
"run_alignment": true,
"run_diarization": true
}
}
}
```
```json Minimal — Transcription + FinBERT Only theme={null}
{
"feature_extractor": {
"feature_extractor_name": "audio_sentiment_extractor",
"version": "v1",
"input_mappings": {
"audio": "audio_url"
},
"parameters": {
"split_method": "time",
"time_split_interval": 60,
"run_transcription": true,
"run_finbert": true,
"run_prosodics": false,
"run_alignment": false,
"run_diarization": false
}
}
}
```
## Performance & Costs
### Processing Speed
| Configuration | Speed | Example |
| ------------------ | --------------- | ---------------------- |
| Transcription only | \~0.5x realtime | 60-min call → \~30 min |
| + FinBERT | \~0.6x realtime | 60-min call → \~36 min |
| + Prosodics | \~0.8x realtime | 60-min call → \~48 min |
| + Diarization | \~1.5x realtime | 60-min call → \~90 min |
| Full pipeline | \~1.5x realtime | 60-min call → \~90 min |
| Feature | Per-Segment Latency |
| ----------------------- | -------------------- |
| Transcription (Whisper) | \~150ms/sec of audio |
| FinBERT classification | \~25ms |
| Prosodic extraction | \~50ms |
| Speaker diarization | \~200ms/sec of audio |
| LLM enrichment | \~1.5s |
### Cost Estimates (per hour of audio)
| Configuration | Cost |
| ----------------------------------------- | ------ |
| **Minimal** (transcription + FinBERT) | \$0.08 |
| **Standard** (+ prosodics + alignment) | \$0.15 |
| **Full** (+ diarization + LLM enrichment) | \$0.25 |
**Batch processing**: Processing 1,000 S\&P 500 earnings calls (avg 60 min) at full configuration ≈ \$250
## Vector Indexes
### Text Embedding (FinBERT)
| Property | Value |
| -------------------- | --------------------------------------------- |
| **Index name** | `audio_sentiment_extractor_v1_text_embedding` |
| **Dimensions** | 768 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `finbert_sentiment_v1` |
| **Supported inputs** | text (transcription segments) |
### Prosodic Embedding
| Property | Value |
| -------------------- | ------------------------------------------------- |
| **Index name** | `audio_sentiment_extractor_v1_prosodic_embedding` |
| **Dimensions** | 128 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `prosodic_encoder_v1` |
| **Supported inputs** | audio segments |
## Alpha Signal Guide
This section describes the five core prosodic features and their interpretation as quantitative signals.
**What it measures:** Standard deviation of the fundamental frequency (F0) in Hz across the segment.
**Signal interpretation:**
* **High variability (> 50 Hz):** Elevated emotional engagement; can indicate stress or enthusiasm
* **Low variability (\< 15 Hz):** Monotone delivery; associated with rehearsed/scripted language or disengagement
* **Baseline deviation:** Compare against the speaker's historical mean F0 std dev for true anomaly detection
**Quant application:** Track CEO pitch variability during forward guidance vs. historical questions. Anomalous drops on guidance segments may precede earnings misses.
**What it measures:** Words per minute derived from Whisper word-level timestamps.
**Signal interpretation:**
* **High rate (> 180 wpm):** Urgency, anxiety, or over-rehearsed scripted answers
* **Low rate (\< 100 wpm):** Deliberate, careful language; common when discussing negative surprises
* **Rate deceleration mid-answer:** Suggests real-time reasoning, less scripted — higher authenticity signal
**Quant application:** Significant speech rate slowdown during Q\&A relative to prepared remarks may signal management is processing unexpected analyst questions.
**What it measures:** Root mean square energy of the audio signal in decibels.
**Signal interpretation:**
* **High energy:** Assertiveness and confidence; common in positive guidance delivery
* **Energy drop mid-sentence:** Hedging or trailing off; linguistic uncertainty
* **Segment-relative drop:** Cross-call energy tracking shows conviction level
**Quant application:** Energy drop on forward EPS guidance sentences (identifiable via LLM topic tagging) is a stress-linked signal distinct from text sentiment.
**What it measures:** Fraction of segment duration classified as silence (VAD threshold -40 dB).
**Signal interpretation:**
* **High pause ratio (> 0.35):** Cognitive load; speaker is reasoning in real time rather than reciting
* **Low pause ratio (\< 0.10):** Scripted, rehearsed delivery — less information content
* **Q\&A vs. prepared remarks delta:** A large increase in pause ratio during Q\&A is a well-documented stress marker
**Quant application:** Pause ratio on Q\&A segments answering analyst questions about inventory / margin / guidance has shown predictive value for negative guidance revisions in academic literature.
**What it measures:** Cosine similarity between the FinBERT sentiment direction (text) and prosodic valence (audio). Range: -1.0 to +1.0.
**Signal interpretation:**
* **High alignment (> 0.6):** Voice and words agree — higher conviction, less masking
* **Low alignment (0.1–0.4):** Moderate divergence — common in hedged language
* **Negative alignment (\< 0):** Voice contradicts words — strongest stress/deception marker; e.g., "We feel very good about guidance" delivered with high pitch variability, low energy, and high pauses
**Quant application:** This is the most novel of the five features. Text NLP cannot capture it. Segments with positive text sentiment but negative alignment are the primary alpha generation target.
### Composite Stress Index
The `stress_index` field (0.0–1.0) is a normalized composite of all five prosodic features:
```
stress_index = normalize(
0.25 * pitch_variability_z +
0.20 * speech_rate_z + # inverted: lower rate = higher stress
0.20 * (1 - vocal_energy_z) +
0.20 * pause_ratio_z +
0.15 * vocal_tremor_z
)
```
Where `_z` values are Z-scores computed against the speaker's rolling 4-quarter baseline when `speaker_id` is consistent across calls.
### Recommended Factor Construction
```python theme={null}
# Example: CEO guidance confidence factor
import mixpeek
client = mixpeek.Client(api_key="YOUR_KEY")
# Query for CEO guidance segments with sentiment divergence
results = client.retrievers.run(
retriever_id="earnings-sentiment-retriever",
inputs={
"query": "revenue guidance outlook fiscal year",
"filters": {
"speaker_role": "CEO",
"forward_looking": True,
"audio_text_alignment": {"$lt": 0.3}, # divergence signal
"stress_index": {"$gt": 0.6} # high stress
}
}
)
```
## Limitations
* **Speaker diarization accuracy**: pyannote achieves \~90% DER on clean 2-speaker recordings; accuracy degrades with > 8 speakers or poor audio quality
* **Non-English**: Whisper transcription supports 99 languages; FinBERT is English-only — for non-English calls, disable `run_finbert` and use multilingual sentiment models
* **Audio quality**: Prosodic features require 16kHz+ audio; compressed phone audio (8kHz) reduces pitch extraction accuracy by \~30%
* **Baseline dependency**: `stress_index` Z-score normalization requires at least 4 prior segments from the same `speaker_id` to be meaningful
* **Segment length**: Prosodic features are unreliable for segments \< 5 seconds; short interjections are best excluded
* **LLM enrichment latency**: `run_llm_enrichment=true` adds 1–2s per segment; disable for batch throughput
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Text Extractor](/docs/processing/extractors/text)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
* [Document Extractor](/docs/processing/extractors/document)
# Course Content Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/course-content
Decompose educational content into atomic learning units with text, code, and visual embeddings
**Not yet available.** This extractor is a design/roadmap page — it is not in the
platform's extractor registry today, and referencing it in a collection returns a
validation error. For working alternatives see the
[extractor catalog](/docs/processing/feature-extractors).
Configuring collections by built-in extractor name is a **deprecated** path — collections are now created by picking [features](/docs/processing/features). This extractor does not yet have a direct feature-key replacement; existing `feature_extractor` configs keep working. See the [migration guide](/docs/processing/extractor-migration).
Runnable reference for every built-in Mixpeek extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry, so it always matches production.
![Course content extractor pipeline showing video segmentation, PDF extraction, code decomposition, and multimodal embeddings]()
The course content extractor decomposes educational content into atomic learning units optimized for semantic retrieval. Processes video lectures with automatic transcription, PDF slides with layout awareness, and code archives with function-level granularity. Each unit receives E5-Large text embeddings (1024D), Jina Code embeddings (768D) for code snippets, and optional SigLIP visual embeddings (768D) for figures and screenshots.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/course\_content\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/course_content_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Filter Dataset** (if collection\_id provided)
* Filter to specified collection
2. **Content Detection & Routing**
* Auto-detect content type: video, PDF, or code archive
* Route to appropriate processor
3. **Video Segmentation** (if video input)
* Scene-based segmentation or SRT subtitle-based segmentation
* Extract transcripts via Whisper ASR (or use provided SRT)
* OCR video frames for screen text detection
4. **PDF Decomposition** (if PDF input)
* Layout detection: paragraphs, headers, tables, lists, figures, code blocks
* Layout-aware extraction per element or per page
* Extract images and figures with bounding boxes
5. **Code Archive Processing** (if code input)
* Extract source files from ZIP archive
* Segment code into individual functions/classes
* Auto-detect programming language
6. **Multi-Modal Embedding Generation**
* E5-Large (1024D) for transcripts, PDF text, and captions
* Jina Code v2 (768D) for code snippets and functions
* SigLIP (768D) for figures, screenshots, diagrams (optional)
7. **LLM Enrichment** (optional: if `enrich_with_llm=true`)
* Generate summaries using Gemini
* Add semantic context and key concepts
8. **Output**
* Learning units with text\_content, code\_content, screen\_text
* Layout types, timing info, language tags
* Multiple embeddings per unit for diverse search scenarios
## When to Use
| Use Case | Description |
| --------------------------- | ----------------------------------------------------------------- |
| **Online courses** | Extract lectures, slides, and code into searchable learning units |
| **Technical documentation** | Decompose guides with code examples into semantic chunks |
| **Code tutorials** | Segment video + PDF + code into aligned learning units |
| **Educational archives** | Index historical lecture materials with multiple content types |
| **Multilingual learning** | Process educational content across 100+ languages |
| **API documentation** | Extract text, code examples, and diagrams with visual search |
## When NOT to Use
| Scenario | Recommended Alternative |
| -------------------------- | ----------------------------------------------------------- |
| Simple text documents only | `text_extractor` (faster, simpler) |
| Images and photos only | `image_extractor` |
| Single PDF documents | `document_graph_extractor` (better OCR, confidence scoring) |
| Pre-transcribed videos | `text_extractor` (use transcripts directly) |
## Input Schema
| Field | Type | Required | Description |
| -------------- | ------ | --------------------- | -------------------------------------------------------------------------------------------------- |
| `video` | string | (one of three) | URL or S3 path to video file (MP4, WebM, MOV). Maximum: 4 hours. Auto-detect format. |
| `srt` | string | optional (with video) | URL or S3 path to SRT subtitle file. Used if present; otherwise Whisper ASR generates transcripts. |
| `pdf` | string | (one of three) | URL or S3 path to PDF document. Multi-page supported. Maximum: 500 pages. |
| `code_archive` | string | (one of three) | URL or S3 path to ZIP archive containing source code. Maximum: 100MB. |
**Exactly one of `video`, `pdf`, or `code_archive` must be provided.**
```json theme={null}
{
"video": "s3://my-bucket/lectures/intro-to-ml.mp4",
"srt": "s3://my-bucket/lectures/intro-to-ml.srt"
}
```
**Input Examples:**
| Type | Example |
| -------------------- | ------------------------------------------------------------------------------------------------ |
| Video with subtitles | `{"video": "https://cdn.example.com/lecture.mp4", "srt": "https://cdn.example.com/lecture.srt"}` |
| PDF slides | `{"pdf": "s3://courses/machine-learning/slides-week-1.pdf"}` |
| Code archive | `{"code_archive": "s3://tutorials/python-algorithms.zip"}` |
## Output Schema
Each learning unit produces one or more documents depending on content type and `expand_to_granular_docs` setting:
| Field | Type | Description |
| ------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------- |
| `unit_type` | string | Type of unit: `video_segment`, `pdf_element`, `code_function`, `screen_text`, `figure` |
| `doc_type` | string | Granular type: `transcript`, `code`, `screen_text`, `visual`, `paragraph`, `table`, `list`, `header`, `figure` |
| `text_content` | string | Extracted text content |
| `code_content` | string | Source code (if applicable) |
| `code_language` | string | Programming language (Python, JavaScript, Java, etc.) |
| `screen_text` | string | OCR text from video frames or PDF screenshots |
| `title` | string | Unit title (lecture title, function name, figure caption) |
| `start_time` | number | Video start time in seconds (video units only) |
| `end_time` | number | Video end time in seconds (video units only) |
| `page_number` | integer | PDF page number (0-indexed, PDF units only) |
| `element_index` | integer | Element position within page (PDF units only) |
| `start_line` | integer | Start line number (code units only) |
| `end_line` | integer | End line number (code units only) |
| `segment_index` | integer | Segment position within source (video units only) |
| `element_type` | string | PDF layout type: `paragraph`, `header`, `list`, `table`, `figure`, `code`, `footer` |
| `bbox` | object | Bounding box `{x, y, width, height}` (PDF elements with visual positioning) |
| `thumbnail_url` | string | S3 URL of thumbnail image (video frames, figure screenshots) |
| `intfloat__multilingual_e5_large_instruct` | float\[1024] | E5-Large text embedding, L2 normalized |
| `jinaai__jina_embeddings_v2_base_code` | float\[768] | Jina Code embedding (code units only) |
| `google__siglip_base_patch16_224` | float\[768] | SigLIP visual embedding (if `run_visual_embedding=true`) |
| `llm_summary` | string | LLM-generated summary (if `enrich_with_llm=true`) |
```json theme={null}
{
"unit_type": "video_segment",
"doc_type": "transcript",
"text_content": "In this section, we explore supervised learning algorithms...",
"screen_text": "SUPERVISED LEARNING\n- Regression\n- Classification",
"title": "Intro to ML: Supervised Learning",
"start_time": 120.5,
"end_time": 245.3,
"segment_index": 3,
"thumbnail_url": "s3://mixpeek/ns_123/thumbnails/seg_3.jpg",
"intfloat__multilingual_e5_large_instruct": [0.023, -0.041, 0.018, ...],
"llm_summary": "Introduction to supervised learning covering regression and classification techniques"
}
```
## Parameters
### Video Segmentation Parameters
| Parameter | Type | Default | Range | Description |
| ---------------------------- | ------- | --------- | ---------------- | --------------------------------------------------------------------------------------------- |
| `target_segment_duration_ms` | integer | 120000 | 30000-600000 | Target duration for each video segment (30 sec - 10 min) |
| `min_segment_duration_ms` | integer | 30000 | 10000+ | Minimum segment duration to create |
| `segmentation_method` | string | `"scene"` | scene, srt, time | Segmentation strategy: scene detection, SRT markers, or fixed time intervals |
| `scene_detection_threshold` | float | 0.3 | 0.1-0.9 | Scene change sensitivity (lower = more scenes detected) |
| `use_whisper_asr` | boolean | true | - | Use Whisper ASR for transcription if SRT not provided |
| `expand_to_granular_docs` | boolean | true | - | Create separate documents for transcript, screen\_text, and visual (one per granularity type) |
| `ocr_frames_per_segment` | integer | 3 | 1-10 | Number of frames to OCR per segment |
#### Segmentation Methods
| Method | Description | Best For |
| ------- | ---------------------------------------- | -------------------------------------------- |
| `scene` | ML-based scene detection (PySceneDetect) | Lectures with natural topic breaks |
| `srt` | Use SRT subtitle markers as boundaries | Prepared materials with timing metadata |
| `time` | Fixed time intervals | Uniform segment length regardless of content |
### PDF Extraction Parameters
| Parameter | Type | Default | Description |
| --------------------- | ------- | --------------- | ----------------------------------------------------------------------------- |
| `pdf_extraction_mode` | string | `"per_element"` | `per_page` (one doc per page) or `per_element` (one doc per detected element) |
| `pdf_render_dpi` | integer | 150 | DPI for rendering PDF pages (72-300). Higher = better OCR quality, slower |
| `detect_code_in_pdf` | boolean | true | Automatically detect and tag code blocks in PDF text |
### Code Extraction Parameters
| Parameter | Type | Default | Description |
| --------------------- | ------- | ------------------------------------------------------------ | ---------------------------------------------------- |
| `segment_functions` | boolean | true | Segment code files into individual functions/classes |
| `supported_languages` | array | `["python", "javascript", "java", "go", "rust", "c", "cpp"]` | Programming languages to extract and embed |
### Feature Extraction Parameters
| Parameter | Type | Default | Description |
| --------------------------- | ------- | ----------- | ------------------------------------------------------------------------------------------- |
| `run_text_embedding` | boolean | true | Generate E5-Large text embeddings for transcripts and text content |
| `run_code_embedding` | boolean | true | Generate Jina Code embeddings for code snippets |
| `run_visual_embedding` | boolean | false | Generate SigLIP visual embeddings for figures and screenshots |
| `visual_embedding_use_case` | string | `"lecture"` | Context for visual embedding: `lecture`, `code_demo`, `tutorial`, `presentation`, `dynamic` |
| `extract_screen_text` | boolean | true | Run OCR on video frames to extract on-screen text |
| `generate_thumbnails` | boolean | true | Generate and store thumbnail images |
| `use_cdn` | boolean | false | Use CDN for thumbnail delivery (if available) |
### LLM Enrichment Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------- | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `enrich_with_llm` | boolean | false | Enable LLM-generated summaries and key concept extraction |
| `llm_prompt` | string | `"Summarize this educational content, highlighting key concepts, learning objectives, and main takeaways"` | Custom prompt for LLM enrichment |
## Configuration Examples
```json Video with Scene Detection theme={null}
{
"feature_extractor": {
"feature_extractor_name": "course_content_extractor",
"version": "v1",
"input_mappings": {
"video": "lecture_url",
"srt": "subtitle_url"
},
"field_passthrough": [
{ "source_path": "metadata.course_id" },
{ "source_path": "metadata.lesson_number" }
],
"parameters": {
"target_segment_duration_ms": 120000,
"segmentation_method": "scene",
"scene_detection_threshold": 0.3,
"use_whisper_asr": true,
"expand_to_granular_docs": true,
"ocr_frames_per_segment": 3,
"run_text_embedding": true,
"run_code_embedding": true,
"run_visual_embedding": false,
"generate_thumbnails": true
}
}
}
```
```json PDF Slides with Code Detection theme={null}
{
"feature_extractor": {
"feature_extractor_name": "course_content_extractor",
"version": "v1",
"input_mappings": {
"pdf": "slides_url"
},
"field_passthrough": [
{ "source_path": "metadata.course_id" },
{ "source_path": "metadata.instructor" }
],
"parameters": {
"pdf_extraction_mode": "per_element",
"pdf_render_dpi": 150,
"detect_code_in_pdf": true,
"run_text_embedding": true,
"run_code_embedding": true,
"run_visual_embedding": false,
"generate_thumbnails": true
}
}
}
```
```json Code Archive with All Embeddings theme={null}
{
"feature_extractor": {
"feature_extractor_name": "course_content_extractor",
"version": "v1",
"input_mappings": {
"code_archive": "source_code_url"
},
"field_passthrough": [
{ "source_path": "metadata.tutorial_name" },
{ "source_path": "metadata.difficulty_level" }
],
"parameters": {
"segment_functions": true,
"supported_languages": ["python", "javascript", "java", "go", "rust"],
"run_text_embedding": true,
"run_code_embedding": true,
"run_visual_embedding": false
}
}
}
```
```json Video with Full Enrichment theme={null}
{
"feature_extractor": {
"feature_extractor_name": "course_content_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url",
"srt": "srt_url"
},
"parameters": {
"target_segment_duration_ms": 180000,
"segmentation_method": "srt",
"use_whisper_asr": false,
"expand_to_granular_docs": true,
"ocr_frames_per_segment": 5,
"run_text_embedding": true,
"run_code_embedding": true,
"run_visual_embedding": true,
"visual_embedding_use_case": "lecture",
"extract_screen_text": true,
"generate_thumbnails": true,
"enrich_with_llm": true,
"llm_prompt": "Extract learning objectives, key concepts, and prerequisites from this lecture segment"
}
}
}
```
```json PDF with LLM Summaries theme={null}
{
"feature_extractor": {
"feature_extractor_name": "course_content_extractor",
"version": "v1",
"input_mappings": {
"pdf": "textbook_chapter"
},
"parameters": {
"pdf_extraction_mode": "per_element",
"pdf_render_dpi": 200,
"detect_code_in_pdf": true,
"run_text_embedding": true,
"run_code_embedding": true,
"generate_thumbnails": true,
"enrich_with_llm": true,
"llm_prompt": "Generate a concise summary focusing on practical applications and code examples"
}
}
}
```
## Performance & Costs
| Metric | Value |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Video processing** | \~1 minute per 10 minutes of video (depends on segmentation) |
| **PDF processing** | \~2-5 seconds per page (depends on DPI and layout complexity) |
| **Code processing** | \~50-100ms per 1KB of code |
| **Embedding latency** | \~5ms per text unit (E5), \~10ms per code unit (Jina), \~50ms per visual unit (SigLIP) |
| **Cost** | Billed per video minute / PDF page / code token — see [Billing & Pricing](/docs/platform/billing); rates come from `GET /v1/billing/pricing` |
| **GPU acceleration** | Recommended for 10+ videos; 2-3x speedup |
## Vector Indexes
All three embeddings are stored as [MVS](https://mixpeek.com/mvs) named vectors for hybrid search:
| Property | Value |
| ------------------- | ------------------------------------------ |
| **Index 1 name** | `intfloat__multilingual_e5_large_instruct` |
| **Dimensions** | 1024 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Normalization** | L2 normalized |
| Property | Value |
| ------------------- | -------------------------------------- |
| **Index 2 name** | `jinaai__jina_embeddings_v2_base_code` |
| **Dimensions** | 768 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Normalization** | L2 normalized |
| Property | Value |
| ------------------- | ----------------------------------------- |
| **Index 3 name** | `google__siglip_base_patch16_224` |
| **Dimensions** | 768 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Inference model** | `google_siglip_base_v1` |
| **Status** | Optional (if `run_visual_embedding=true`) |
## Comparison with Other Extractors
| Feature | course\_content\_extractor | text\_extractor | multimodal\_extractor | document\_graph\_extractor |
| --------------------- | ------------------------------------------ | ----------------------- | --------------------- | -------------------------- |
| **Input types** | Video, PDF, Code | Text only | Video, Image, Text | PDF only |
| **Segmentation** | Scene/SRT/time | Word/sentence/paragraph | N/A | Layout-based |
| **Text embeddings** | E5-Large (1024D) | E5-Large (1024D) | Vertex AI (1408D) | E5-Large (1024D) |
| **Code embeddings** | Jina Code (768D) | ✗ | ✗ | ✗ |
| **Visual embeddings** | SigLIP (768D) optional | ✗ | Vertex AI (1408D) | ✗ |
| **Best for** | Educational content | Text search | Unified multimodal | Complex PDF layouts |
| **Cost per unit** | See [Billing & Pricing](/docs/platform/billing) | Per token | Per video minute | Per page |
## Limitations
* **Video length**: Optimized for videos up to 4 hours. Longer videos may require segmentation.
* **Transcription quality**: Whisper ASR works best with clear audio; noisy lectures may have reduced accuracy.
* **Code extraction**: Requires valid ZIP archives; loose files not supported.
* **Language support**: Code embedding works with common languages; domain-specific DSLs have reduced accuracy.
* **PDF complexity**: Complex layouts with nested tables may have reduced extraction quality.
* **Visual embeddings**: Optional and add significant processing cost.
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Text Extractor](/docs/processing/extractors/text)
* [Image Extractor](/docs/processing/extractors/image)
* [Document Graph Extractor](/docs/processing/extractors/document)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
# Document Graph Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/document
Extract spatial blocks from PDFs with layout classification, confidence scoring, optional VLM correction, and 1024-d E5 text embeddings
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["document_layout"]` (layout/structure extraction on top of the `document_search` base). Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
![Document graph extractor pipeline showing PDF parsing, layout detection, and block extraction]()
The document graph extractor decomposes PDFs into **spatial blocks** — paragraphs, tables, forms, lists, headers, footers, figures, and handwritten content — each with a bounding box, a layout class, and a confidence score. Low-confidence blocks can be corrected by a vision language model (Gemini, GPT-4V, or Claude). Block text is optionally embedded with E5-Large (1024-d) for semantic search. It is the right tool for archival documents, scanned files, and anything that needs spatial understanding rather than a flat text dump.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/document\_graph\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/document_graph_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Layout detection** (if `use_layout_detection`, the default) — find all document elements with `layout_detector` (`pymupdf` fast rule-based, or `docling` SOTA ML/DiT). Detects text regions **and** non-text elements (scanned images, figures, charts) as separate blocks.
2. **Block grouping** — when layout detection is off, group text spans into blocks using `vertical_threshold` / `horizontal_threshold`; drop blocks shorter than `min_text_length`.
3. **Confidence scoring** — assign `base_confidence` to native text (penalties for OCR artifacts / encoding issues), then tag each block A/B/C/D.
4. **VLM correction** (if `use_vlm_correction` and confidence \< `min_confidence_for_vlm`) — re-read low-confidence blocks with `vlm_provider`/`vlm_model`. Skipped entirely in `fast_mode`.
5. **Text embedding** (if `run_text_embedding`) — embed block text with E5-Large (1024-d).
6. **Thumbnails** (if `generate_thumbnails`) — render full-page and/or per-block thumbnails per `thumbnail_mode`.
7. **Output** — one document per block with layout class, bbox, text, confidence, and optional embedding/thumbnails.
## When to Use
| Use Case | Description |
| ------------------------------ | -------------------------------------------------------------------------------- |
| **Archival / scanned PDFs** | OCR + layout recovery with VLM correction for noisy scans and historical records |
| **Forms & tables** | Classify and isolate form fields, tables, and structured regions |
| **Spatial search** | Retrieve blocks by location and type, not just text |
| **Confidence-gated pipelines** | Route low-confidence blocks (tags C/D) to review |
| **Multi-layout documents** | Reports, contracts, and multi-column layouts that need block-level granularity |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Plain text you already have, or born-digital PDFs with perfect text | [`text_extractor`](/docs/processing/extractors/text) (faster, simpler) |
| Whole-document multimodal embedding | [`multimodal_extractor`](/docs/processing/extractors/multimodal) / [`universal_extractor`](/docs/processing/extractors/universal) |
| Maximum throughput, no spatial detail | `text_extractor`, or this extractor with `fast_mode: true` |
| Images only | [`image_extractor`](/docs/processing/extractors/image) |
| Non-PDF inputs | This extractor is PDF-only |
## Input Schema
| Field | Type | Required | Description |
| ----- | ------ | -------- | --------------------------------------------------------------------------------------- |
| `pdf` | string | **Yes** | URL or path to the PDF file. Supports multi-page PDFs. Populated from `input_mappings`. |
```json theme={null}
{
"pdf": "s3://my-bucket/contracts/lease.pdf"
}
```
**Input Examples:**
| Type | Example |
| ---------------- | --------------------------------------------- |
| Invoice | `s3://documents/invoices/inv-001.pdf` |
| Contract | `https://cdn.example.com/contracts/lease.pdf` |
| Scanned document | `s3://archive/scanned/1985-report.pdf` |
| Form | `s3://forms/application-form.pdf` |
Supported input types: **PDF only** (max 1 PDF per object). Max file size 100MB. For scanned documents, 150–300 DPI originals give the best OCR.
## Output Schema
One document per detected block:
| Field | Type | Description |
| -------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------ |
| `page_number` | integer | Page number in the original PDF (1-indexed) |
| `object_type` | enum | `paragraph`, `table`, `form`, `list`, `header`, `footer`, `figure`, or `handwritten` |
| `block_index` | integer | Block index within the page (0-indexed) |
| `bbox` | object | Bounding box: `{ x0, y0, x1, y1 }` |
| `text_raw` | string | Original extracted text |
| `text_corrected` | string \| null | Cleaned / VLM-corrected text |
| `overall_confidence` | float | Confidence score 0.0–1.0 |
| `confidence_tag` | enum | `A` (`≥0.85`), `B` (`≥0.70`), `C` (`≥0.50`), `D` (`<0.50`) |
| `document_graph_extractor_v1_text_embedding` | float\[1024] \| null | E5-Large block embedding (when `run_text_embedding`) |
| `thumbnail_url` / `segment_thumbnail_url` | string \| null | Full-page / per-block thumbnail URLs |
| `total_pages` | integer \| null | Total pages in the source PDF |
| `source_file` | string \| null | Original source file name |
```json theme={null}
{
"page_number": 1,
"object_type": "table",
"block_index": 3,
"bbox": { "x0": 72.0, "y0": 220.4, "x1": 540.0, "y1": 410.9 },
"text_raw": "Item Qty Price\nWidget 10 $4.00",
"text_corrected": "Item | Qty | Price\nWidget | 10 | $4.00",
"overall_confidence": 0.91,
"confidence_tag": "A"
}
```
## Parameters
### Layout Detection
| Parameter | Type | Default | Range | Description |
| ---------------------- | ------- | ----------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `use_layout_detection` | boolean | `true` | — | Enable ML-based layout detection to find all elements (text, images, tables, figures). When disabled, falls back to text-only extraction (faster, misses images). |
| `layout_detector` | string | `"pymupdf"` | `pymupdf`, `docling` | Engine used when `use_layout_detection=true`. `pymupdf`: fast rule-based (\~15 pages/sec). `docling`: SOTA ML/DiT — better semantic type detection and true table structure (\~3–8 sec/doc). |
| `render_dpi` | integer | `150` | 72–300 | DPI for page rendering (used for VLM correction). 72 fast/lower quality, 150 balanced, 300 high quality/slower. |
### Spatial Clustering (text-only fallback)
Only used when `use_layout_detection=false`.
| Parameter | Type | Default | Range | Description |
| ---------------------- | ------- | ------- | --------- | ------------------------------------------------------------------- |
| `vertical_threshold` | float | `15.0` | 1.0–100.0 | Max vertical gap (points) between lines grouped into the same block |
| `horizontal_threshold` | float | `50.0` | 1.0–200.0 | Max horizontal distance (points) for overlap/column detection |
| `min_text_length` | integer | `20` | 1–500 | Minimum block text length (chars); filters noise/fragments |
### Confidence & VLM Correction
| Parameter | Type | Default | Range | Description |
| ------------------------ | -------------- | -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `base_confidence` | float | `0.85` | 0.0–1.0 | Base confidence score for embedded (native) text |
| `min_confidence_for_vlm` | float | `0.6` | 0.0–1.0 | Confidence threshold below which VLM correction is triggered (only when `use_vlm_correction=true`) |
| `use_vlm_correction` | boolean | `true` | — | Enable VLM correction for low-confidence blocks |
| `fast_mode` | boolean | `false` | — | Skip VLM correction entirely for max throughput (\~15 pages/sec). Overrides `use_vlm_correction` |
| `vlm_provider` | string | `"google"` | `google`, `openai`, `anthropic` | LLM provider for VLM correction |
| `vlm_model` | string | `"gemini-2.5-flash"` | — | Correction model, e.g. `gemini-2.5-flash`, `gpt-4o`, `claude-3-5-sonnet` |
| `llm_api_key` | string \| null | `null` | — | BYOK key for VLM correction. Supports secret references, e.g. `{{SECRET.openai_api_key}}`. Falls back to Mixpeek's default keys if unset. |
### Embedding & Thumbnails
| Parameter | Type | Default | Range | Description |
| --------------------- | ------- | -------- | ------------------------------ | -------------------------------------------------------------------- |
| `run_text_embedding` | boolean | `true` | — | Generate E5-Large (1024-d) text embeddings for block content |
| `generate_thumbnails` | boolean | `true` | — | Generate thumbnail images for blocks |
| `thumbnail_mode` | string | `"both"` | `full_page`, `segment`, `both` | Which thumbnails to render (`segment` = cropped to the block's bbox) |
| `thumbnail_dpi` | integer | `72` | 36–150 | DPI for thumbnail generation. Lower DPI = smaller files. |
## Configuration Examples
```json Default (layout + VLM + embeddings) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "document_graph_extractor",
"version": "v1",
"input_mappings": {
"pdf": "file_url"
},
"field_passthrough": [
{ "source_path": "metadata.invoice_id" },
{ "source_path": "metadata.vendor" }
],
"parameters": {}
}
}
```
```json Fast Mode (max throughput, no VLM) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "document_graph_extractor",
"version": "v1",
"input_mappings": {
"pdf": "file_url"
},
"parameters": {
"fast_mode": true,
"generate_thumbnails": false
}
}
}
```
```json High-Accuracy Scanned Docs (docling + VLM) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "document_graph_extractor",
"version": "v1",
"input_mappings": {
"pdf": "scanned_doc"
},
"parameters": {
"layout_detector": "docling",
"min_confidence_for_vlm": 0.75,
"vlm_provider": "anthropic",
"vlm_model": "claude-3-5-sonnet",
"render_dpi": 200
}
}
}
```
```json Text-Only Fallback (no layout detection) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "document_graph_extractor",
"version": "v1",
"input_mappings": {
"pdf": "pdf_url"
},
"parameters": {
"use_layout_detection": false,
"vertical_threshold": 15.0,
"horizontal_threshold": 50.0,
"run_text_embedding": true
}
}
}
```
## Layout Types
The extractor classifies blocks into these `object_type` values:
| Type | Description | Example |
| ------------- | -------------------------- | -------------------------------- |
| `paragraph` | Body text blocks | Article content, descriptions |
| `table` | Tabular data | Financial tables, data grids |
| `form` | Form fields and labels | Application forms, surveys |
| `list` | Bulleted or numbered lists | Requirements, instructions |
| `header` | Page headers | Document titles, section headers |
| `footer` | Page footers | Page numbers, disclaimers |
| `figure` | Images and captions | Charts, diagrams, photos |
| `handwritten` | Handwritten text | Signatures, annotations |
## Confidence Tags
Extraction quality is graded with confidence tags (thresholds on `overall_confidence`):
| Tag | Confidence | Description | Action |
| ----- | ---------- | ----------- | -------------------------------- |
| **A** | ≥ 0.85 | Excellent | Use directly |
| **B** | ≥ 0.70 | Good | Reliable, minor issues |
| **C** | ≥ 0.50 | Fair | Verify or trigger VLM correction |
| **D** | \< 0.50 | Poor | Needs VLM correction |
## Performance & Costs
| Metric | Value |
| ------------------------ | ------------------------------------------------------------------------------------------------------- |
| **Cost** | Billed per page — see [Billing & Pricing](/docs/platform/billing); rates come from `GET /v1/billing/pricing` |
| **`pymupdf` throughput** | \~15 pages/sec (rule-based) |
| **`docling` throughput** | \~3–8 sec/doc (ML/DiT) |
| **Fast mode** | \~15 pages/sec (skips VLM correction) |
## Vector Index
| Property | Value |
| ------------------- | ------------------------------------------------------------------------------- |
| **Index name** | `document_graph_extractor_v1_text_embedding` |
| **Dimensions** | 1024 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `multilingual_e5_large_instruct_v1` (`intfloat/multilingual-e5-large-instruct`) |
| **Normalization** | L2 normalized |
The embedding is optional. Set `run_text_embedding: false` for a layout-only extraction with no vector index.
## Limitations
* **PDF only**: Accepts a single PDF per object; does not process Word docs, images, or other formats.
* **VLM cost**: Each correction adds cost (see [Billing & Pricing](/docs/platform/billing)); gate it with `min_confidence_for_vlm` or disable via `fast_mode`.
* **Layout-detector tradeoff**: `docling` is more accurate but markedly slower than `pymupdf`.
* **Memory**: Large PDFs (100+ pages) may require increased memory.
* **Language / handwriting**: OCR works best with Latin scripts; handwriting detection is experimental and less reliable.
* **External dependency**: VLM correction depends on the selected provider's availability.
## Search the Extracted Text
Extracted block text (native or OCR/VLM-corrected) is embedded into the `document_graph_extractor_v1_text_embedding` index, so you search it with a [`feature_search`](/docs/retrieval/stages/feature-search) stage against `mixpeek://document_graph_extractor@v1/intfloat__multilingual_e5_large_instruct` (`input_mode: "text"`). For a ready-to-copy retriever — including confidence and layout-type filtering — see [Cookbook → Search OCR Text from Scanned PDFs](/docs/retrieval/cookbook#search-ocr-text-from-scanned-pdfs).
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Text Extractor](/docs/processing/extractors/text)
* [Image Extractor](/docs/processing/extractors/image)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
* [Universal Extractor](/docs/processing/extractors/universal)
* [Retrieval Cookbook — OCR / scanned-document search](/docs/retrieval/cookbook#search-ocr-text-from-scanned-pdfs)
# Face Identity Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/face-identity
Production-grade face recognition (SCRFD + ArcFace, 99.8%+ accuracy) — track a specific person or person-of-interest (POI) across images and video
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["faces"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
![Face identity extractor pipeline showing detection, alignment, and embedding generation]()
The face identity extractor provides production-grade face recognition using state-of-the-art models (SCRFD for detection + ArcFace for embeddings). Detects faces, aligns to canonical template, and generates 512-dimensional embeddings with 99.8%+ verification accuracy (LFW benchmark).
**Tracking a specific person (person-of-interest / POI).** This is the extractor to reach for when the goal is "find every clip a **person / individual / face** appears in" or "trace a person-of-interest across a video library." Pass a reference face image as a `content` query against the face embedding to run 1:N identification — see the [Face Search recipe](/docs/retrieval/cookbook#face-search-1n-identification). (For *what a person said*, pair this with cross-lingual transcript search via the [text extractor](/docs/processing/extractors/text).)
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/face\_identity\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/face_identity_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Filter Dataset** (if collection\_id provided)
* Filter to specified collection
2. **Content Type Routing**
* **Images:** Direct to Step 3
* **Videos:** Frame extraction (sampling at `video_sampling_fps`) → Step 3
* **PDFs:** Page rendering → Step 3
* **Mixed:** Branch by type, process separately, union results
3. **Face Detection** (SCRFD)
* Detect all faces per image/frame/page
* Extract 5-point facial landmarks (eyes, nose, mouth)
* Filter by `min_face_size` and `detection_threshold`
4. **5-Point Affine Face Alignment**
* Warp face to canonical 112×112 template
* Ensures consistent embeddings
5. **ArcFace Embedding Generation**
* arcface\_r100 model
* 512D L2-normalized embeddings
* Cosine similarity for matching
6. **Quality Scoring** (conditional: if `enable_quality_scoring=true`)
* Assess blur, size, landmark confidence
* Filter by `quality_threshold` if specified
7. **Video Deduplication** (conditional: if `video_deduplication=true` AND video content)
* Remove duplicate faces across frames
* Threshold-based similarity matching
* Track face timelines in video
8. **Output Validation**
9. **Output**
* Per-face documents with embeddings, bbox, landmarks, quality scores
## When to Use
| Use Case | Description |
| ------------------------- | --------------------------------------------- |
| **Face verification** | 1:1 matching to verify identity |
| **Face identification** | 1:N search to identify a person in a database |
| **Face clustering** | Group photos by person automatically |
| **Employee verification** | Workplace identity systems |
| **Photo organization** | Organize photo libraries by people |
| **Surveillance** | Security and monitoring applications |
## When NOT to Use
| Scenario | Recommended Alternative |
| ---------------------- | ----------------------- |
| General image search | `image_extractor` |
| Object/scene detection | `multimodal_extractor` |
| Video content analysis | `multimodal_extractor` |
| Non-face biometrics | Specialized extractors |
## Supported Input Types
| Input | Type | Description | Processing |
| ------------- | ------ | -------------- | ---------------------------------------- |
| `image` | string | URL or S3 path | Detect and embed all faces |
| `video` | string | URL or S3 path | Sample frames, detect faces, deduplicate |
| `video_frame` | string | URL or S3 path | Treated as image |
**Supported formats:**
* **Image**: JPEG, PNG, WebP, BMP
* **Video**: MP4, MOV, AVI, MKV, WebM
**Recommended resolution**: 640px+ for optimal face detection
## Input Schema
Provide **one** of the following inputs:
```json theme={null}
{
"image": "s3://photos/john-doe-portrait.jpg"
}
```
```json theme={null}
{
"video": "s3://segments/interview-clip.mp4"
}
```
| Field | Type | Description |
| ------------- | ------ | --------------------------------------------------------- |
| `image` | string | Image URL or S3 path containing faces |
| `video` | string | Video URL or S3 path. Subject to `max_video_length` limit |
| `video_frame` | string | Single video frame URL or S3 path (treated as image) |
## Output Schema
Each detected face produces one document with the following fields:
| Field | Type | Description |
| -------------------------------------- | ----------- | ---------------------------------------------- |
| `face_identity_extractor_v1_embedding` | float\[512] | ArcFace embedding, L2 normalized |
| `face_index` | integer | Index of this face in source image (0-based) |
| `bbox` | object | Bounding box `{x1, y1, x2, y2, width, height}` |
| `detection_score` | number | SCRFD detection confidence (0.0-1.0) |
| `landmarks` | object | 5 facial landmarks for alignment |
| `quality_score` | number | Face quality score (0.0-1.0) |
| `quality_components` | object | Quality component scores (blur, size, etc.) |
| `aligned_face_crop` | string | Base64 aligned 112x112 face crop (optional) |
| `frame_number` | integer | Frame number in source video |
| `timestamp` | number | Timestamp in source video (seconds) |
| `embedding_model` | string | Embedding model used |
| `detection_model` | string | Detection model used |
| `processing_time_ms` | number | Processing time (milliseconds) |
```json theme={null}
{
"face_identity_extractor_v1_embedding": [0.023, -0.041, 0.018, ...],
"face_index": 0,
"bbox": {"x1": 120, "y1": 80, "x2": 280, "y2": 300, "width": 160, "height": 220},
"detection_score": 0.98,
"landmarks": {"left_eye": [150, 140], "right_eye": [230, 142], ...},
"quality_score": 0.85,
"embedding_model": "arcface_r100",
"detection_model": "scrfd_2.5g",
"processing_time_ms": 45.2
}
```
## Parameters
### Detection Parameters
| Parameter | Type | Default | Description |
| --------------------- | ------- | -------------- | ------------------------------------- |
| `detection_model` | string | `"scrfd_2.5g"` | SCRFD model variant |
| `min_face_size` | integer | `20` | Minimum face size in pixels to detect |
| `detection_threshold` | float | `0.5` | Confidence threshold (0.0-1.0) |
| `max_faces_per_image` | integer | `null` | Maximum faces to process per image |
#### Detection Models
| Model | Speed | Accuracy | Best For |
| ------------ | ------- | -------- | -------------------------- |
| `scrfd_500m` | 2-3ms | Good | Real-time applications |
| `scrfd_2.5g` | 5-7ms | Better | **Recommended** - balanced |
| `scrfd_10g` | 10-15ms | Best | Maximum accuracy |
### Embedding Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ------- | ---------------- | ---------------------------- |
| `embedding_model` | string | `"arcface_r100"` | Face embedding model |
| `normalize_embeddings` | boolean | `true` | L2-normalize to unit vectors |
#### Embedding Models
| Model | Accuracy (LFW) | Speed | Notes |
| -------------- | -------------- | -------- | ---------------------------------- |
| `arcface_r100` | 99.8%+ | Standard | **Recommended** - highest accuracy |
| `arcface_r50` | 99.5%+ | Faster | Slightly lower accuracy |
| `magface_r100` | 99.7%+ | Standard | Includes built-in quality score |
### Quality Parameters
| Parameter | Type | Default | Description |
| ------------------------ | ------- | ------- | -------------------------------------------- |
| `enable_quality_scoring` | boolean | `true` | Compute quality scores (adds \~5ms per face) |
| `quality_threshold` | float | `null` | Minimum quality to index (null = index all) |
**Quality threshold guide:**
* `null` - Index all detected faces
* `0.5` - Moderate filtering (removes low quality)
* `0.7` - High quality only
### Video Parameters
| Parameter | Type | Default | Description |
| ------------------------------- | ------- | ------- | ------------------------------------ |
| `max_video_length` | integer | `60` | Maximum video length in seconds |
| `video_sampling_fps` | float | `1.0` | Frames per second to sample |
| `video_deduplication` | boolean | `true` | Remove duplicate faces across frames |
| `video_deduplication_threshold` | float | `0.8` | Cosine similarity for deduplication |
### Output Parameters
| Parameter | Type | Default | Description |
| -------------------------- | ------- | ------------ | -------------------------------------------- |
| `output_mode` | string | `"per_face"` | `per_face` or `per_image` |
| `include_face_crops` | boolean | `false` | Include aligned 112x112 face crops as base64 |
| `store_detection_metadata` | boolean | `true` | Store bbox, landmarks, detection scores |
## Configuration Examples
```json Employee Verification (High Quality) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "face_identity_extractor",
"version": "v1",
"input_mappings": {
"image": "photo_url"
},
"field_passthrough": [
{ "source_path": "metadata.employee_id" }
],
"parameters": {
"detection_model": "scrfd_2.5g",
"detection_threshold": 0.7,
"embedding_model": "arcface_r100",
"enable_quality_scoring": true,
"quality_threshold": 0.5,
"max_faces_per_image": 1,
"min_face_size": 40
}
}
}
```
```json Group Photo Processing theme={null}
{
"feature_extractor": {
"feature_extractor_name": "face_identity_extractor",
"version": "v1",
"input_mappings": {
"image": "image_url"
},
"field_passthrough": [
{ "source_path": "metadata.photo_id" },
{ "source_path": "metadata.event_name" }
],
"parameters": {
"detection_model": "scrfd_10g",
"detection_threshold": 0.5,
"embedding_model": "arcface_r100",
"max_faces_per_image": null,
"enable_quality_scoring": true
}
}
}
```
```json Surveillance Video theme={null}
{
"feature_extractor": {
"feature_extractor_name": "face_identity_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"field_passthrough": [
{ "source_path": "metadata.camera_id" },
{ "source_path": "metadata.location" }
],
"parameters": {
"detection_model": "scrfd_10g",
"detection_threshold": 0.6,
"embedding_model": "arcface_r100",
"max_video_length": 300,
"video_sampling_fps": 1.0,
"video_deduplication": true,
"video_deduplication_threshold": 0.8,
"min_face_size": 30,
"quality_threshold": 0.4
}
}
}
```
```json Photo Library Organization theme={null}
{
"feature_extractor": {
"feature_extractor_name": "face_identity_extractor",
"version": "v1",
"input_mappings": {
"image": "photo_url"
},
"field_passthrough": [
{ "source_path": "metadata.album" },
{ "source_path": "metadata.date_taken" }
],
"parameters": {
"detection_model": "scrfd_2.5g",
"embedding_model": "arcface_r100",
"enable_quality_scoring": true,
"include_face_crops": true,
"store_detection_metadata": true
}
}
}
```
```json Real-time Access Control theme={null}
{
"feature_extractor": {
"feature_extractor_name": "face_identity_extractor",
"version": "v1",
"input_mappings": {
"image": "capture_url"
},
"parameters": {
"detection_model": "scrfd_500m",
"detection_threshold": 0.8,
"embedding_model": "arcface_r50",
"max_faces_per_image": 1,
"min_face_size": 60,
"enable_quality_scoring": false
}
}
}
```
## Face Matching
Use cosine similarity to match faces:
| Similarity Score | Interpretation |
| ---------------- | ----------------------- |
| > 0.30 | Very likely same person |
| 0.25 - 0.30 | Likely same person |
| 0.20 - 0.25 | Possibly same person |
| \< 0.20 | Different people |
**Recommended threshold**: 0.25-0.30 for same person verification
## Performance & Costs
| Metric | Value |
| ------------------------- | -------------------------------------------------------------------------------------- |
| **Detection accuracy** | 99%+ (WIDER FACE benchmark) |
| **Verification accuracy** | 99.8%+ (LFW benchmark) |
| **Processing speed** | Detection: 5-7ms, Embedding: 10-15ms per face |
| **Cost** | See [Billing & Pricing](/docs/platform/billing) — rates come from `GET /v1/billing/pricing` |
### Video Processing
* **Deduplication**: Reduces 90-95% redundancy in video
* **Sampling**: 1 FPS recommended for most use cases
* **Max length**: 300 seconds (extraction only)
## Vector Index
| Property | Value |
| ------------------- | -------------------------------------- |
| **Index name** | `face_identity_extractor_v1_embedding` |
| **Dimensions** | 512 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Inference model** | `face_identity_arcface_r100_v1` |
## Pipeline Overview
1. **SCRFD Detection** - Bounding boxes + 5 landmarks
2. **5-Point Affine Alignment** - 112x112 canonical face
3. **ArcFace Embedding** - 512-d L2-normalized vector
4. **Quality Scoring** (optional) - Filter low-quality faces
## Limitations
* **Face only**: Does not identify age, gender, or expressions
* **Pose sensitivity**: Extreme angles may reduce accuracy
* **Occlusion**: Masks, glasses, hair may affect detection
* **Resolution**: Minimum 20px face size, 40px+ recommended
* **Lighting**: Poor lighting reduces quality scores
* **Video length**: Maximum 300 seconds per video
## Search by face
Once faces are indexed, search them with a reference image (1:N identification) using a [`feature_search`](/docs/retrieval/stages/feature-search) stage — pass the reference face as a `content` query. The feature URI is `mixpeek://face_identity_extractor@v1/insightface__arcface` — the output name is the model (`insightface__arcface`), not the internal vector-index name (`face_identity_extractor_v1_embedding`). If unsure, `GET /v1/collections/features/extractors/face_identity_extractor_v1` returns the exact `feature_uri`:
```json theme={null}
{
"stage_name": "face_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://face_identity_extractor@v1/insightface__arcface",
"query": { "input_mode": "content", "value": "{{INPUT.reference_face_url}}" },
"top_k": 50
}
],
"final_top_k": 20
}
}
}
```
Define `reference_face_url` in the retriever's `input_schema`, then execute with `{"inputs": {"reference_face_url": "https://.../person.jpg"}}` to find every document featuring that person.
## Build & maintain a named identity list
The 1:N search above matches **one** reference face at a time. To track a
**roster of known people** — a watchlist that puts a *name* on faces found in
your videos, that you add to and refine over time — build a **reference
collection** of labeled faces and let new footage auto-identify against it.
That whole lifecycle — enroll reference faces, label them with names,
auto-label people in incoming video, review the "unknown" faces, and promote a
newly-confirmed face back into the reference set so the list **self-improves** —
is the [**Bootstrap a Labeled Dataset**](/docs/tutorials/bootstrap-labeled-dataset)
tutorial (see its *People Identification* / *Face Recognition System* path).
Adding, correcting, or removing a person is just editing a document in the
reference collection; the next video ingest identifies against the updated list.
Rule of thumb: use **1:N search** (above) when you already have the one photo
you're chasing; use the **reference-collection roster** when you're maintaining
an ongoing list of named identities to match every new video against.
## Related
* [Feature Search stage](/docs/retrieval/stages/feature-search) — query face embeddings
* [Bootstrap a Labeled Dataset](/docs/tutorials/bootstrap-labeled-dataset) — build & maintain a named identity roster that auto-labels new video
* [Video Understanding](/docs/tutorials/video-understanding) — faces alongside visual + transcript search
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Image Extractor](/docs/processing/extractors/image)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
# Gemini Multifile Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/gemini-multifile
Embed multiple files from one object into a single 3072-d vector using Gemini Embedding 2 — one embedding per object, not per file
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["multimodal_understanding"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
The Gemini Multifile Extractor uses **Gemini Embedding 2** (`gemini-embedding-exp-03-07`, 3072-d) to embed all files of an object — images, PDFs, video, audio, and text — into **one unified vector per object** in a single API call.
This is fundamentally different from other extractors: instead of producing one document per file, it collapses all of an object's blobs into a single embedding representing the object as a whole.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/gemini\_multifile\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/gemini_multifile_extractor_v1)
## When to Use
| Use Case | Example |
| ----------------------- | ----------------------------------------------------------- |
| **Product catalogs** | Embed product photo + spec sheet + description together |
| **Medical records** | Embed scan + report + clinical notes as one object |
| **Legal documents** | Embed contract + exhibits + summary together |
| **E-commerce** | Embed product image + manual + label as one searchable unit |
| **Object-level search** | Find objects similar to a combination of inputs |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------------------------- | ---------------------------------------------------- |
| Per-file granularity needed (search within a PDF) | `document_graph_extractor` or `multimodal_extractor` |
| Single-file objects | `image_extractor` or `text_extractor` |
| Video frame-level search | `multimodal_extractor` |
## How It Works
Standard extractors produce **one document per blob** (file). The Gemini Multifile Extractor uses **array `input_mappings`** to collect multiple blob fields from one object into a single list, then sends all of them to Gemini Embedding 2 in one API call:
```
Object (hero_image + spec_sheet + description)
↓ array input_mappings
Single Gemini API call with all 3 parts
↓
One 3072-d embedding → One document per object
```
## Array `input_mappings`
The key difference from other extractors is the `input_mappings` value: instead of mapping one field, you map a **list of fields**. All listed fields are collected from each object and embedded together.
```json theme={null}
{
"input_mappings": {
"files": ["hero_image", "spec_sheet", "description"]
}
}
```
The key (`"files"`) is the extractor's input name. The value is a list of blob field names from your bucket schema. Each field can be an image, PDF, video, audio, or text blob.
## Output
| Field | Type | Description |
| ----------------------------------------- | ------------------ | ----------------------------------------- |
| `gemini_multifile_extractor_v1_embedding` | `float[]` (3072-d) | Unified embedding for all input files |
| `source_blob_count` | `int` | Number of blobs embedded together |
| `source_blob_properties` | `string[]` | Names of the blob fields that contributed |
## Parameters
| Parameter | Type | Default | Description |
| ----------------------- | ------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output_dimensionality` | integer | `3072` | Embedding dimensions: `3072`, `768`, or `256` |
| `task_type` | string | `RETRIEVAL_DOCUMENT` | Embedding task hint for Gemini Embedding 2. Values: `RETRIEVAL_DOCUMENT`, `RETRIEVAL_QUERY`, `SEMANTIC_SIMILARITY`, `CLASSIFICATION`, `CLUSTERING` |
| `input_key` | string | `files` | Must match the key in `input_mappings` |
At query time, Mixpeek automatically uses `RETRIEVAL_QUERY` — you only need to set `task_type` at index time. The default `RETRIEVAL_DOCUMENT` is correct for most use cases. See [Text Extractor embedding task docs](/docs/processing/extractors/text#embedding-task) for details on each value.
## Complete Collection Setup
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "product-catalog",
"bucket_schema": {
"properties": {
"product_id": {"type": "string"},
"hero_image": {"type": "image"},
"spec_sheet": {"type": "pdf"},
"description": {"type": "text"}
}
}
}'
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/collections" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "product-embeddings",
"source": { "type": "bucket", "bucket_ids": ["bkt_abc123"] },
"feature_extractor": {
"feature_extractor_name": "gemini_multifile_extractor",
"version": "v1",
"input_mappings": {
"files": ["hero_image", "spec_sheet", "description"]
}
}
}'
```
The value of `"files"` is a **list** of field names, not a single string. This is what triggers the multi-file embedding behavior. All three fields are embedded together into one vector.
Each object must have all the mapped fields populated:
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/objects" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"blobs": [
{ "property": "hero_image", "type": "image", "data": "s3://my-bucket/products/sku-001/hero.jpg" },
{ "property": "spec_sheet", "type": "pdf", "data": "s3://my-bucket/products/sku-001/spec.pdf" },
{ "property": "description", "type": "text", "data": "Lightweight carbon-fiber trail running shoe with Vibram outsole" }
],
"metadata": { "product_id": "SKU-001" }
}'
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/batches" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"bucket_id": "bkt_abc123",
"collection_ids": ["col_xyz789"]
}'
```
Each object in the bucket produces **one document** containing one 3072-d embedding representing all its files combined.
## Creating a Retriever
After indexing, create a retriever to search the embedded objects. The `feature_search` stage supports two query input modes for this extractor:
### Single-item query (`text` or `content` mode)
Query with a single URL or text string — Gemini embeds it as-is and searches:
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "product-object-search",
"collection_identifiers": ["col_xyz789"],
"input_schema": {
"query": {"type": "text", "description": "Search query", "required": true}
},
"stages": [
{
"stage_name": "Object Search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://gemini_multifile_extractor@v1/gemini-embedding-exp-03-07",
"query": {
"input_mode": "text",
"value": "{{INPUT.query}}"
},
"top_k": 20
}
],
"final_top_k": 10
}
}
}
]
}'
```
### Multi-file query (`multi_content` mode)
Query with **multiple files at once** — Gemini embeds all of them together, matching how objects were indexed. This produces the most accurate similarity scores because the query vector is built the same way as the index vectors.
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "product-multifile-search",
"collection_identifiers": ["col_xyz789"],
"input_schema": {
"image_url": {"type": "text", "description": "Product image URL"},
"description": {"type": "text", "description": "Product description text"}
},
"stages": [
{
"stage_name": "Multi-file Object Search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://gemini_multifile_extractor@v1/gemini-embedding-exp-03-07",
"query": {
"input_mode": "multi_content",
"values": [
"{{INPUT.image_url}}",
"{{INPUT.description}}"
]
},
"top_k": 20
}
],
"final_top_k": 10
}
}
}
]
}'
```
Execute it by passing multiple files in `inputs`:
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers/$RETRIEVER_ID/execute" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"inputs": {
"image_url": "https://example.com/query-shoe.jpg",
"description": "trail running shoe lightweight"
}
}'
```
`multi_content` is only valid for feature URIs backed by `gemini_multifile_extractor`. Using it with any other extractor returns a `400` error.
## Dimensionality Reduction
Gemini Embedding 2 supports truncated dimensions for storage cost reduction:
```json theme={null}
{
"feature_extractor": {
"feature_extractor_name": "gemini_multifile_extractor",
"version": "v1",
"input_mappings": {
"files": ["hero_image", "spec_sheet", "description"]
},
"params": {
"output_dimensionality": 768
}
}
}
```
| Dimensions | Storage per vector | Quality |
| ---------------- | ------------------ | ------------------------ |
| `3072` (default) | 12 KB | Best |
| `768` | 3 KB | Near-identical |
| `256` | 1 KB | Good — \~2% quality loss |
## Pricing
See [Billing & Pricing](/docs/platform/billing) — rates come from `GET /v1/billing/pricing`. One charge covers all N files within an object — not per file.
## Related
* [Feature Search Stage](/docs/retrieval/stages/feature-search) — Search with `multi_content` query mode
* [Image Extractor](/docs/processing/extractors/image) — Per-image embeddings (one document per image)
* [Multimodal Extractor](/docs/processing/extractors/multimodal) — Per-segment video/audio embeddings
* [Text Extractor](/docs/processing/extractors/text) — Per-chunk text embeddings
# Image Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/image
Dense vector embeddings for images using Google SigLIP (768D) for visual similarity search
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["image_search"]` (for PDF input, use `document_search`). Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
![Image extractor pipeline showing SigLIP processing and embedding generation]()
The image extractor generates dense vector embeddings from images using Google's SigLIP model (768D). Optimized for visual similarity search, product matching, and cross-modal search with text queries. Fast (\~50-100ms per image) and cost-effective.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/image\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/image_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Filter Dataset** (if collection\_id provided)
* Filter to specified collection
2. **Detect Content Types**
* Sample 100 rows to identify images vs PDFs
3. **PDF Page Expansion** (conditional: if PDF content detected)
* Render each PDF page at 72 DPI using PyMuPDF
* Create separate image for each page
4. **SigLIP Image Embedding Generation**
* Resize to 224×224 internally
* GPU-accelerated inference
* Generate 768D visual embeddings
5. **Thumbnail Generation** (conditional: if `enable_thumbnails=true`)
* Resize to 640px width at 85% quality
* Upload to S3 with optional CDN
6. **Output**
* Image/page documents with embeddings
* Optional thumbnail URLs
## When to Use
| Use Case | Description |
| ---------------------- | -------------------------------------------------------- |
| **Image search** | Find visually similar images in large collections |
| **Visual similarity** | Match products, artwork, or content by appearance |
| **Content discovery** | Recommend similar visual content |
| **Cross-modal search** | Find images using text queries (via SigLIP text encoder) |
| **E-commerce** | Product image search and visual recommendations |
| **Stock photo search** | Media library search by visual content |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------- | --------------------------------------- |
| Face recognition | `face_identity_extractor` |
| Video content | `multimodal_extractor` |
| Text-heavy images requiring OCR | `multimodal_extractor` with OCR enabled |
| Audio content | `audio_extractor` |
## Input Schema
| Field | Type | Required | Description |
| ------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------ |
| `image` | string | **Yes** | URL or S3 path to image file. Formats: JPEG, PNG, WebP, BMP. Any resolution (resized to 224x224 internally). |
```json theme={null}
{
"image": "s3://my-bucket/products/laptop-pro.jpg"
}
```
**Input Examples:**
| Type | Example |
| ------------- | ------------------------------------------------- |
| Product image | `s3://my-bucket/products/laptop-pro.jpg` |
| Stock photo | `https://cdn.example.com/photos/sunset-beach.jpg` |
| Catalog image | `s3://catalog/items/SKU-12345.png` |
**Supported Formats**: JPEG, PNG, WebP, BMP, GIF (static)
**Recommended Resolution**: 224x224 or larger (automatically resized)
**Max File Size**: 10MB recommended
## Output Schema
| Field | Type | Description |
| ------------------------------ | ----------- | -------------------------------------------- |
| `image_extractor_v1_embedding` | float\[768] | SigLIP image embedding, L2 normalized |
| `processing_time_ms` | number | Processing time in milliseconds |
| `thumbnail_url` | string | S3 URL of the thumbnail image (if generated) |
```json theme={null}
{
"image_extractor_v1_embedding": [0.023, -0.041, 0.018, ...],
"processing_time_ms": 85.2,
"thumbnail_url": "s3://mixpeek-storage/ns_123/thumbnails/thumb_001.jpg"
}
```
## Parameters
The image extractor uses sensible defaults and requires no additional parameters for basic usage.
| Parameter | Type | Default | Description |
| --------------- | ---- | ------- | ------------------------------------- |
| *None required* | - | - | All parameters use optimized defaults |
## Configuration Examples
```json Basic Image Embedding theme={null}
{
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": {
"image": "image_url"
},
"field_passthrough": [
{ "source_path": "metadata.product_id" }
],
"parameters": {}
}
}
```
```json E-commerce Product Images theme={null}
{
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": {
"image": "product_image"
},
"field_passthrough": [
{ "source_path": "metadata.sku" },
{ "source_path": "metadata.category" },
{ "source_path": "metadata.brand" }
],
"parameters": {}
}
}
```
```json Stock Photo Library theme={null}
{
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": {
"image": "photo_url"
},
"field_passthrough": [
{ "source_path": "metadata.photographer" },
{ "source_path": "metadata.tags" },
{ "source_path": "metadata.license" }
],
"parameters": {}
}
}
```
```json Art Collection theme={null}
{
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": {
"image": "artwork_url"
},
"field_passthrough": [
{ "source_path": "metadata.artist" },
{ "source_path": "metadata.title" },
{ "source_path": "metadata.year" },
{ "source_path": "metadata.medium" }
],
"parameters": {}
}
}
```
## Performance & Costs
| Metric | Value |
| -------------------- | -------------------------------------------------------------------------------------------------------- |
| **Processing speed** | \~50-100ms per image |
| **Batch processing** | Up to 16 images per batch |
| **GPU acceleration** | Supported for faster inference |
| **Cost** | Billed per image — see [Billing & Pricing](/docs/platform/billing); rates come from `GET /v1/billing/pricing` |
## Vector Index
| Property | Value |
| ------------------- | ---------------------------------------------------- |
| **Feature URI** | `mixpeek://image_extractor@v1/google_siglip_base_v1` |
| **Index name** | `image_extractor_v1_embedding` |
| **Dimensions** | 768 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Inference model** | `google_siglip_base_v1` |
In retrievers, reference this feature by its **Feature URI** above (the output name is `google_siglip_base_v1`, **not** the index name `image_extractor_v1_embedding`).
### Cross-Modal Search
The SigLIP embeddings are compatible with SigLIP text embeddings, enabling cross-modal search where you can:
* Find images using natural language text queries
* Match images to text descriptions
* Build hybrid search combining visual and textual similarity
## Comparison with Other Image Extractors
| Feature | image\_extractor | multimodal\_extractor |
| --------------- | ------------------------------------------ | ------------------------------- |
| **Dimensions** | 768 | 1408 |
| **Model** | SigLIP | Vertex AI Multimodal |
| **Processing** | Image only | Video, Image, Text, GIF |
| **Cross-modal** | SigLIP text encoder | Vertex text encoder |
| **Best For** | Fast image search | Unified multimodal search |
| **Cost** | See [Billing & Pricing](/docs/platform/billing) | Higher (includes more features) |
## Limitations
* **Image only**: Does not process video, audio, or text content
* **No OCR**: Cannot extract text from images; use `multimodal_extractor` with OCR
* **No face recognition**: For face matching, use `face_identity_extractor`
* **Single image**: Processes one image at a time (batch via API)
* **Resolution**: Input is resized to 224x224 internally
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
* [Face Identity Extractor](/docs/processing/extractors/face-identity)
* [Text Extractor](/docs/processing/extractors/text)
# Multimodal Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/multimodal
Unified embeddings for video, image, audio, text, and GIF with transcription, OCR, thumbnails, and structured extraction
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["multimodal_understanding"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Create a managed namespace and run this pipeline on your own files — transcripts, faces, on-screen text, and unified embeddings, searchable in minutes.
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
![Multimodal extractor pipeline showing video splitting, parallel processing with Whisper and embedding models, and output features]()
The multimodal extractor processes **video, audio, image, text, and GIF content** through a unified pipeline. Videos and audio are decomposed into segments with transcription (Whisper), visual embeddings, OCR, and descriptions. Images and text are embedded directly without decomposition.
Two versions are available:
| Version | Embedding Model | Dimensions | Key Difference |
| ------- | --------------------------- | ------------------------------ | ------------------------------------------------------------ |
| **v1** | Vertex Multimodal Embedding | 1408 | Established, lower dimensionality |
| **v2** | Gemini Embedding 2 | 3072 (configurable: 1536, 768) | Higher dimensionality, Matryoshka support, native multimodal |
Both versions share the same pipeline (FFmpeg chunking, Whisper, thumbnails, Gemini vision) and differ only in the multimodal embedding step.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/multimodal\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/multimodal_extractor_v1) or [multimodal\_extractor\_v2](https://api.mixpeek.com/v1/collections/features/extractors/multimodal_extractor_v2). You can also fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Filter Dataset** (if collection\_id provided)
* Filter to specified collection
2. **Apply Input Mappings**
3. **Detect Content Types** (sample 100 rows)
* Identify: video, audio, image, text, or mixed
4. **Content Routing**
* **Video:** FFmpeg chunking (time/scene/silence) → Steps 5-10
* **Audio:** FFmpeg audio chunking (time/silence) → Steps 5-8
* **Image:** Skip to Step 8
* **Text:** Skip to Step 8
* **Mixed:** Branch by type, process separately, union results
5. **Transcription** (conditional: if `run_transcription=true`, video/audio only)
* Whisper API or Local GPU speech-to-text
6. **Transcription Embeddings** (conditional: if `run_transcription_embedding=true`)
* E5-Large text embeddings (1024D) from transcribed audio
7. **Multimodal Embeddings** (conditional: if `run_multimodal_embedding=true`)
* **v1:** Vertex AI embeddings (1408D)
* **v2:** Gemini Embedding 2 (3072D, configurable)
* Unified embedding space enables cross-modal search
8. **Thumbnail Generation** (conditional: if `enable_thumbnails=true`, visual content only)
* 640px width at 85% quality, S3 upload with optional CDN
9. **Visual Analysis** (conditional: if `run_video_description` OR `run_ocr=true`, visual content only)
* Gemini-based descriptions and/or OCR text extraction
10. **Output**
* Segment/document records with embeddings, transcriptions, descriptions, OCR, thumbnails
## When to Use
| Use Case | Description |
| --------------------------- | --------------------------------------------- |
| **Video content libraries** | Search and navigate video segments by content |
| **Media platforms** | Search across spoken and visual content |
| **Educational content** | Find moments in lectures and tutorials |
| **Surveillance/security** | Event detection in footage |
| **Social media** | Process user-generated video content |
| **Broadcasting/streaming** | Large video catalog management |
| **Marketing analytics** | Analyze video campaigns |
| **Cross-modal search** | Find videos/images using text queries |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------------------- | -------------------------------- |
| Static image collections only | `image_extractor` |
| Audio-only content | `audio_extractor` |
| Very short videos (\< 5 seconds) | Processing overhead not worth it |
| Real-time live streams | Specialized streaming extractors |
| 8K+ resolution video | Consider downsampling first |
| Embed all files in one object as one vector | `gemini_multifile_extractor` |
## Supported Input Types
| Input | Type | Description | Processing |
| ------- | ------ | ------------------ | ----------------------------------- |
| `video` | string | URL or S3 path | Decomposed into segments |
| `image` | string | URL or S3 path | Direct embedding (no decomposition) |
| `text` | string | Plain text content | Direct embedding |
| `gif` | string | URL or S3 path | Treated as video, frame-by-frame |
**Supported formats:**
* **Video**: MP4, MOV, AVI, MKV, WebM, FLV
* **Image**: JPG, PNG, WebP, BMP
* **GIF**: Animated GIF
## Input Schema
Provide **one** of the following inputs:
```json theme={null}
{
"video": "s3://bucket/videos/lecture.mp4"
}
```
```json theme={null}
{
"image": "https://cdn.example.com/products/laptop.jpg"
}
```
```json theme={null}
{
"text": "High-performance laptop with M3 chip, perfect for developers"
}
```
| Field | Type | Description |
| ------------------ | ------ | -------------------------------------------------------------- |
| `video` | string | URL/S3 path to video file. Recommended: 720p-1080p, \< 2 hours |
| `image` | string | URL/S3 path to image file. Recommended: \< 10MB |
| `text` | string | Plain text for cross-modal embedding |
| `gif` | string | URL/S3 path to GIF file |
| `custom_thumbnail` | string | Optional custom thumbnail URL instead of auto-generated |
## Output Schema
Each video segment produces one document. Images and text produce one document each without segmentation.
### Segment & Timing Fields
| Field | Type | Description |
| ------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `start_time` | number | Segment start time in seconds |
| `end_time` | number | Segment end time in seconds |
| `start_frame` | integer | Start frame number (`start_time × fps`) |
| `end_frame` | integer | End frame number (`end_time × fps`) |
| `fps` | number | Frame rate of the preprocessed video used for chunking |
| `source_fps` | number | Original source video frame rate before any preprocessing (e.g. 29.97, 30, 23.976). Use this for precise frame-level calculations against the source video |
| `duration` | number | Total duration of the entire source video in seconds (not the segment duration) |
### Content Fields
| Field | Type | Description |
| --------------- | ------ | ------------------------------------------------------------------- |
| `transcription` | string | Transcribed audio content (requires `run_transcription`) |
| `description` | string | AI-generated segment description (requires `run_video_description`) |
| `ocr_text` | string | Text extracted from video frames (requires `run_ocr`) |
### URL Fields
| Field | Type | Description |
| ------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------- |
| `thumbnail_url` | string | S3/CDN URL of the thumbnail image |
| `source_video_url` | string | URL of the original source video |
| `video_segment_url` | string | S3 URL of this specific segment file. Enables [collection-to-collection decomposition](#collection-to-collection-pipelines) |
### Embedding Fields
| Field | Type | Description |
| ------------------------------------------------- | ------------ | -------------------------------- |
| `multimodal_extractor_v1_multimodal_embedding` | float\[1408] | Vertex AI multimodal embedding |
| `multimodal_extractor_v1_transcription_embedding` | float\[1024] | E5-Large transcription embedding |
| Field | Type | Description |
| ------------------------------------------------- | ------------ | ----------------------------------------------------------------- |
| `multimodal_extractor_v2_multimodal_embedding` | float\[3072] | Gemini Embedding 2 multimodal embedding (configurable: 1536, 768) |
| `multimodal_extractor_v2_transcription_embedding` | float\[1024] | E5-Large transcription embedding |
### Example Output
```json theme={null}
{
"start_time": 10.0,
"end_time": 20.0,
"start_frame": 20,
"end_frame": 40,
"fps": 2.0,
"source_fps": 29.97,
"duration": 120.5,
"transcription": "Welcome to today's lecture on machine learning fundamentals...",
"description": "Instructor standing at whiteboard, introducing ML concepts",
"ocr_text": "Machine Learning 101",
"thumbnail_url": "s3://mixpeek-storage/ns_123/thumbnails/thumb_1.jpg",
"source_video_url": "s3://mixpeek-storage/ns_123/obj_456/original.mp4",
"video_segment_url": "s3://mixpeek-storage/ns_123/obj_456/segments/segment_001.mp4",
"multimodal_extractor_v1_multimodal_embedding": [0.023, -0.041, "...1408 floats"],
"multimodal_extractor_v1_transcription_embedding": [0.018, -0.032, "...1024 floats"]
}
```
```json theme={null}
{
"start_time": 10.0,
"end_time": 20.0,
"start_frame": 20,
"end_frame": 40,
"fps": 2.0,
"source_fps": 29.97,
"duration": 120.5,
"transcription": "Welcome to today's lecture on machine learning fundamentals...",
"description": "Instructor standing at whiteboard, introducing ML concepts",
"ocr_text": "Machine Learning 101",
"thumbnail_url": "s3://mixpeek-storage/ns_123/thumbnails/thumb_1.jpg",
"source_video_url": "s3://mixpeek-storage/ns_123/obj_456/original.mp4",
"video_segment_url": "s3://mixpeek-storage/ns_123/obj_456/segments/segment_001.mp4",
"multimodal_extractor_v2_multimodal_embedding": [0.015, -0.038, "...3072 floats"],
"multimodal_extractor_v2_transcription_embedding": [0.018, -0.032, "...1024 floats"]
}
```
`fps` reflects the preprocessed video frame rate (e.g. 2.0 fps after downsampling). `source_fps` is the original video's native frame rate (e.g. 29.97). Use `source_fps` when you need to map timestamps back to exact frame numbers in the original source file.
## Parameters
### Video Splitting
| Parameter | Type | Default | Description |
| ---------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------- |
| `split_method` | string | `"time"` | Primary video splitting strategy: `time`, `scene`, or `silence` |
| `max_segment_duration` | float | `30.0` | Maximum seconds per segment. Scene/silence segments longer than this are subdivided. Set to `null` to disable |
#### Split Methods
**Fixed interval splitting** - Splits video into segments of equal duration.
| Parameter | Type | Default | Description |
| --------------------- | ------- | ------- | ------------------------------------ |
| `time_split_interval` | integer | `10` | Interval in seconds for each segment |
**Characteristics:**
* Predictable segment count: `video_duration / interval`
* Consistent chunk sizes for uniform processing
* May cut mid-sentence or mid-scene
**Best for:** General purpose, consistent chunking, when you need predictable segment counts
```json theme={null}
{
"split_method": "time",
"time_split_interval": 10
}
```
**Visual change detection** - Splits video when significant visual changes occur (shot changes, transitions).
| Parameter | Type | Default | Description |
| --------------------------- | ----- | ------- | ------------------------------- |
| `scene_detection_threshold` | float | `0.5` | Sensitivity threshold (0.0-1.0) |
**Threshold guide:**
* `0.3` - High sensitivity, detects subtle changes (more segments)
* `0.5` - Balanced (default)
* `0.7` - Low sensitivity, only major scene changes (fewer segments)
**Characteristics:**
* Variable segment count (typically 2-20 per minute)
* Segments align with visual content boundaries
* Better for content with distinct shots/scenes
**Best for:** Movies, dynamic content, shot changes, music videos, advertisements
```json theme={null}
{
"split_method": "scene",
"scene_detection_threshold": 0.5
}
```
**Audio pause detection** - Splits video at moments of silence or low audio.
| Parameter | Type | Default | Description |
| ---------------------- | ------- | ------- | ---------------------------------------------------- |
| `silence_db_threshold` | integer | `-40` | Decibel level below which audio is considered silent |
**Threshold guide:**
* `-50` dB - Detects very quiet moments (more segments)
* `-40` dB - Balanced (default)
* `-30` dB - Only detects near-silence (fewer segments)
**Characteristics:**
* Variable segment count (typically 5-30 per minute)
* Segments align with natural speech pauses
* Preserves complete sentences/thoughts
**Best for:** Lectures, presentations, conversations, podcasts, interviews
```json theme={null}
{
"split_method": "silence",
"silence_db_threshold": -40
}
```
#### Split Methods Comparison
| Method | Segments/Min | Predictability | Best For |
| --------- | ------------------ | -------------- | ----------------------------------- |
| `time` | 60 / interval\_sec | High | General purpose, batch processing |
| `scene` | Variable (2-20) | Low | Movies, ads, dynamic visual content |
| `silence` | Variable (5-30) | Medium | Lectures, podcasts, spoken content |
### Feature Extraction Parameters
| Parameter | Type | Default | Description |
| ----------------------------- | ------- | -------------------------- | ------------------------------------------------ |
| `run_transcription` | boolean | `true` (v1) / `false` (v2) | Run Whisper transcription on audio |
| `transcription_language` | string | `"en"` | Language for transcription |
| `run_transcription_embedding` | boolean | `true` (v1) / `false` (v2) | Generate E5 embeddings for transcriptions |
| `run_multimodal_embedding` | boolean | `true` | Generate multimodal embeddings |
| `run_video_description` | boolean | `false` | Generate AI descriptions (adds 1-2s per segment) |
| `run_ocr` | boolean | `false` | Extract text from video frames |
### Thumbnail Parameters
| Parameter | Type | Default | Description |
| ------------------- | ------- | ------- | --------------------------------- |
| `enable_thumbnails` | boolean | `true` | Generate thumbnail images |
| `use_cdn` | boolean | `false` | Use CloudFront CDN for thumbnails |
**CDN benefits**: Faster global delivery, permanent URLs, reduced bandwidth costs.
### v2-Only Parameters
These parameters are only available on `multimodal_extractor` v2:
| Parameter | Type | Default | Description |
| ----------------------- | ------- | ---------------------- | ------------------------------------------------------------------------------------------------------- |
| `output_dimensionality` | integer | `3072` | Embedding dimensions. Gemini Embedding 2 supports Matryoshka reduction: `3072` (full), `1536`, or `768` |
| `task_type` | string | `"RETRIEVAL_DOCUMENT"` | Embedding task hint: `RETRIEVAL_DOCUMENT`, `RETRIEVAL_QUERY`, `SEMANTIC_SIMILARITY`, `CLASSIFICATION` |
At query time, Mixpeek automatically uses `RETRIEVAL_QUERY` — you only need to set `task_type` at index time. The default `RETRIEVAL_DOCUMENT` is correct for most use cases.
### Embedding Task
When `run_transcription_embedding` is enabled, the E5 model generates text embeddings from transcribed audio. By default, these use `retrieval_document` for asymmetric search.
Set `embedding_task` at the **collection level**, not on the extractor. See [Collection Embedding Task](/docs/platform/processing#embedding-task) for full details and examples.
This only affects the E5 transcription embeddings. Vertex AI multimodal embeddings (v1) and Gemini Embedding 2 (v2) are not instruction-aware and ignore this parameter.
### Description Generation Parameters
| Parameter | Type | Default | Description |
| ------------------------------------- | ------- | ----------------------------------------- | ----------------------------------- |
| `description_prompt` | string | `"Describe the video segment in detail."` | Prompt for Gemini |
| `generation_config.temperature` | float | `0.7` | Randomness (higher = more creative) |
| `generation_config.max_output_tokens` | integer | `1024` | Maximum description length |
| `generation_config.top_p` | float | `0.8` | Nucleus sampling |
### LLM Structured Extraction
| Parameter | Type | Default | Description |
| ---------------- | ---------------- | ------- | ------------------------------- |
| `response_shape` | string \| object | `null` | Custom structured output schema |
**Natural Language Mode:**
```json theme={null}
{
"response_shape": "Extract product names, colors, materials, and aesthetic style labels from this fashion segment"
}
```
**JSON Schema Mode:**
```json theme={null}
{
"response_shape": {
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string" },
"category": { "type": "string" },
"visibility_percentage": { "type": "integer", "minimum": 0, "maximum": 100 }
}
}
},
"aesthetic": { "type": "string" }
}
}
}
```
## Configuration Examples
```json v1 — Video with Time-Based Splitting theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"field_passthrough": [
{ "source_path": "metadata.video_id" }
],
"parameters": {
"split_method": "time",
"time_split_interval": 10,
"run_transcription": true,
"run_multimodal_embedding": true,
"enable_thumbnails": true
}
}
}
```
```json v1 — Video with Scene Detection theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"parameters": {
"split_method": "scene",
"scene_detection_threshold": 0.5,
"run_transcription": true,
"run_video_description": true,
"enable_thumbnails": true
}
}
}
```
```json v1 — Lecture Video with Silence Splitting theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"video": "lecture_url"
},
"parameters": {
"split_method": "silence",
"silence_db_threshold": -40,
"run_transcription": true,
"transcription_language": "en",
"run_ocr": true,
"enable_thumbnails": true
}
}
}
```
```json v2 — Video with Gemini Embedding 2 theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v2",
"input_mappings": {
"video": "video_url"
},
"parameters": {
"split_method": "time",
"time_split_interval": 10,
"run_multimodal_embedding": true,
"output_dimensionality": 3072,
"enable_thumbnails": true
}
}
}
```
```json v2 — Compact Embeddings (768D) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v2",
"input_mappings": {
"video": "video_url"
},
"parameters": {
"split_method": "scene",
"scene_detection_threshold": 0.5,
"run_multimodal_embedding": true,
"output_dimensionality": 768,
"run_transcription": true,
"run_transcription_embedding": true,
"enable_thumbnails": true
}
}
}
```
```json Image Embedding (v1 or v2) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"image": "image_url"
},
"field_passthrough": [
{ "source_path": "metadata.product_id" }
],
"parameters": {
"run_multimodal_embedding": true,
"enable_thumbnails": true
}
}
}
```
```json Text Embedding (Cross-Modal Search) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"text": "product_description"
},
"parameters": {
"run_multimodal_embedding": true
}
}
}
```
```json v1 — Full Extraction with All Features theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"parameters": {
"split_method": "scene",
"scene_detection_threshold": 0.5,
"run_transcription": true,
"run_transcription_embedding": true,
"run_multimodal_embedding": true,
"run_video_description": true,
"run_ocr": true,
"enable_thumbnails": true,
"use_cdn": true,
"description_prompt": "Describe what is happening in this video segment, including any visible products, people, and actions.",
"generation_config": {
"temperature": 0.7,
"max_output_tokens": 1024
}
}
}
}
```
```json Fashion/E-commerce with Structured Extraction theme={null}
{
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": {
"video": "fashion_video_url"
},
"parameters": {
"split_method": "scene",
"run_multimodal_embedding": true,
"run_video_description": true,
"response_shape": {
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string" },
"category": { "type": "string" },
"color": { "type": "string" },
"visibility_percentage": { "type": "integer" }
}
}
},
"aesthetic": { "type": "string" },
"setting": { "type": "string" }
}
}
}
}
}
```
## Performance & Costs
### Processing Speed
| Content Type | Speed |
| ------------ | --------------------------------------------- |
| Video | 0.5-2x realtime (depends on features enabled) |
| Image | \< 1 second |
| Text | \< 100ms |
**Example**: 10-minute video → 5-20 minutes processing time
| Feature | Latency per Segment |
| ---------------- | --------------------------- |
| Transcription | \~200ms per second of audio |
| Visual embedding | \~50ms |
| OCR | \~300ms |
| Description | \~2s |
### Cost Estimates (per minute of video)
| Configuration | Cost |
| ---------------------------------------- | ------ |
| **Minimal** (transcription + embeddings) | \$0.01 |
| **Standard** (+ OCR) | \$0.05 |
| **Full** (+ descriptions) | \$0.15 |
**Images**: $0.001 per image **Text**: $0.0001 per query
## Vector Indexes
### Multimodal Embedding
| Property | Value |
| -------------------- | --------------------------------------------------------------- |
| **Feature URI** | `mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding` |
| **Index name** | `multimodal_extractor_v1_multimodal_embedding` |
| **Dimensions** | 1408 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `vertex_multimodal_embedding` |
| **Supported inputs** | video, text, image |
### Transcription Embedding
| Property | Value |
| -------------------- | --------------------------------------------------------------------- |
| **Feature URI** | `mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1` |
| **Index name** | `multimodal_extractor_v1_transcription_embedding` |
| **Dimensions** | 1024 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `multilingual_e5_large_instruct_v1` |
| **Supported inputs** | text, string |
In retrievers, reference these by their **Feature URI** — the output name is the model name, **not** the `multimodal_extractor_v1_*` index name.
### Multimodal Embedding
| Property | Value |
| -------------------- | ---------------------------------------------- |
| **Index name** | `multimodal_extractor_v2_multimodal_embedding` |
| **Dimensions** | 3072 (configurable: 1536, 768) |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `google/gemini-embedding-2` |
| **Supported inputs** | video, text, image, audio |
### Transcription Embedding
| Property | Value |
| -------------------- | ------------------------------------------------- |
| **Index name** | `multimodal_extractor_v2_transcription_embedding` |
| **Dimensions** | 1024 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `intfloat/multilingual-e5-large-instruct` |
| **Supported inputs** | text, string |
## Choosing v1 vs v2
| Consideration | v1 | v2 |
| ----------------------- | ---------------------- | ------------------------------------------ |
| **Embedding quality** | Good | Better (natively multimodal) |
| **Dimensions** | 1408 (fixed) | 3072, 1536, or 768 (configurable) |
| **Storage per vector** | 5.5 KB | 12 KB (3072D), 6 KB (1536D), 3 KB (768D) |
| **Audio input support** | Via transcription only | Native audio embedding |
| **Matryoshka support** | No | Yes — reduce dimensions without reindexing |
| **Stability** | Production-proven | Newer |
**Recommendation:** Use **v2** for new projects. Use **v1** if you have existing collections and don't need higher dimensions or native audio embedding.
## Limitations
* **Video duration**: Recommend \< 2 hours for optimal processing
* **Resolution**: 8K+ videos should be downsampled
* **Real-time**: Not suitable for live streaming
* **Short videos**: \< 5 second videos have disproportionate overhead
* **Audio quality**: Transcription accuracy depends on audio clarity
* **OCR/Description**: Add significant processing time, enable only when needed
## Collection-to-Collection Pipelines
The `video_segment_url` output enables decomposition chains:
1. **Initial collection**: Time-based segments (5s intervals)
2. **Downstream collection**: Scene detection within each segment
3. **Final collection**: Enhanced processing with different models
```json theme={null}
{
"input_mappings": {
"video": "video_segment_url"
}
}
```
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Gemini Multifile Extractor](/docs/processing/extractors/gemini-multifile) — Embed multiple files per object into one vector
* [Passthrough Extractor](/docs/processing/extractors/passthrough)
* [Text Extractor](/docs/processing/extractors/text)
# Passthrough Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/passthrough
Copy source fields without processing for metadata propagation and vector passthrough
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["storage_only"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
![Passthrough extractor showing direct field copy from source to output]()
The passthrough extractor copies source fields without any ML processing. By default, all source fields are included. Use `field_passthrough` to specify specific fields or `include_all_source_fields` to control behavior. Supports vector passthrough from collection to collection.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/passthrough\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/passthrough_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## When to Use
| Use Case | Description |
| -------------------------------------- | --------------------------------------------------------------------------------------- |
| **Metadata propagation** | Pass metadata fields from bucket objects to collection documents without transformation |
| **Vector passthrough** | Copy pre-computed embeddings from one collection to another |
| **Schema normalization** | Select specific fields to include in the output schema |
| **Collection-to-collection pipelines** | Route data through multi-tier processing without re-embedding |
## When NOT to Use
* When you need to generate embeddings → Use [text\_extractor](/docs/processing/extractors/text) or [multimodal\_extractor](/docs/processing/extractors/multimodal)
* When you need to transform or enrich data → Use extractors with ML models
* When you need to decompose content (chunking, video splitting) → Use appropriate extractors
## Input Schema
The passthrough extractor accepts **any input type** and copies fields as-is.
```json theme={null}
{
"type": "object",
"properties": {},
"description": "Accepts any fields - no required inputs"
}
```
## Output Schema
The output mirrors the input based on configuration:
| Configuration | Behavior |
| ------------------------------------------- | ---------------------------------- |
| Default (`include_all_source_fields: true`) | All source fields copied to output |
| `field_passthrough` specified | Only listed fields copied |
| `include_all_source_fields: false` | Must specify `field_passthrough` |
## Parameters
The passthrough extractor has **no required parameters**. Configuration is handled through `field_passthrough` and `input_mappings` at the collection level.
## Configuration Examples
```json Copy All Fields theme={null}
{
"feature_extractor": {
"feature_extractor_name": "passthrough_extractor",
"version": "v1",
"input_mappings": {},
"field_passthrough": [],
"parameters": {}
}
}
```
```json Select Specific Fields theme={null}
{
"feature_extractor": {
"feature_extractor_name": "passthrough_extractor",
"version": "v1",
"input_mappings": {},
"field_passthrough": [
{ "source_path": "metadata.category" },
{ "source_path": "metadata.brand" },
{ "source_path": "metadata.price" }
],
"parameters": {}
}
}
```
```json Vector Passthrough theme={null}
{
"feature_extractor": {
"feature_extractor_name": "passthrough_extractor",
"version": "v1",
"input_mappings": {},
"field_passthrough": [
{ "source_path": "text_extractor_v1_embedding" },
{ "source_path": "metadata.product_id" }
],
"parameters": {}
}
}
```
```json Rename Fields During Passthrough theme={null}
{
"feature_extractor": {
"feature_extractor_name": "passthrough_extractor",
"version": "v1",
"input_mappings": {},
"field_passthrough": [
{ "source_path": "metadata.old_name", "target_path": "new_name" },
{ "source_path": "payload.description", "target_path": "content" }
],
"parameters": {}
}
}
```
## Performance & Costs
| Metric | Value |
| ------------------ | ---------------------------- |
| **Latency** | \< 1ms |
| **Cost** | Free |
| **GPU Required** | No |
| **Max Throughput** | Unlimited (no ML processing) |
## Vector Indexes
The passthrough extractor creates **no vector indexes**. If you pass through existing embeddings, they retain their original index configuration from the source collection.
## Best Practices
1. **Use for multi-tier pipelines** – When downstream collections need upstream data without reprocessing
2. **Minimize field selection** – Only pass through fields you need to reduce storage and query overhead
3. **Preserve lineage** – The passthrough extractor maintains `root_object_id` and `source_collection_id` for data lineage tracking
4. **Combine with other extractors** – Use passthrough fields alongside ML extractors in the same collection to include metadata with generated features
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Text Extractor](/docs/processing/extractors/text)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
# Scrolling Text Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/scrolling-text
Extract scrolling/marquee text from video using phase-correlation band detection, panoramic stitching, and VLM OCR
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["onscreen_text"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Create a managed namespace and run this pipeline on your own files — recover scrolling and static on-screen text, OCR it, and make it searchable in minutes.
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
The scrolling text extractor recovers **scrolling or marquee text** from video — tickers, lower-third banners, end credits, and legal disclaimers — that no single frame ever shows in full. It detects scrolling bands via phase correlation, stitches frames panorama-style to reconstruct the complete text, then OCRs the panorama with a vision language model (Gemini). Output is payload-only (no vector); pair it with `text_extractor` if you need semantic search over the recovered text.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/scrolling\_text\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/scrolling_text_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Sample frames** — extract frames at `fps` frames per second.
2. **Phase correlation** — scan `strip_height`-pixel strips to measure per-frame pixel shift and detect motion.
3. **Classify bands** — a band counts as scrolling when shift exceeds `min_shift_px` and at least `consistency_ratio` of frame pairs agree.
4. **Crop** — crop each detected band with `pad` pixels of padding above and below.
5. **Stitch** — reconstruct the full scrolling content as a panorama image per band.
6. **VLM OCR** — read the panorama with a vision language model (Gemini).
7. **Output** — combined text plus per-band metadata (axis, direction, shift).
## When to Use
| Use Case | Description |
| -------------------------- | ---------------------------------------------------------------- |
| **News tickers** | Recover the full crawl text from a horizontally scrolling ticker |
| **End credits** | Capture vertically scrolling credit rolls |
| **Compliance disclaimers** | Extract fast-scrolling legal/disclaimer banners for audit |
| **Sports/finance banners** | Read scrolling score or price strips |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------------------- | ----------------------------------------------------------- |
| Static on-screen text / captions | `text_extractor` on transcription, or a frame OCR extractor |
| Spoken-word transcription | A transcription/audio extractor |
| Semantic search over recovered text | Chain `text_extractor` on the `scrolling_text` field |
| Non-video inputs | This extractor is video-only |
## Input Schema
| Field | Type | Required | Description |
| ------- | ------ | -------- | --------------------------------------------------------------- |
| `video` | string | **Yes** | URL or path to the video file. Populated from `input_mappings`. |
```json theme={null}
{
"video": "s3://my-bucket/clips/newscast.mp4"
}
```
Supported input types: **VIDEO**.
## Output Schema
| Field | Type | Description |
| ---------------- | ----------------- | ---------------------------------------------------------------- |
| `scrolling_text` | string \| null | Combined, deduplicated text from all detected scrolling bands |
| `scroll_bands` | object\[] \| null | Per-band details: `axis`, `direction`, `shift_per_frame`, `text` |
| `bands_detected` | integer \| null | Number of scrolling text bands detected in the video |
```json theme={null}
{
"scrolling_text": "BREAKING: Markets rally as inflation cools ...",
"bands_detected": 1,
"scroll_bands": [
{
"axis": "horizontal",
"direction": "right_to_left",
"shift_per_frame": 6.4,
"text": "BREAKING: Markets rally as inflation cools ..."
}
]
}
```
## Parameters
| Parameter | Type | Default | Range | Description |
| ------------------- | ------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------ |
| `fps` | float | `5.0` | 1.0–30.0 | Frame sampling rate. Higher values improve detection for fast-scrolling text but increase processing time |
| `strip_height` | integer | `40` | 10–200 | Height (px) of each scanning strip used for phase correlation. Should roughly match the scrolling text band height |
| `min_shift_px` | float | `2.0` | 0.5–20.0 | Minimum per-frame pixel shift to consider a strip 'scrolling'. Lower detects slower text; higher filters noise |
| `consistency_ratio` | float | `0.6` | 0.3–1.0 | Fraction of frame pairs that must show consistent shift for a band to count as scrolling (0.6 = 60%) |
| `pad` | integer | `8` | 0–50 | Pixel padding above/below the detected band when cropping for stitching |
## Configuration Examples
```json Default Detection theme={null}
{
"feature_extractor": {
"feature_extractor_name": "scrolling_text_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"parameters": {}
}
}
```
```json Fast Ticker (high fps, thin band) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "scrolling_text_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"parameters": {
"fps": 15.0,
"strip_height": 30,
"min_shift_px": 5.0
}
}
}
```
```json Slow Credits Roll (tall band, sensitive) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "scrolling_text_extractor",
"version": "v1",
"input_mappings": {
"video": "video_url"
},
"parameters": {
"fps": 3.0,
"strip_height": 120,
"min_shift_px": 1.0,
"consistency_ratio": 0.5
}
}
}
```
## Performance & Costs
| Metric | Value |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cost** | Billed per video minute (frame extraction + stitching + VLM OCR) — see [Billing & Pricing](/docs/platform/billing); rates come from `GET /v1/billing/pricing` |
| **External API** | Google Gemini (VLM OCR) |
| **Tradeoff** | Higher `fps` improves fast-scroll accuracy at the cost of processing time |
## Vector Index
This extractor produces **payload-only output** — no vector index. The recovered text lives in the `scrolling_text` field. To make it semantically searchable, run `text_extractor` against `scrolling_text`.
## Limitations
* **Video only**: Accepts video inputs exclusively.
* **No embedding**: Output is payload-only; semantic search requires chaining a text extractor.
* **Band-height sensitivity**: `strip_height` should approximate the actual band height for reliable detection.
* **VLM dependency**: OCR quality depends on Gemini VLM availability and panorama clarity.
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Text Extractor](/docs/processing/extractors/text)
* [Universal Extractor](/docs/processing/extractors/universal)
# Text Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/text
Dense multilingual (E5-Large) text embeddings for semantic and cross-lingual search — query in one language, match content in 100+ others
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["text_search"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
![Text extractor pipeline showing chunking, E5-Large embedding, and optional LLM extraction]()
The text extractor generates dense vector embeddings from text using the E5-Large multilingual model. Optimized for semantic search, RAG applications, and general-purpose text retrieval. Supports text chunking/decomposition with multiple splitting strategies. Fast (5ms/doc), cost-effective (free), and supports 100+ languages.
**Cross-lingual retrieval.** Because all 100+ languages share one embedding space, a query in one language matches content in **any** language — an English query (e.g. "foreign language", "translate what they said") retrieves Farsi, Mandarin, or Spanish text with **no translation step and no per-language index**. Note that this is *retrieval*, not translation: matched text is returned in its original language. To render it in the reader's language, add an `llm_enrich` translate stage — see the [Search Across Languages recipe](/docs/retrieval/cookbook#search-across-languages-cross-lingual).
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/text\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/text_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Filter Dataset** (if collection\_id provided)
* Filter to specified collection
2. **Apply Input Mappings**
* Resolve text field from source (e.g., `transcription`, `content`, `data`)
3. **Text Chunking** (conditional: if `split_by != "none"`)
* Split by: characters, words, sentences, paragraphs, or pages
* Configure `chunk_size` and `chunk_overlap`
* Each chunk becomes a separate document
4. **E5 Text Embedding Generation**
* Multilingual E5-Large model (1024D)
* L2 normalized vectors
* Batch size: 4,096 texts
5. **Output**
* Text documents with embeddings
* One document per input (or per chunk if chunking enabled)
## When to Use
| Use Case | Description |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Product search** | Search products by natural language descriptions |
| **FAQ matching** | Match user questions to knowledge base articles |
| **Document retrieval** | Find relevant documents from large corpora |
| **Content discovery** | Recommend similar content based on semantic similarity |
| **RAG chunking** | Split documents into chunks for retrieval-augmented generation |
| **Multi-language / cross-lingual search** | Query in one language, match content in another (foreign-language, translate-intent queries) across 100+ languages with a single model |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------------------------------ | ----------------------------------------------------------------------------------- |
| Exact phrase / keyword matching (SKUs, codes, names) | Add a [lexical (BM25) search](/docs/retrieval/stages/feature-search#lexical-bm25-search) |
| Keyword-heavy queries (e.g. "iPhone 15 Pro Max 256GB") | Lexical (BM25) search alongside the dense one |
| Critical technical terms or short texts (1–5 words) | Lexical (BM25) search |
Dense embeddings and exact-keyword matching are complementary. Keep `text_extractor` for semantic recall and add a `lexical: true` search over a `text` index for exact tokens — fuse them with `rrf`. See [Lexical (BM25) Search](/docs/retrieval/stages/feature-search#lexical-bm25-search).
## Input Schema
| Field | Type | Required | Description |
| ------ | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `text` | string | **Yes** | Text content to process. Recommended: 10-400 words for optimal quality. Maximum: 512 tokens (\~400 words), longer text is truncated. |
```json theme={null}
{
"text": "Premium wireless Bluetooth headphones with active noise cancellation, 30-hour battery life, and premium sound quality."
}
```
**Input Examples:**
| Type | Example |
| ------------------- | ---------------------------------------------------------------------------- |
| Product description | "Premium wireless Bluetooth headphones with active noise cancellation" |
| FAQ question | "How do I reset my password if I forgot it?" |
| Article paragraph | "Machine learning models have revolutionized natural language processing..." |
| User query | "best restaurants near Times Square" |
## Output Schema
| Field | Type | Description |
| ----------------------------- | ------------ | ----------------------------------------------- |
| `text` | string | The processed text content (full text or chunk) |
| `text_extractor_v1_embedding` | float\[1024] | Dense vector embedding, L2 normalized |
```json theme={null}
{
"text": "Premium wireless Bluetooth headphones with active noise cancellation",
"text_extractor_v1_embedding": [0.023, -0.041, 0.018, ...]
}
```
When chunking is enabled, each chunk becomes a separate document with tracking metadata stored in `metadata` (not in the document payload):
* `chunk_index` – Position of this chunk in the original document
* `chunk_text` – The text content of this chunk
* `total_chunks` – Total number of chunks from the source
## Parameters
### Chunking Parameters
| Parameter | Type | Default | Description |
| --------------- | ------- | -------- | ------------------------------------------------------- |
| `split_by` | string | `"none"` | Strategy for splitting text into chunks |
| `chunk_size` | integer | `1000` | Target size for each chunk (units depend on `split_by`) |
| `chunk_overlap` | integer | `0` | Number of units to overlap between consecutive chunks |
#### Split Strategies
| Strategy | Description | Best For |
| ------------ | ------------------------------------ | ------------------------------------------------- |
| `characters` | Split by character count | Uniform sizes, quick testing |
| `words` | Split by word boundaries | General text, preserves words |
| `sentences` | Split by sentence boundaries | Q\&A, precise retrieval, preserves semantic units |
| `paragraphs` | Split by paragraph (double newlines) | Articles, documentation, natural structure |
| `pages` | Split by page breaks | PDFs, paginated documents |
| `none` | No splitting (default) | Short texts \< 400 words |
**Recommended chunk sizes:**
* `characters`: 500-2000
* `words`: 100-400
* `sentences`: 3-10
* `paragraphs`: 1-3
* `pages`: 1
**Chunk overlap:** 10-20% of `chunk_size` helps preserve context across boundaries. Example: `chunk_size: 1000`, `chunk_overlap: 100-200`.
### Embedding Model
| Parameter | Type | Default | Description |
| ----------------- | ------ | ---------------------- | --------------------------------------------------------------------------------- |
| `embedding_model` | string | *current TEXT default* | Override the embedding model via the [model registry](/docs/processing/model-registry) |
The text extractor resolves its model through the central embedding registry. Leave unset to use the current TEXT modality default (`intfloat_e5_large_instruct_v1`, 1024d) — the registry swaps hot when a new frontier text model ships, so existing collections pick it up without a code change.
Dimensions are locked at namespace creation. Switching `embedding_model` on an existing namespace requires a migration since the vector index dimensionality is fixed.
### Embedding Task
Instruction-aware embedding models (E5, Gemini) use a **task hint** to optimize the embedding for a specific downstream use case. By default, all extractors use `retrieval_document` at ingestion time, which produces embeddings optimized for asymmetric search (queries find documents).
Set `embedding_task` at the **collection level**, not on the extractor. See [Collection Embedding Task](/docs/platform/processing#embedding-task) for full details and examples.
| Task | Use Case | Effect on E5 | Effect on Gemini |
| --------------------- | ------------------------------------------------ | ---------------------- | ---------------------------------------- |
| `retrieval_document` | **Default.** Search: find documents from queries | Prepends `"passage: "` | Instructs "represent for retrieval" |
| `retrieval_query` | Rare at index time. Query-side is automatic | Prepends `"query: "` | Instructs "represent this query" |
| `semantic_similarity` | Symmetric comparison (deduplication, matching) | Prepends `"query: "` | Instructs "represent for similarity" |
| `classification` | Document categorization pipelines | Prepends `"query: "` | Instructs "represent for classification" |
| `clustering` | Grouping documents into clusters | Prepends `"query: "` | Not applied |
You almost never need to set this. The default `retrieval_document` is correct for search, and at query time Mixpeek automatically uses `retrieval_query`. Only override if your collection is primarily used for clustering, classification, or symmetric similarity — not retrieval.
Non-instruction-aware models (SigLIP, CLIP, Vertex multimodal) ignore this parameter.
### LLM Structured Extraction Parameters
| Parameter | Type | Default | Description |
| ---------------- | ---------------- | ------- | ---------------------------------------------------- |
| `response_shape` | string \| object | `null` | Define custom structured output using LLM extraction |
| `llm_provider` | string | `null` | LLM provider: `openai`, `google`, `anthropic` |
| `llm_model` | string | `null` | Specific model for extraction |
#### response\_shape Modes
**Natural Language Mode (string):**
```json theme={null}
{
"response_shape": "Extract key entities, sentiment (positive/negative/neutral), and main topics from the text"
}
```
The service automatically infers JSON schema from your description.
**JSON Schema Mode (object):**
```json theme={null}
{
"response_shape": {
"type": "object",
"properties": {
"sentiment": {
"type": "string",
"enum": ["positive", "negative", "neutral"]
},
"entities": {
"type": "array",
"items": { "type": "string" }
},
"topics": {
"type": "array",
"items": { "type": "string" },
"maxItems": 5
}
},
"required": ["sentiment"]
}
}
```
#### LLM Provider & Model Options
| Provider | Example Models |
| ----------- | --------------------------------------------------------------------------------- |
| `openai` | `gpt-4o-mini-2024-07-18` (cost-effective), `gpt-4o-2024-08-06` (best quality) |
| `google` | `gemini-2.5-flash` (fastest), `gemini-1.5-flash-001` |
| `anthropic` | `claude-3-5-haiku-20241022` (fast), `claude-3-5-sonnet-20241022` (best reasoning) |
## Configuration Examples
```json Basic Embedding (No Chunking) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {
"text": "description"
},
"field_passthrough": [
{ "source_path": "metadata.product_id" }
],
"parameters": {}
}
}
```
```json Sentence Chunking for RAG theme={null}
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {
"text": "document_content"
},
"field_passthrough": [
{ "source_path": "metadata.document_id" }
],
"parameters": {
"split_by": "sentences",
"chunk_size": 5,
"chunk_overlap": 1
}
}
}
```
```json Paragraph Chunking for Articles theme={null}
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {
"text": "article_body"
},
"field_passthrough": [
{ "source_path": "metadata.title" },
{ "source_path": "metadata.author" }
],
"parameters": {
"split_by": "paragraphs",
"chunk_size": 2,
"chunk_overlap": 0
}
}
}
```
```json Word-Level Chunking with Overlap theme={null}
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {
"text": "content"
},
"parameters": {
"split_by": "words",
"chunk_size": 300,
"chunk_overlap": 50
}
}
}
```
```json LLM Extraction (Natural Language) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {
"text": "review_text"
},
"parameters": {
"response_shape": "Extract sentiment (positive/negative/neutral), key product features mentioned, and overall rating impression",
"llm_provider": "openai",
"llm_model": "gpt-4o-mini-2024-07-18"
}
}
}
```
```json LLM Extraction (JSON Schema) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {
"text": "document"
},
"parameters": {
"response_shape": {
"type": "object",
"properties": {
"entities": {
"type": "array",
"items": { "type": "string" },
"description": "Named entities mentioned"
},
"sentiment": {
"type": "string",
"enum": ["positive", "negative", "neutral"]
},
"topics": {
"type": "array",
"items": { "type": "string" },
"maxItems": 5
}
},
"required": ["sentiment"]
},
"llm_provider": "anthropic",
"llm_model": "claude-3-5-haiku-20241022"
}
}
}
```
## Performance & Costs
| Metric | Value |
| --------------------- | --------------------------------------- |
| **Embedding latency** | \~5ms per document (batched: \~2ms/doc) |
| **Query latency** | 5-10ms for top-100 results |
| **Cost** | Free (self-hosted E5-Large) |
| **GPU required** | No (but 5-10x faster with GPU) |
| **Memory** | \~4GB per 1M documents |
| **Index build** | \~1 hour per 10M documents |
**LLM extraction** adds cost and latency based on provider pricing. Only use when structured extraction is needed.
## Dense Embeddings vs Lexical (BM25)
`text_extractor` produces **dense** embeddings — great for semantic recall, weak on exact tokens. For exact-keyword precision, pair it with a **lexical (BM25)** search (the `lexical: true` option on a `feature_search` stage, backed by a `text` payload index — not a separate extractor).
| Dimension | Dense (`text_extractor`) | Lexical (BM25, `lexical: true`) |
| ------------------- | ------------------------ | --------------------------------- |
| **Matches** | Meaning / paraphrase | Exact tokens, SKUs, codes, prices |
| **Semantic recall** | Excellent | Poor |
| **Exact matching** | Poor | Excellent |
| **Multi-language** | Excellent (E5-Large) | Good (token-based) |
| **Requires** | Embedding index | `text` payload index |
The strongest setup is **hybrid**: run a dense search and a lexical search in the same `feature_search` stage and fuse with `rrf`. See [Lexical (BM25) Search](/docs/retrieval/stages/feature-search#lexical-bm25-search).
## Vector Index
| Property | Value |
| ------------------- | --------------------------------------------------------------- |
| **Feature URI** | `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1` |
| **Index name** | `text_extractor_v1_embedding` |
| **Dimensions** | 1024 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Inference model** | `multilingual_e5_large_instruct_v1` |
| **Normalization** | L2 normalized |
In retrievers, reference this feature by its **Feature URI** above (the output name is `multilingual_e5_large_instruct_v1`, **not** the index name `text_extractor_v1_embedding`).
## Limitations
* **Token limit**: 512 tokens (\~400 words). Longer text is automatically truncated.
* **Exact phrases**: Cannot reliably match exact phrases or technical terms.
* **Domain jargon**: Struggles with very domain-specific jargon or acronyms.
* **Terminology variance**: May miss documents that use different terminology for the same concept.
* **Short texts**: Less effective for very short texts (1-5 words) where lexical matching is sufficient.
* **Keyword-heavy queries**: Less effective for queries like "iPhone 15 Pro Max 256GB".
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Passthrough Extractor](/docs/processing/extractors/passthrough)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
# Universal Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/universal
All-in-one multimodal extractor — image, video, audio, and documents — producing 3072-d Gemini embeddings plus text descriptions, OCR, and transcription
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["video_search"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
The universal extractor is an all-in-one feature extractor that handles **image, video, audio, and documents** through Google's Gemini APIs. It produces a single 3072-dimensional embedding (Gemini Embedding 2) per object alongside rich text extraction — AI-generated descriptions, OCR for images and documents, and transcription for audio and video. It runs on Celery (not Ray) for zero cluster-startup latency, making it a fast path for mixed-modality corpora.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/universal\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/universal_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Resolve input** — apply `input_mappings` to get the file URL/path from the source object (`content` field).
2. **Detect modality** — classify the object as image, video, audio, or document.
3. **Segment (if needed)** — video is processed in up to `max_video_segments` 30s segments; documents up to `max_document_pages` pages.
4. **Gemini embedding** — generate a 3072-d Gemini Embedding 2 vector (`output_dimensionality` configurable 256–3072).
5. **Text extraction** (if `extract_text`) — OCR for images/documents, transcription for audio/video.
6. **Description** (if `generate_description`) — Gemini vision/understanding produces a natural-language description.
7. **Output** — one document per object (or per segment/page for chunked content).
## When to Use
| Use Case | Description |
| -------------------------- | ------------------------------------------------------------------------------------------------ |
| **Mixed-modality corpora** | A single bucket containing images, video, audio, and PDFs you want searchable with one extractor |
| **Fast onboarding** | Celery fast-path avoids Ray cluster startup, so small batches return quickly |
| **Cross-modal search** | One shared 3072-d embedding space across all four modalities |
| **Rich metadata** | Need descriptions, OCR text, and transcription alongside the vector |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------------------ | ----------------------------------------------------------------- |
| High-volume single-modality at lowest cost | Modality-specific extractor (`text_extractor`, `image_extractor`) |
| Audio fingerprinting / sound-mark matching | `audio_fingerprint_extractor` |
| Spatial/layout document analysis | `document_graph_extractor` |
| Self-hosted, no external API calls | `text_extractor` / `image_extractor` |
## Input Schema
| Field | Type | Required | Description |
| --------- | ------ | -------- | -------------------------------------------------------------------- |
| `content` | string | **Yes** | URL or path to the file to process. Populated from `input_mappings`. |
```json theme={null}
{
"content": "s3://my-bucket/assets/report.pdf"
}
```
Supported input types: **IMAGE, VIDEO, AUDIO, PDF, TEXT, STRING**.
## Output Schema
| Field | Type | Description |
| ---------------------------------- | --------------- | ----------------------------------------------------------- |
| `universal_extractor_v1_embedding` | float\[3072] | Gemini Embedding 2 vector for the content |
| `modality` | string | Detected modality: `image`, `video`, `audio`, or `document` |
| `text` | string \| null | Extracted text (OCR, transcription, or document text) |
| `description` | string \| null | AI-generated description of the content |
| `segment_index` | integer \| null | Segment index (chunked video/audio/documents) |
| `segment_total` | integer \| null | Total segments for this source object |
| `page_number` | integer \| null | Page number (documents only) |
| `start_time_s` / `end_time_s` | float \| null | Segment start/end time in seconds (video/audio) |
| `duration_s` | float \| null | Total file duration in seconds (video/audio) |
```json theme={null}
{
"universal_extractor_v1_embedding": [0.012, -0.034, 0.008, ...],
"modality": "document",
"text": "Quarterly revenue grew 12% year over year...",
"description": "A financial report page with a revenue bar chart",
"page_number": 1,
"segment_total": 12
}
```
## Parameters
| Parameter | Type | Default | Range | Description |
| ----------------------- | ------- | ---------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------- |
| `output_dimensionality` | integer | `3072` | 256–3072 | Output embedding dimensions (Gemini Embedding 2 supports 256–3072) |
| `task_type` | string | `"RETRIEVAL_DOCUMENT"` | — | Embedding intent for Gemini Embedding 2. Common values: `RETRIEVAL_DOCUMENT`, `RETRIEVAL_QUERY`, `SEMANTIC_SIMILARITY` |
| `generate_description` | boolean | `true` | — | Generate a text description via Gemini vision/understanding |
| `extract_text` | boolean | `true` | — | Extract text (OCR for images/docs, transcription for audio/video) |
| `max_video_segments` | integer | `10` | 1–50 | Maximum number of 30s segments to process for video files |
| `max_document_pages` | integer | `50` | 1–200 | Maximum number of pages to process for document files |
| `max_file_download_mb` | integer | `500` | 1–1024 | Maximum file download size (MB) for Celery fast-path processing |
| `max_concurrency` | integer | `4` | 1–32 | Maximum per-task object concurrency for Celery fast-path processing |
Dimensions are locked at namespace creation. Switching `output_dimensionality` on an existing namespace requires a migration since the vector index dimensionality is fixed.
## Configuration Examples
```json All-in-One Defaults theme={null}
{
"feature_extractor": {
"feature_extractor_name": "universal_extractor",
"version": "v1",
"input_mappings": {
"content": "file_url"
},
"parameters": {}
}
}
```
```json Embeddings Only (No Text/Description) theme={null}
{
"feature_extractor": {
"feature_extractor_name": "universal_extractor",
"version": "v1",
"input_mappings": {
"content": "file_url"
},
"parameters": {
"generate_description": false,
"extract_text": false
}
}
}
```
```json Compact Embeddings for Clustering theme={null}
{
"feature_extractor": {
"feature_extractor_name": "universal_extractor",
"version": "v1",
"input_mappings": {
"content": "file_url"
},
"parameters": {
"output_dimensionality": 768,
"task_type": "CLUSTERING"
}
}
}
```
```json Long-Form Video & Documents theme={null}
{
"feature_extractor": {
"feature_extractor_name": "universal_extractor",
"version": "v1",
"input_mappings": {
"content": "file_url"
},
"parameters": {
"max_video_segments": 30,
"max_document_pages": 200
}
}
}
```
## Performance & Costs
| Metric | Value |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Compute** | Celery fast-path (no Ray cluster startup) |
| **Cost** | See [Billing & Pricing](/docs/platform/billing) — rates come from `GET /v1/billing/pricing`. One charge covers all Gemini API calls (embedding, description, OCR/transcription) |
| **External API** | Google Gemini (embedding + vision/understanding) |
| **Max download** | 500 MB per object (configurable to 1024 MB) |
## Vector Index
| Property | Value |
| ------------------- | ---------------------------------- |
| **Index name** | `universal_extractor_v1_embedding` |
| **Dimensions** | 3072 (configurable 256–3072) |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `google/gemini-embedding-2` |
## Limitations
* **External dependency**: Requires Google Gemini API availability; subject to its rate limits.
* **Per-object cost**: Higher per-object cost than self-hosted single-modality extractors.
* **Segment/page caps**: Video beyond `max_video_segments` and documents beyond `max_document_pages` are truncated.
* **Download ceiling**: Files larger than `max_file_download_mb` are skipped on the Celery fast-path.
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
* [Gemini Multifile Extractor](/docs/processing/extractors/gemini-multifile)
* [Audio Fingerprint Extractor](/docs/processing/extractors/audio-fingerprint)
# Web Scraper Extractor
Source: https://docs.mixpeek.com/docs/processing/extractors/web-scraper
Recursive website crawling with multimodal content extraction and semantic embeddings
Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](/docs/processing/features). This pipeline is selected with `features: ["web_crawl"]`. Existing `feature_extractor` configs keep working; see the [migration guide](/docs/processing/extractor-migration).
Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.
![Web scraper extractor pipeline showing crawling, content extraction, chunking, and multimodal embeddings]()
The web scraper extractor recursively crawls websites to extract multimodal content with semantic embeddings. Automatically discovers and extracts text, code blocks, images, and asset links from web pages. Each extracted document receives E5-Large text embeddings (1024D) for semantic search, Jina Code embeddings (768D) for code snippets, and optional SigLIP visual embeddings (768D) for images. Supports JavaScript-rendered SPAs, includes resilience features like retry logic, proxy rotation, and captcha detection.
View extractor details at [api.mixpeek.com/v1/collections/features/extractors/web\_scraper\_v1](https://api.mixpeek.com/v1/collections/features/extractors/web_scraper_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.
## Pipeline Steps
1. **Filter Dataset** (if collection\_id provided)
* Filter to specified collection
2. **Crawl Configuration & Setup**
* Parse seed URL and configure crawl parameters
* Set up URL filtering rules, rendering strategy, resilience options
3. **Recursive Web Crawling**
* BFS-based link traversal with depth limit
* JavaScript rendering support (auto-detect or explicit)
* URL filtering (include/exclude patterns)
* Resilience: retry logic, proxy rotation, captcha detection
4. **Content Extraction Per Page**
* Extract text content, title, metadata
* Identify and extract code blocks with language detection
* Discover images with alt text, dimensions
* Find asset links (PDFs, documents, archives)
* Optional: Structured extraction via LLM (`response_shape`)
5. **Content Chunking** (optional)
* Split page content by strategy: sentences, paragraphs, words, characters
* Configurable chunk size and overlap
* Track chunk metadata for joined results
6. **Document Expansion**
* Create separate documents for page content, each code block, each image
* Preserve parent URL and crawl depth metadata
7. **Multi-Modal Embedding Generation**
* E5-Large (1024D) for page text content
* Jina Code (768D) for code blocks
* SigLIP (768D) for images (if `generate_image_embeddings=true`)
8. **Output**
* Documents with text content, code blocks, images
* Asset links discovered but not crawled
* Multiple embeddings per document for hybrid search
## When to Use
| Use Case | Description |
| --------------------------- | ------------------------------------------------------------- |
| **API documentation** | Index technical documentation with code examples and diagrams |
| **Knowledge base crawling** | Extract FAQs, guides, and tutorials from support sites |
| **Job board scraping** | Find job listings with parsed content and structured fields |
| **News aggregation** | Collect and index articles with multimodal content |
| **Competitive analysis** | Monitor competitor websites for content changes |
| **Open source docs** | Index project documentation from GitHub Pages, ReadTheDocs |
| **Product research** | Gather product information from multiple websites |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------- | ------------------------------------------------------------- |
| Protected/authenticated content | Configure via `custom_headers` with auth tokens |
| PDF-only extraction | `document_graph_extractor` (better OCR, layout detection) |
| Social media scraping | Use platform-specific APIs (Twitter API, Instagram Graph API) |
| E-commerce product catalogs | Use platform APIs when available (better data structure) |
| Very large sites (10K+ pages) | Increase `max_pages`, implement crawl goal filtering |
## Input Schema
| Field | Type | Required | Description |
| ----- | ------ | -------- | ------------------------------------------------------------------------- |
| `url` | string | **Yes** | Seed URL to start crawling from. Example: `https://docs.example.com/api/` |
```json theme={null}
{
"url": "https://docs.example.com/getting-started"
}
```
**Input Examples:**
| Type | Example |
| ----------------- | ----------------------------------- |
| API documentation | `https://docs.openai.com/api/` |
| Knowledge base | `https://help.example.com/` |
| Blog | `https://blog.example.com/` |
| Job board | `https://jobs.example.com/listings` |
## Output Schema
Each crawled page produces one or more documents depending on content extraction and expansion settings:
| Field | Type | Description |
| ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------- |
| `content` | string | Extracted text content from page |
| `title` | string | Page title (from `` tag or heading) |
| `page_url` | string | Full URL of crawled page |
| `code_blocks` | array | Code blocks found on page (structure: `[{language, code, line_start, line_end}]`) |
| `images` | array | Images found on page (structure: `[{src, alt, title, width, height}]`) |
| `asset_links` | array | Downloadable assets discovered (structure: `[{url, file_type, link_text, file_extension}]`) |
| `chunk_index` | integer | Position within page chunks (if chunking enabled) |
| `total_chunks` | integer | Total chunks from this page (if chunking enabled) |
| `crawl_depth` | integer | Depth from seed URL (0 = seed, 1 = links from seed, etc.) |
| `parent_url` | string | Referrer URL (previous page in crawl path) |
| `intfloat__multilingual_e5_large_instruct` | float\[1024] | E5-Large text embedding, L2 normalized |
| `jinaai__jina_embeddings_v2_base_code` | float\[768] | Jina Code embedding (if code blocks extracted) |
| `google__siglip_base_patch16_224` | float\[768] | SigLIP visual embedding (if `generate_image_embeddings=true`) |
```json theme={null}
{
"content": "The REST API provides endpoints for creating, reading, updating, and deleting resources...",
"title": "REST API Overview - Example Docs",
"page_url": "https://docs.example.com/api/overview",
"code_blocks": [
{
"language": "python",
"code": "import requests\nresponse = requests.get('https://api.example.com/users')",
"line_start": 1,
"line_end": 2
}
],
"images": [
{
"src": "https://docs.example.com/images/api-flow.png",
"alt": "API request flow diagram",
"width": 800,
"height": 600
}
],
"asset_links": [
{
"url": "https://docs.example.com/downloads/openapi.yaml",
"file_type": "openapi",
"link_text": "Download OpenAPI Spec",
"file_extension": "yaml"
}
],
"crawl_depth": 2,
"parent_url": "https://docs.example.com/api/",
"intfloat__multilingual_e5_large_instruct": [0.023, -0.041, 0.018, ...],
"jinaai__jina_embeddings_v2_base_code": [0.045, -0.023, ...],
"google__siglip_base_patch16_224": [0.078, -0.091, ...]
}
```
## Parameters
### Crawl Configuration Parameters
| Parameter | Type | Default | Range | Description |
| --------------- | ------- | ----------------- | ----------------------- | --------------------------------------------------------------------------------------------- |
| `max_depth` | integer | 2 | 0-1000 | Maximum link depth from seed (0 = seed URL only, higher = deeper crawl) |
| `max_pages` | integer | 50 | 1-1000000 | Maximum pages to crawl in single run |
| `crawl_timeout` | integer | 300 | 10-3600 | Maximum time for crawl in seconds (10s - 1h) |
| `crawl_mode` | enum | `"deterministic"` | deterministic, semantic | BFS deterministic or LLM-guided semantic crawling |
| `crawl_goal` | string | null | - | Goal for semantic crawling (e.g., "find all API endpoints"). Used with `crawl_mode: semantic` |
### Rendering Parameters
| Parameter | Type | Default | Description |
| ----------------- | ---- | -------- | -------------------------------------------------------------------------------------- |
| `render_strategy` | enum | `"auto"` | Rendering method: `static` (HTML only), `javascript` (Puppeteer), `auto` (auto-detect) |
### URL Filtering Parameters
| Parameter | Type | Default | Description |
| ------------------ | ----- | ------- | -------------------------------------------------------------------------------------- |
| `include_patterns` | array | null | Regex patterns for URLs to include (whitelist). Example: `["/docs/.*", "/api/.*"]` |
| `exclude_patterns` | array | null | Regex patterns for URLs to exclude (blacklist). Example: `["/admin/.*", ".*logout.*"]` |
### Content Chunking Parameters
| Parameter | Type | Default | Range | Description |
| ---------------- | ------- | -------- | ---------------------------------------------- | ------------------------------------------------ |
| `chunk_strategy` | enum | `"none"` | none, sentences, paragraphs, words, characters | How to split page content |
| `chunk_size` | integer | 500 | 1-10000 | Target size per chunk (units depend on strategy) |
| `chunk_overlap` | integer | 50 | 0-5000 | Overlap between consecutive chunks |
### Document Identity Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------ |
| `document_id_strategy` | enum | `"url"` | How to generate document IDs: `url` (unique per page), `position` (sequential), `content` (hash-based) |
### Embedding Parameters
| Parameter | Type | Default | Description |
| --------------------------- | ------- | ------- | -------------------------------------------------- |
| `generate_text_embeddings` | boolean | true | Generate E5-Large text embeddings for page content |
| `generate_code_embeddings` | boolean | true | Generate Jina Code embeddings for code blocks |
| `generate_image_embeddings` | boolean | true | Generate SigLIP embeddings for discovered images |
### LLM Structured Extraction Parameters
| Parameter | Type | Default | Description |
| ---------------- | ---------------- | ------- | ---------------------------------------------------------------------------------- |
| `response_shape` | string or object | null | Define structured extraction: natural language description or JSON schema |
| `llm_provider` | string | null | LLM provider: `openai`, `google`, `anthropic` (required if using `response_shape`) |
| `llm_model` | string | null | Specific LLM model (e.g., `gpt-4o-mini`, `gemini-2.5-flash`) |
| `llm_api_key` | string | null | API key (supports secret vault references like `${vault:openai-key}`) |
### Resilience: Retry Parameters
| Parameter | Type | Default | Range | Description |
| --------------------- | ------- | ------- | --------- | -------------------------------------------- |
| `max_retries` | integer | 3 | 0-10 | Maximum retry attempts on request failure |
| `retry_base_delay` | number | 1.0 | 0.1-30.0 | Base delay for exponential backoff (seconds) |
| `retry_max_delay` | number | 30.0 | 1.0-300.0 | Maximum delay between retries (seconds) |
| `respect_retry_after` | boolean | true | - | Respect `Retry-After` header from server |
### Resilience: Proxy Parameters
| Parameter | Type | Default | Description |
| ------------------------------- | ------- | ------- | -------------------------------------------------------------------------------- |
| `proxies` | array | null | Proxy URLs for rotation. Example: `["http://proxy1:8080", "http://proxy2:8080"]` |
| `rotate_proxy_on_error` | boolean | true | Rotate proxy when request fails |
| `rotate_proxy_every_n_requests` | integer | 0 | Rotate proxy every N requests (0 = no periodic rotation) |
### Resilience: Captcha Parameters
| Parameter | Type | Default | Description |
| -------------------------- | ------- | ------- | ---------------------------------------------------------------- |
| `captcha_service_provider` | string | null | Captcha solving service: `2captcha`, `anti-captcha`, `capsolver` |
| `captcha_service_api_key` | string | null | API key for captcha service (supports secret vault references) |
| `detect_captcha` | boolean | true | Auto-detect captcha challenges and attempt to solve |
### Resilience: Session Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------- | ------- | ----------------------------------------------------------------------------------------- |
| `persist_cookies` | boolean | true | Persist cookies across requests within single crawl |
| `custom_headers` | object | null | Custom HTTP headers. Example: `{"Authorization": "Bearer token", "User-Agent": "Custom"}` |
### Politeness Parameters
| Parameter | Type | Default | Range | Description |
| ------------------------ | ------ | ------- | -------- | -------------------------------------------- |
| `delay_between_requests` | number | 0.0 | 0.0-60.0 | Delay between consecutive requests (seconds) |
## Configuration Examples
<CodeGroup>
```json Basic Documentation Crawl theme={null}
{
"feature_extractor": {
"feature_extractor_name": "web_scraper",
"version": "v1",
"input_mappings": {
"url": "docs_url"
},
"field_passthrough": [
{ "source_path": "metadata.vendor" },
{ "source_path": "metadata.product" }
],
"parameters": {
"max_depth": 2,
"max_pages": 50,
"crawl_timeout": 300,
"render_strategy": "auto",
"generate_text_embeddings": true,
"generate_code_embeddings": true,
"generate_image_embeddings": false,
"delay_between_requests": 0.5
}
}
}
```
```json API Docs with Structured Extraction theme={null}
{
"feature_extractor": {
"feature_extractor_name": "web_scraper",
"version": "v1",
"input_mappings": {
"url": "api_docs_url"
},
"parameters": {
"max_depth": 3,
"max_pages": 100,
"include_patterns": ["/api/.*", "/reference/.*"],
"exclude_patterns": ["/changelog/.*", "/deprecated/.*"],
"render_strategy": "javascript",
"response_shape": {
"type": "object",
"properties": {
"endpoints": {
"type": "array",
"items": {
"type": "object",
"properties": {
"method": { "type": "string" },
"path": { "type": "string" },
"description": { "type": "string" },
"parameters": { "type": "array", "items": { "type": "string" } }
}
}
},
"authentication": { "type": "string" },
"rate_limits": { "type": "string" }
}
},
"llm_provider": "openai",
"llm_model": "gpt-4o-mini",
"generate_text_embeddings": true,
"generate_code_embeddings": true,
"generate_image_embeddings": true
}
}
}
```
```json Knowledge Base with Semantic Crawling theme={null}
{
"feature_extractor": {
"feature_extractor_name": "web_scraper",
"version": "v1",
"input_mappings": {
"url": "kb_url"
},
"parameters": {
"max_depth": 4,
"max_pages": 200,
"crawl_mode": "semantic",
"crawl_goal": "Find all articles related to troubleshooting and error resolution",
"chunk_strategy": "paragraphs",
"chunk_size": 1000,
"chunk_overlap": 100,
"generate_text_embeddings": true,
"generate_code_embeddings": false,
"generate_image_embeddings": false
}
}
}
```
```json Job Board with Resilience theme={null}
{
"feature_extractor": {
"feature_extractor_name": "web_scraper",
"version": "v1",
"input_mappings": {
"url": "job_board_url"
},
"parameters": {
"max_depth": 2,
"max_pages": 500,
"render_strategy": "javascript",
"max_retries": 5,
"retry_base_delay": 2.0,
"retry_max_delay": 60.0,
"proxies": ["http://proxy1:8080", "http://proxy2:8080"],
"rotate_proxy_on_error": true,
"rotate_proxy_every_n_requests": 10,
"persist_cookies": true,
"delay_between_requests": 1.0,
"captcha_service_provider": "2captcha",
"captcha_service_api_key": "${vault:captcha-api-key}",
"generate_text_embeddings": true,
"generate_code_embeddings": false
}
}
}
```
```json High-Volume Crawl with Filtering theme={null}
{
"feature_extractor": {
"feature_extractor_name": "web_scraper",
"version": "v1",
"input_mappings": {
"url": "website_url"
},
"parameters": {
"max_depth": 5,
"max_pages": 1000,
"crawl_timeout": 3600,
"include_patterns": ["^https://example\\.com/.*"],
"exclude_patterns": [".*login.*", ".*admin.*", ".*/search\\?.*"],
"chunk_strategy": "sentences",
"chunk_size": 500,
"chunk_overlap": 50,
"document_id_strategy": "url",
"generate_text_embeddings": true,
"generate_code_embeddings": true,
"generate_image_embeddings": false,
"max_retries": 3,
"delay_between_requests": 0.2
}
}
}
```
```json Premium: Full Featured theme={null}
{
"feature_extractor": {
"feature_extractor_name": "web_scraper",
"version": "v1",
"input_mappings": {
"url": "target_url"
},
"parameters": {
"max_depth": 3,
"max_pages": 250,
"crawl_timeout": 1800,
"crawl_mode": "semantic",
"crawl_goal": "Find all technical content, code examples, and API documentation",
"render_strategy": "javascript",
"include_patterns": ["/docs/.*", "/api/.*", "/guide/.*"],
"exclude_patterns": ["/legacy/.*"],
"chunk_strategy": "paragraphs",
"chunk_size": 1000,
"chunk_overlap": 100,
"response_shape": "Extract key topics, code language, and frameworks mentioned",
"llm_provider": "anthropic",
"llm_model": "claude-3-5-haiku-20241022",
"generate_text_embeddings": true,
"generate_code_embeddings": true,
"generate_image_embeddings": true,
"max_retries": 5,
"retry_base_delay": 1.5,
"proxies": ["http://proxy1:8080"],
"rotate_proxy_every_n_requests": 5,
"persist_cookies": true,
"custom_headers": {
"User-Agent": "Mozilla/5.0 (compatible; MixpeekBot/1.0)"
},
"delay_between_requests": 0.5
}
}
}
```
</CodeGroup>
## Performance & Costs
| Metric | Value |
| ------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Average page load** | 2-5 seconds (depends on page complexity and rendering) |
| **Pages per minute** | 12-30 pages (with delays and retries) |
| **Code block extraction** | \~10ms per 1KB of code |
| **Image extraction** | \~50ms per 10 images |
| **Embedding latency** | \~5ms per text page (E5), \~10ms per code block (Jina), \~50ms per image (SigLIP) |
| **Cost** | Billed per crawled page — see [Billing & Pricing](/docs/platform/billing); rates come from `GET /v1/billing/pricing` |
| **Memory usage** | \~100MB base + \~1MB per 100 pages in crawl queue |
## Vector Indexes
All three embeddings are stored as [MVS](https://mixpeek.com/mvs) named vectors for hybrid search:
| Property | Value |
| ------------------- | ------------------------------------------ |
| **Index 1 name** | `intfloat__multilingual_e5_large_instruct` |
| **Dimensions** | 1024 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Normalization** | L2 normalized |
| Property | Value |
| ------------------- | -------------------------------------- |
| **Index 2 name** | `jinaai__jina_embeddings_v2_base_code` |
| **Dimensions** | 768 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Normalization** | L2 normalized |
| Property | Value |
| ------------------- | ---------------------------------------------- |
| **Index 3 name** | `google__siglip_base_patch16_224` |
| **Dimensions** | 768 |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Datatype** | float32 |
| **Status** | Optional (if `generate_image_embeddings=true`) |
## Comparison with Other Extractors
| Feature | web\_scraper | text\_extractor | multimodal\_extractor | document\_graph\_extractor |
| ------------------------- | ------------------------------------------------------------- | --------------- | --------------------- | -------------------------- |
| **Input types** | URLs (crawling) | Text only | Video, Image, Text | PDF only |
| **Recursive crawling** | ✅ Yes | ✗ | ✗ | ✗ |
| **Code extraction** | ✅ Yes | ✗ | ✗ | ✗ |
| **Image extraction** | ✅ Yes | ✗ | ✅ Yes | ✗ |
| **Multimodal embeddings** | ✅ Yes | Text only | ✅ Yes | Text only |
| **LLM extraction** | ✅ Yes | ✅ Yes | ✗ | ✗ |
| **Resilience features** | ✅ Yes | ✗ | ✗ | ✗ |
| **Best for** | Web crawling | Text search | Video/image/text | PDF analysis |
| **Cost** | Per crawled page — see [Billing & Pricing](/docs/platform/billing) | Per token | Per video minute | Per page |
## Resilience & Robustness
The web scraper includes enterprise-grade resilience features:
### Retry Strategy
* Exponential backoff with configurable base and max delays
* Respects server `Retry-After` headers
* Retries on network errors, timeouts, and temporary failures (5xx)
### Proxy Rotation
* Support for multiple proxies with automatic rotation
* Rotate on error or periodic rotation every N requests
* Helps avoid rate limiting and IP bans
### Captcha Detection & Solving
* Auto-detect common captcha types (reCAPTCHA, hCaptcha)
* Integration with 2captcha, Anti-Captcha, CapSolver services
* Fallback to manual review if solving fails
### Session Management
* Persistent cookies across requests within a single crawl
* Custom HTTP headers for authentication
* Support for API key and bearer token injection
### URL Filtering
* Include patterns (whitelist): Only crawl matching URLs
* Exclude patterns (blacklist): Skip URLs matching patterns
* Prevent crawling auth/admin pages, search results, etc.
## Limitations
* **Content-only crawling**: Does not execute custom JavaScript actions (clicking, form submission, scrolling)
* **Authentication**: Limited to HTTP headers (Bearer tokens, API keys). No interactive login flows.
* **Dynamic content**: JavaScript-rendering adds 2-3x latency per page
* **Large sites**: 10K+ page sites may require high `max_pages` and long timeouts
* **Robots.txt**: Does not parse `robots.txt`; respect via `delay_between_requests` and `max_pages`
* **Rate limiting**: May be blocked by aggressive rate limiting; use proxies and delays
## Related
* [Feature Extractors Overview](/docs/processing/feature-extractors)
* [Text Extractor](/docs/processing/extractors/text)
* [Document Graph Extractor](/docs/processing/extractors/document)
* [Image Extractor](/docs/processing/extractors/image)
* [Multimodal Extractor](/docs/processing/extractors/multimodal)
# Pipeline Configuration (Advanced)
Source: https://docs.mixpeek.com/docs/processing/feature-extractors
Advanced knobs behind a collection's processing pipeline — input mappings, field passthrough, parameters, and feature URIs
<Info>
**Start with [Features](/docs/processing/features).** `features: ["image_search"]` is the way to create collections — the platform resolves the pipeline and defaults the wiring. This page covers the advanced configuration underneath: explicit input mappings, field passthrough, pipeline parameters, and the feature URIs that retrievers query. The `feature_extractor` config object shown here still works everywhere but is a **deprecated alias** — see the [migration guide](/docs/processing/extractor-migration).
</Info>
<Frame>
<img alt="Processing pipeline turning objects into reusable features" />
</Frame>
Every collection runs one processing pipeline: Ray-powered workflows that read objects, run ML models, and write features into collection documents. Each pipeline output exposes a stable **feature URI** so retrievers, taxonomies, and clusters can reference it with confidence. When you create a collection with `features: [...]`, Mixpeek picks the pipeline, pins its version, and defaults its inputs; the knobs below are for when you need explicit control.
## Anatomy of an explicit pipeline config
The explicit config is attached via the `feature_extractor` field — mutually exclusive with `features` (provide one or the other):
```json theme={null}
{
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {
"text": "product_text"
},
"field_passthrough": [
{ "source_path": "metadata.category" },
{ "source_path": "metadata.brand" }
],
"parameters": {
"model": "multilingual-e5-large-instruct",
"normalize": true
}
}
}
```
Four knobs control what the pipeline sees and produces.
### `feature_extractor_name` + `version`
Identifies the pipeline and pins its version per collection — a new `v2` ships without disturbing collections on `v1`. You roll forward deliberately via a [namespace migration](/docs/api-reference/namespace-migrations/list-migrations), not in place. Built-in names are accepted as deprecated aliases of their [feature](/docs/processing/extractor-migration); custom plugin names are yours and stay first-class.
### `input_mappings` — bind pipeline inputs to object fields
Every pipeline declares an `input_schema` (e.g., the text pipeline expects a `text` input). `input_mappings` bind those named inputs to fields in your bucket schema: `product_text`, `video_url`, etc. Use **flat field names** directly, not prefixed paths like `payload.product_text`.
On the `features: [...]` path these default to the standard `uploads` bucket properties (`image`, `video`, `audio`, `pdf`, `content`, `url`) — you only need explicit mappings for custom bucket schemas.
### `field_passthrough` — carry source fields into documents
Pipeline outputs alone are rarely enough at query time. You want to filter by `metadata.category` or `status` without re-joining against the source bucket. `field_passthrough` copies selected source fields onto every output document so retrievers can use them directly in metadata filters.
<Tip>
If you plan to filter by a field at query time, pass it through at ingestion time. Reaching back into the source bucket from a retriever stage is 100–1000x slower than a local metadata predicate.
</Tip>
### `parameters` — tune the pipeline
Parameters are pipeline-specific: chunk strategy, embedding model variant, OCR enable/disable, transcription language, thumbnail quality. Invalid parameters return a teaching `422` that includes the pipeline's parameter schema (also available via `GET /v1/collections/features/extractors`).
## The output schema
When you configure a collection, it immediately calculates an `output_schema` that merges passthrough fields with pipeline outputs. You can inspect this before any processing runs:
```bash theme={null}
curl "https://api.mixpeek.com/v1/collections/{collection_id}/features" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
The response lists every feature URI the collection will produce, its type, its dimensions (for embeddings), and its description. Use this to validate downstream retrievers, taxonomies, and clusters before you ingest a single object.
## Feature URIs: the stable contract
Every pipeline output publishes a URI in the form `mixpeek://{name}@{version}/{output}`:
* `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1`
* `mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding`
* `mixpeek://universal_extractor@v1/gemini-embedding-2`
The URI is the contract between ingestion and everything downstream:
* **Retrievers** reference features by URI in `feature_search` stages.
* **Taxonomies** classify against a feature URI.
* **Clusters** group documents by a feature URI.
* **Migrations** swap one URI for another across retrievers atomically.
Because the URI pins the pipeline version, a retriever built against `v1` keeps working even after `v2` is released — it simply doesn't see `v2` features unless you explicitly migrate. URIs embed internal pipeline names; that's expected — they're version-pinned implementation references, not config vocabulary. Discover the URIs your collection actually produces with `GET /v1/collections/{id}/features` rather than constructing them by hand.
<Tip>
Treat feature URIs like API versions: pin them in production retrievers, and roll them forward deliberately rather than in-place.
</Tip>
## Vector index registration
When a pipeline publishes an embedding, Mixpeek registers a vector index in [MVS](/docs/vector-store/overview) at the matching URI. Retrievers then query that exact index via `feature_search`.
For built-in pipelines this is automatic. For [custom extractors](/docs/processing/custom-extractors), it's the single biggest pitfall: the `features` list in `manifest.py` must use the exact keys `feature_type`, `feature_name`, `embedding_dim`, `distance_metric`. Intuitive-but-wrong names (`type`, `name`, `dimensions`, `distance`) silently create a collection with **zero vector indexes**, and the ingestion task reports `COMPLETED` with 0 documents written.
## What happens at runtime
For a single-pipeline collection, the end-to-end path is:
1. **Flatten.** API flattens the manifest into per-pipeline row artifacts (Parquet) and stores them in S3.
2. **Schedule.** Ray poller discovers pending batches and submits a job.
3. **Run.** Workers load the dataset, run the pipeline flow (GPU if available), and emit features plus passthrough fields.
4. **Write.** `MVSBatchProcessor` writes vectors and payloads to MVS, emits webhook events, and updates index signatures.
Every document records lineage metadata so you can trace any feature back to the object that produced it:
```json theme={null}
{
"root_object_id": "obj_123",
"source_collection_id": "col_source",
"processing_tier": 1,
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
}
```
When pipelines are chained across collections, the `processing_tier` field becomes load-bearing — see [Multi-Tier Feature Extraction](/docs/processing/multi-tier-extractors) for how the DAG executes.
## Performance and scaling
* **GPU workers** deliver 5–10x faster throughput for embeddings, reranking, and video processing. CPU-only pipelines skip GPU allocation (saves \~3 minutes of cluster startup and \~6x on cost).
* **Ray Data** handles batching, shuffling, and parallelization automatically. Default batch size is 64 rows; tune via the pipeline's compute profile.
* **Autoscaling** maintains target utilization (`0.7` CPU, `0.8` GPU by default).
* **Inference cache** short-circuits repeated model calls when inputs hash to the same key — handy when reprocessing or when ingesting near-duplicates.
* **Normalization** happens once at ingest (720p video mezzanine, capped image edges, 16kHz mono audio) — it bounds per-unit cost and is included in the [modality rates](/docs/platform/billing). Opt out per collection with `full_res: true` (2x multiplier).
## Operational checklist
1. **Prefer `features: [...]`.** Reach for the explicit config only when you need custom input mappings, passthrough, or parameters.
2. **Pin versions.** Upgrade deliberately via new collection + migration, never in-place.
3. **Batch uploads.** Keep ingestion batches in the 1k–10k object range to maximize parallelism without overwhelming the scheduler.
4. **Use `field_passthrough` for every metadata filter.** If you plan to filter by `category` at query time, pass it through at ingestion time.
5. **Inspect features before querying.** `GET /v1/collections/{id}/features` returns the feature URIs, dimensions, and descriptions actually registered — use this to confirm the pipeline ran correctly before building retrievers.
6. **Watch lineage, not just counts.** A task reporting `COMPLETED` with 0 features written almost always means an `input_mappings` mistake or (for custom extractors) a bad `features` manifest. The [Document Lineage API](/docs/api-reference/document-lineage/get-document-lineage) and the `vector_indexes` field on the collection are your diagnostic tools.
## Reference
* **[Features](/docs/processing/features)** — the feature menu per modality, discovery endpoint, and cost estimates.
* **[Migration guide](/docs/processing/extractor-migration)** — mapping deprecated built-in extractor names to feature keys.
* **[Custom Extractors](/docs/processing/custom-extractors)** — bring your own pipeline; runs on the same Ray infrastructure with full GPU support, versioning, and observability, and prices per unit from its declared compute profile.
# Features
Source: https://docs.mixpeek.com/docs/processing/features
Features are what you want to search by — pick them per file type; Mixpeek resolves the pipeline, defaults the wiring, and prices per natural unit
Features answer the third of the [three pricing questions](/docs/platform/billing#how-pricing-works) — after *what kind of files* and *how much content* comes **what you want to search by**. Instead of choosing models or pipelines, you declare the capabilities you want — visual similarity (`image_search`), faces (`faces`), on-screen text (`onscreen_text`), document layout (`document_layout`) — and the platform resolves the implementation internally. Swapping or upgrading the models behind a feature never changes your config, your API calls, or your pricing.
Every feature applies to a **modality** (the *what kind of files* question) and is billed in that modality's natural unit (the *how much* question):
| Modality | Unit | Base feature (included in base rate) | Add-on features |
| ---------- | ---------------- | ------------------------------------ | ------------------------------------------------------------------------- |
| `image` | per image | `image_search` | `faces`, `multimodal_understanding` |
| `video` | per minute | `video_search` | `faces`, `onscreen_text`, `audio_fingerprint`, `multimodal_understanding` |
| `audio` | per minute | `audio_search` | `multimodal_understanding` |
| `document` | per page | `document_search` | `faces`, `document_layout`, `multimodal_understanding` |
| `text` | per token | `text_search` | `multimodal_understanding` |
| `web` | per crawled page | `web_crawl` | — |
Clustering and taxonomy enrichment are **included** on every modality at no additional charge. For rates, tiers, and how usage is billed, see [Billing & Pricing](/docs/platform/billing).
## Discover features
`GET /v1/collections/features` returns the live feature menu — modalities, units, feature keys, display names, and current rates. No authentication required. This endpoint is the source of truth: studio, the homepage pricing page, and the billing engine all read the same catalog.
```bash theme={null}
curl "https://api.mixpeek.com/v1/collections/features"
```
```json theme={null}
{
"modalities": [
{
"modality": "video",
"unit": "minute",
"unit_display": "per minute",
"base": {
"key": "video_search",
"name": "Video search (scene embeddings)",
"kind": "base",
"included_in_base_rate": true,
"rate_usd": 0.05,
"per": 1
},
"addons": [
{ "key": "faces", "name": "Face detection + identity", "kind": "addon", "rate_usd": 0.10, "per": 1, "companion": true },
{ "key": "onscreen_text", "name": "On-screen text (video OCR)", "kind": "addon", "rate_usd": 0.10, "per": 1, "companion": true },
{ "key": "clustering", "name": "Clustering (included)", "kind": "included", "included": true, "pricing_note": "included — no additional charge" }
]
}
],
"custom_features": [],
"full_res_multiplier": 2.0
}
```
Response abbreviated — the live payload covers all six modalities. Feature `kind` tells you how it's priced:
| Kind | Meaning |
| ---------- | ------------------------------------------------------------------------------------------------------ |
| `base` | Covered by the modality's per-unit base rate — you get it by creating any collection for that modality |
| `addon` | Priced per the same unit, added on top of the base rate |
| `external` | Backed by external LLM inference — usage-based passthrough pricing, never flat-rated |
| `included` | Bundled enrichment (clustering, taxonomy) — no additional charge |
## Create a collection with features
Pass `features: [...]` to [Create Collection](/docs/api-reference/collections/create-collection). The platform resolves the pipeline, pins its version, and defaults the input wiring for you:
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/collections" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "press-photos",
"source": { "type": "bucket", "bucket_ids": ["'$BUCKET_ID'"] },
"features": ["image_search"]
}'
```
```python Python theme={null}
import os
from mixpeek import Mixpeek
# namespace= is required — it's sent as the X-Namespace header on every call
# (collection create returns 403 without it).
client = Mixpeek(
api_key=os.environ["MIXPEEK_API_KEY"],
namespace=os.environ["MIXPEEK_NAMESPACE"],
)
collection = client.collections.create(
collection_name="press-photos",
source={"type": "bucket", "bucket_ids": [os.environ["BUCKET_ID"]]},
features=["image_search"],
)
```
```javascript JavaScript theme={null}
import { Mixpeek } from "mixpeek";
// namespace is sent as the X-Namespace header on every call (collection
// create is namespace-scoped).
const client = new Mixpeek({
apiKey: process.env.MIXPEEK_API_KEY,
namespace: process.env.MIXPEEK_NAMESPACE,
});
const { data: collection } = await client.collections.createCollection({
createCollectionRequest: {
collection_name: "press-photos",
source: { type: "bucket", bucket_ids: [process.env.BUCKET_ID] },
features: ["image_search"],
},
});
```
</CodeGroup>
<Note>
Both SDKs accept `features` on collection create — Python **`mixpeek>=1.3.29`** (`client.collections.create(...)`) and JavaScript **`mixpeek>=0.81.28`** (`client.collections.createCollection({ createCollectionRequest: { ... } })`). Set the namespace when you construct the client (Python `namespace=`, JS `namespace:`, or the `MIXPEEK_NAMESPACE` env var) — collection create is namespace-scoped and returns 403 without it. The `feature_extractor` field is a [deprecated alias](/docs/processing/extractor-migration).
</Note>
### Default input wiring
Every organization's scaffolded `uploads` bucket exposes one standard source property per modality — `image`, `video`, `audio`, `pdf`, `content` (text), and `url` (web). A features-based create maps the pipeline's inputs to those properties automatically, so the collection is runnable with zero wiring.
If your bucket uses different property names, the create returns a teaching `422` naming the fields that exist so you can rename the property — or fall back to explicit `input_mappings` via the [advanced pipeline config](/docs/processing/feature-extractors).
### Errors teach
Validation on the features path is designed for humans and agents guessing their way in — every rejection tells you what to do instead:
* **Unknown key** — `features: ["face"]` responds with the full list of valid feature keys (and reminds you about `custom:<plugin>`).
* **One pipeline per collection** — some feature combinations need separate collections (e.g. a base feature plus a `companion: true` add-on). The error lists exactly which features group into which collection: create one collection per group over the same source bucket. Add-ons flagged `companion: true` in discovery always create a companion collection alongside the base.
* **`features` XOR `feature_extractor`** — provide one or the other, never both.
* **Not yet available** — feature keys that appear on the roadmap but haven't shipped are rejected explicitly rather than silently ignored.
### Custom features (bring your own)
[Custom extractors](/docs/processing/custom-extractors) you publish are *your* vocabulary — they stay explicitly named and are selected as `custom:<plugin_name>`:
```json theme={null}
{
"collection_name": "product-defects",
"source": { "type": "bucket", "bucket_ids": ["bkt_123"] },
"features": ["custom:defect_detector"]
}
```
Custom features are priced per unit from the compute profile the plugin declares — the same machinery that prices native features. See [Custom Extractors](/docs/processing/custom-extractors).
### Full resolution opt-out
By default, Mixpeek normalizes content once at ingest (video to a 720p mezzanine, images capped at \~1568px max edge, audio to 16kHz mono) — originals are always kept untouched in your bucket. If a workload needs extraction on the original resolution (fine print OCR, tiny logos), opt out per collection:
```json theme={null}
{
"collection_name": "fine-print",
"source": { "type": "bucket", "bucket_ids": ["bkt_123"] },
"features": ["document_layout"],
"full_res": true
}
```
Full-res processing bills at a **2x multiplier** on the modality's base and add-on rates (`full_res_multiplier` in the pricing payload).
### Chain collections (advanced)
Searching by one thing often produces content worth searching by another — e.g. search video by scenes, then feed those documents into a second collection to search the results by something else. Collection-to-collection pipelines do exactly this: a collection can read another collection's documents as its source, forming a processing DAG. See [Multi-Tier Feature Extraction](/docs/processing/multi-tier-extractors).
## Find your `feature_uri` (to search)
A [`feature_search`](/docs/retrieval/stages/feature-search) stage needs a **`feature_uri`** — the exact vector index to search. You picked *features* by name; each one produces a vector index whose `feature_uri` looks like `mixpeek://<extractor>@<version>/<model>`.
<Warning>
The last segment is the embedding **model name**, not the literal word `embedding`. A wrong `feature_uri` (e.g. `.../embedding`) does **not** error — the stage matches nothing and silently returns **0 results** (with a self-correcting warning in the response). Always use the exact string.
</Warning>
**The reliable, version-proof way — read it from the collection.** After processing, `GET` the collection and copy the URI; this always matches what you actually built, even if model versions change:
<CodeGroup>
```bash cURL theme={null}
curl "https://api.mixpeek.com/v1/collections/$COLLECTION_ID" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
# → vector_indexes[].feature_uri e.g. "mixpeek://universal_extractor@v1/gemini-embedding-2"
```
```python Python theme={null}
col = client.collections.get(collection_id)
for vi in col["vector_indexes"]:
print(vi["feature_uri"]) # paste the one you want into your feature_search stage
```
</CodeGroup>
**Common features → `feature_uri`** (the exact string a fresh `@v1` collection produces):
| Feature (`features: [...]`) | Extractor | `feature_uri` |
| --------------------------- | ---------------------- | --------------------------------------------------------------- |
| `video_search` | `universal_extractor` | `mixpeek://universal_extractor@v1/gemini-embedding-2` |
| `text_search` | `text_extractor` | `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1` |
| `multimodal_understanding` | `multimodal_extractor` | `mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding` |
<Note>
**Video transcript search.** `video_search` produces **one** blended scene embedding (`gemini-embedding-2`); the transcript rides along as a `text` payload field, not a separate searchable vector. To search a transcript as its own vector, add the `multimodal_understanding` feature with transcription enabled — it creates a second index `mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1`. When in doubt, `GET` the collection and read the `feature_uri`s it actually has.
</Note>
## Estimate before you run
`POST /v1/organizations/billing/estimate` quotes planned ingestion using the **same rating engine that bills you** — the quote and the charge can't disagree:
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/billing/estimate" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"items": [
{ "mime_type": "video/mp4", "minutes": 42 },
{ "mime_type": "application/pdf", "pages": 300 }
],
"features": ["faces", "onscreen_text"]
}'
```
```python Python theme={null}
resp = requests.post(
"https://api.mixpeek.com/v1/organizations/billing/estimate",
headers={"Authorization": f"Bearer {os.environ['MIXPEEK_API_KEY']}"},
json={
"items": [
{"mime_type": "video/mp4", "minutes": 42},
{"mime_type": "application/pdf", "pages": 300},
],
"features": ["faces", "onscreen_text"],
},
)
quote = resp.json()
```
```javascript JavaScript theme={null}
const resp = await fetch(
"https://api.mixpeek.com/v1/organizations/billing/estimate",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.MIXPEEK_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
items: [
{ mime_type: "video/mp4", minutes: 42 },
{ mime_type: "application/pdf", pages: 300 },
],
features: ["faces", "onscreen_text"],
}),
}
);
const quote = await resp.json();
```
</CodeGroup>
```json theme={null}
{
"pricing_model": "v2",
"placeholder_rates": false,
"line_items": [
{ "modality": "video", "key": "video_search", "units": 42.0, "rate_usd": 0.05, "per": 1, "amount_usd": 2.10 },
{ "modality": "video", "key": "faces", "units": 42.0, "rate_usd": 0.10, "per": 1, "amount_usd": 4.20 },
{ "modality": "video", "key": "onscreen_text", "units": 42.0, "rate_usd": 0.10, "per": 1, "amount_usd": 4.20 },
{ "modality": "document", "key": "document_search", "units": 300.0, "rate_usd": 1.50, "per": 1000, "amount_usd": 0.45 },
{ "modality": "document", "key": "faces", "units": 300.0, "rate_usd": 1.00, "per": 1000, "amount_usd": 0.30 }
],
"total_usd": 11.25,
"batch_minimum_applied": false,
"allowance_pool_usd": 10.0,
"allowance_covered_usd": 10.0,
"estimated_overage_usd": 1.25
}
```
Base features are quoted automatically per the modality of each item — you only list add-ons. Unknown MIME types quote at the cheapest band rather than failing (be forgiving is a design rule here).
## How this relates to extractors
Under the hood every feature resolves to an extraction pipeline, and [custom extractors](/docs/processing/custom-extractors) let you register your own. But **built-in extractor names are no longer part of the public API surface** — they remain accepted in configs only as deprecated aliases. If you have existing configs using `feature_extractor`, see the [migration guide](/docs/processing/extractor-migration); for advanced pipeline knobs (input mappings, field passthrough, parameters), see [Pipeline Configuration](/docs/processing/feature-extractors).
# Migrate Embedding Models
Source: https://docs.mixpeek.com/docs/processing/model-migration
Switch a namespace to a new embedding model by re-extracting into a target namespace — safely, with validation and dry-run
Embedding dimensions are **locked at namespace creation** — you can't swap the embedding model in place, because the vector index dimensionality is fixed and vectors from different models aren't comparable. To move to a new model, you **re-extract** your data into a target namespace using the new extractor config, validate it, then cut over.
<Note>
This is the right path whenever you change the embedding model (e.g. a new frontier text model, or switching `text_extractor` → a higher-dim model). For routine model upgrades within the same family, the [model registry](/docs/processing/model-registry) hot-swaps the default for **new** collections without touching existing ones.
</Note>
## How it works
A **migration** of type `re_extract` reads your source namespace's objects and re-runs extraction with a new `feature_extractors` config (the new model) into a target namespace. Your source stays live and untouched until you choose to cut over.
| Migration type | What it does |
| -------------- | --------------------------------------------------------------------------------------- |
| `re_extract` | Re-run extraction with new extractor/model config (use this to change embedding models) |
| `copy` | Copy resources as-is to another namespace (no re-extraction) |
| `extend` | Add new features to existing documents without a full re-extract |
## 1. Validate first (dry run)
Always validate the migration config before committing compute — it checks the source, target, and extractor config without extracting anything.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/namespaces/migrations/validate" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"config": {
"migration_type": "re_extract",
"source_namespace_id": "ns_prod",
"target_namespace_name": "prod-v2",
"feature_extractors": [
{ "feature_extractor_name": "text_extractor", "version": "v1",
"params": { "embedding_model": "<new-model-id>" } }
]
}
}'
```
## 2. Create the migration
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/namespaces/migrations/" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"config": {
"migration_type": "re_extract",
"source_namespace_id": "ns_prod",
"target_namespace_name": "prod-v2",
"feature_extractors": [
{ "feature_extractor_name": "text_extractor", "version": "v1",
"params": { "embedding_model": "<new-model-id>" } }
],
"dry_run": false,
"webhook_url": "https://example.com/migration-status"
},
"start_immediately": false
}'
```
The response includes a `migration_id`.
| Config field | Purpose |
| ----------------------------------------------- | ------------------------------------------------------- |
| `migration_type` | `re_extract` to change the model |
| `source_namespace_id` | Namespace to migrate from |
| `target_namespace_name` / `target_namespace_id` | Where the re-extracted data lands |
| `feature_extractors` | New extractor config — the new embedding model |
| `filters` | Optionally migrate a subset (by collection, date, etc.) |
| `batch_options` | Tune batch size / parallelism |
| `dry_run` | Validate only, don't execute |
| `webhook_url` | Get progress callbacks |
## 3. Start and monitor
```bash theme={null}
# Start (if you didn't set start_immediately)
curl -sS -X POST "$MP_API_URL/v1/namespaces/migrations/{migration_id}/start" \
-H "Authorization: Bearer $MP_API_KEY"
# Poll status
curl -sS "$MP_API_URL/v1/namespaces/migrations/{migration_id}" \
-H "Authorization: Bearer $MP_API_KEY"
```
Cancel a running migration with `POST /v1/namespaces/migrations/{migration_id}/cancel`.
## 4. Cut over
Re-extraction re-pays GPU/extraction cost for every document, so validate on a subset first (use `filters`) and confirm quality before migrating everything. Once the target namespace is populated and validated:
1. Re-run your [evaluations](/docs/retrieval/evaluations) against the target namespace to confirm relevance is at least as good. For a head-to-head old-vs-new read on real query logs, run the same query set through both namespaces' retrievers as [benchmarks](/docs/retrieval/benchmarks), or [generate an evaluation dataset from recorded interactions](/docs/retrieval/interactions) so the comparison reflects production traffic rather than synthetic queries.
2. Point your application's `X-Namespace` (and retrievers) at the target namespace.
3. Retire the source namespace when you're confident.
## Rollback
The source namespace stays live and untouched until you retire it — that **is** the rollback path. If post-cutover metrics regress, point `X-Namespace` back at the source namespace; no re-extraction is needed in either direction. Only retire the source once the target has survived real traffic for as long as your risk tolerance requires.
<Warning>
Vectors from different embedding models are **not** comparable — you cannot mix old and new vectors in the same index, and you cannot copy embeddings across models (only `re_extract` regenerates them). Plan for the full re-extraction cost.
</Warning>
## Related
* [Model Registry](/docs/processing/model-registry) — how default models are resolved and hot-swapped
* [Feature Extractors](/docs/processing/feature-extractors) — extractor + model configuration
* [Evaluations](/docs/retrieval/evaluations) — verify the new model before cutting over
* [Reprocess existing content](/docs/tutorials/reprocess-existing-content) — the underlying re-extraction flow, step by step
* [Monitoring ingestion](/docs/processing/tasks) — track re-extraction progress
# Model Registry
Source: https://docs.mixpeek.com/docs/processing/model-registry
Load HuggingFace models, built-in models, or your own fine-tuned weights inside custom extractors
Models run inside [custom extractors](/docs/processing/custom-extractors). The model registry handles downloading, caching, and serving — you just declare which model to use and the infrastructure shares it across all workers via Ray's object store.
<Info>
**How models are consumed.** Built-in and HuggingFace models work on any plan — reference them by feature URI or load them in a pipeline. **Custom (uploaded) models** are consumed from inside a custom extractor (`model_source="namespace"`), so wiring one into ingest/retrieval also requires a [dedicated deployment](/docs/processing/custom-extractors#availability) (or an extractor merged via [Submissions](/docs/processing/extractor-marketplace)). Uploading + deploying a custom model is supported wherever your org has Enterprise infra.
</Info>
## Three Ways to Load Models
| Approach | When to Use |
| ------------------------------ | ---------------------------------------------------------------------------------------------------- |
| **Built-in Models** | Common tasks — embeddings, transcription, reranking. No code needed, just reference the feature URI. |
| **HuggingFace Models** | Any public HF model. Cached cluster-wide on first download. |
| **Custom Models** (Enterprise) | Your own fine-tuned weights uploaded as `.tar.gz`. Stored in S3, deployed to Ray. |
## HuggingFace Models (Recommended)
Use `LazyModelMixin` in your extractor's pipeline. Models load on first batch, not at actor creation, and are shared zero-copy across all workers.
```python theme={null}
from engine.models.lazy import LazyModelMixin
from engine.inference.services import BaseBatchInferenceService
class MyEmbeddingProcessor(LazyModelMixin, BaseBatchInferenceService):
model_id = "intfloat/multilingual-e5-large-instruct"
model_class = "AutoModel"
tokenizer_class = "AutoTokenizer"
torch_dtype = "float16"
model_source = "huggingface"
def _process_batch(self, batch):
model, tokenizer = self.get_model()
inputs = tokenizer(
batch["text"].tolist(),
padding=True,
truncation=True,
return_tensors="pt",
)
with torch.no_grad():
outputs = model(**inputs)
batch["embedding"] = outputs.last_hidden_state.mean(dim=1).tolist()
return batch
```
### LazyModelMixin Attributes
| Attribute | Type | Default | Description |
| ----------------- | ----------- | ----------------- | ------------------------------------------ |
| `model_id` | str | `""` | HuggingFace model ID or namespace model ID |
| `model_class` | str | `"AutoModel"` | Transformers model class name |
| `tokenizer_class` | str \| None | `"AutoTokenizer"` | Tokenizer class, or `None` to skip |
| `torch_dtype` | str | `"float32"` | `"float16"`, `"float32"`, or `"bfloat16"` |
| `model_source` | str | `"huggingface"` | `"huggingface"` or `"namespace"` |
Call `self.get_model()` to get a `(model, tokenizer)` tuple. Override `_instantiate_model(cached_data)` for non-standard architectures.
## Custom Models (Enterprise)
<Note>
Custom models require an **Enterprise** subscription. [Contact sales](https://mixpeek.com/contact) to enable.
</Note>
Upload fine-tuned weights and use them in extractors. Three steps:
<Steps>
<Step title="Upload">
```bash theme={null}
tar -czvf my_model.tar.gz ./model_weights/
curl -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/models" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-F "file=@my_model.tar.gz" \
-F "name=my-embedding-model" \
-F "version=1.0.0" \
-F "model_format=pytorch" \
-F "task_type=embedding" \
-F "num_gpus=0" \
-F "memory_gb=4.0"
```
Supported formats: `pytorch` (`.pt`, `.pth`), `safetensors`, `onnx`, `huggingface` (directory).
</Step>
<Step title="Deploy to Ray">
```bash theme={null}
curl -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/models/my-embedding-model_1_0_0/deploy" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
</Step>
<Step title="Use in your extractor">
Set `model_source = "namespace"` and override `_instantiate_model()`:
```python theme={null}
class MyCustomProcessor(LazyModelMixin, BaseBatchInferenceService):
model_id = "my-embedding-model_1_0_0"
model_source = "namespace"
def _instantiate_model(self, weights):
import torch
model = torch.nn.Linear(768, 256)
model.load_state_dict(weights)
model.to(self._detect_device())
model.eval()
return model, None
def _process_batch(self, batch):
model, _ = self.get_model()
# Use model...
```
</Step>
</Steps>
## Model Versioning
Models are versioned independently. Deploy a new version alongside the existing one, test in staging, then shift traffic:
```
my-embedding-model_1_0_0 (production)
my-embedding-model_2_0_0 (staging — validate before promoting)
```
The feature URI updates with the extractor version (`mixpeek://my_extractor@2.0.0/my_embedding`), so both versions can coexist.
## Python SDK
```python theme={null}
from mixpeek import Mixpeek
from mixpeek.api.custom_models_api import CustomModelsApi
client = Mixpeek(api_key="mxp_sk_...")
models = CustomModelsApi(client.api_client)
result = models.upload_model_namespaces(
namespace_id="ns_abc123",
file="my_model.tar.gz", # path to the .tar.gz archive
name="my-reranker",
version="1.0.0",
model_format="pytorch",
task_type="reranking",
)
models.deploy_model_namespaces(
namespace_id="ns_abc123",
model_id=result.model_id,
)
```
<Note>
The generated SDK exposes these under `CustomModelsApi` (`upload_model_namespaces`, `deploy_model_namespaces`, `list_models_namespaces`, …). See the [Models API reference](/docs/api-reference/custom-models/upload-a-custom-model) for exact parameters. The cURL form above is verified end-to-end.
</Note>
## Limits
| Limit | Value |
| --------------------------- | --------------------------------------- |
| Max models per organization | 50 |
| Max archive size | 5 GB |
| Supported formats | pytorch, safetensors, onnx, huggingface |
## Related
<CardGroup>
<Card title="Custom Extractors" icon="plug" href="/processing/custom-extractors">
Package and deploy extractors that use these models.
</Card>
<Card title="Extractor Quickstart" icon="rocket" href="/tutorials/custom-extractor-quickstart">
Build a working extractor with model loading end-to-end.
</Card>
<Card title="Model API Reference" icon="cloud-arrow-up" href="/api-reference/custom-models/upload-a-custom-model">
Upload, deploy, list, and delete model archives.
</Card>
<Card title="Self-Improving CV Pipeline" icon="eye" href="/tutorials/annotations-improve-features">
Full tutorial: deploy YOLO, annotate, fine-tune, redeploy.
</Card>
</CardGroup>
# Multi-Tier Feature Extraction
Source: https://docs.mixpeek.com/docs/processing/multi-tier-extractors
How chained collections, dependency tiers, and cross-tier lineage turn extraction into a composable DAG
<Card title="Browse the extractor catalog on GitHub" icon="github" href="https://github.com/mixpeek/mixpeek-extractors">
Runnable reference for every built-in Mixpeek extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry, so it always matches production.
</Card>
<Frame>
<img alt="Multi-tier extraction: raw objects flow through Tier 1 (decompose), Tier 2 (embed), Tier 3 (classify) as a chained collection DAG" />
</Frame>
A single extractor call gives you one pass over raw data. That works for homogeneous inputs and a single model. It falls apart the moment you need to transcribe audio *and then* embed the transcription, or OCR a PDF *and then* chunk the text *and then* classify each chunk. Multi-tier extraction solves this by turning ingestion into a declarative DAG: collections chain together, each tier reads the output of the previous one, and the engine schedules them in the correct order.
<Note>
This is the composition story for feature extractors. For the core concept — how a single extractor is configured — see [Feature Extractors](/docs/processing/feature-extractors). For the catalog, see the pages under **Built-in Feature Extractors**.
</Note>
## Why One Extractor Isn't Enough
Three problems force you out of the single-extractor model:
**1. Heterogeneous inputs.** A single 30-minute video is not one document — it's dozens of segments, each with its own transcription, thumbnail, embedding, and OCR. A PDF is N pages with layout, text chunks, and figures. One extractor can't own both the decomposition *and* every downstream enrichment without becoming an unmaintainable god-model.
**2. Model boundaries.** Transcription is an audio model. Text embedding is a text model. Taxonomy classification is an LLM. Each has different hardware (GPU vs API), different batch sizes, different failure modes. Trying to run them in a single actor wastes GPU time on I/O-bound work and vice versa.
**3. Reusability.** You want to build one transcription step and reuse the output for embedding, for summarization, for diarization, for taxonomy classification. If transcription is buried inside a monolithic extractor, every downstream use triggers a redundant pass.
Multi-tier extraction moves composition into the ingestion graph itself. One extractor per job, one concern per extractor, composed via collections.
## The Object → Document → Feature Model
Three primitives do all the work in Mixpeek:
| Primitive | What It Is | Why It Matters |
| ------------ | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| **Object** | Raw file or record in a bucket (a video, a PDF, a row of JSON). | The input boundary. You ingest objects. |
| **Document** | One row of output in a collection, produced by **decomposition**. | The query boundary. You search documents. |
| **Feature** | A named output attached to a document (embedding, transcription, OCR text, label, score). | The composition boundary. Retrievers, taxonomies, and clusters reference features by URI. |
The pipeline is always:
```
Object (bucket) → Decomposition → Document (collection) → Feature extractor → Feature (MVS + Mongo)
```
**Decomposition** is the step people miss. A 30-minute video isn't one document — it's dozens of segments with their own transcriptions, thumbnails, and embeddings. A PDF isn't one document — it's N pages with OCR, layout, and text chunks. The extractor decides the decomposition strategy (time/scene/silence for video; page/paragraph/sentence for text), and every resulting document gets its own row. Retrievers search at the document grain, not the object grain.
## Chaining Collections Across Tiers
To chain extractors, you chain collections: collection B's `source_collection_id` points at collection A, and B's extractor reads A's documents as input. The engine assigns a **processing tier** to each collection based on the dependency graph:
```
Tier 1 raw objects → multimodal_extractor → segments with transcription + scene embeddings
Tier 2 tier-1 documents → text_extractor → chunks with text embeddings
Tier 3 tier-2 documents → passthrough + taxonomy enrichment → classifications per chunk
```
Tier 1 runs first on raw objects. Tier 2 runs only once tier 1 has written results. Tier 3 only after tier 2. The engine respects this DAG automatically — you don't trigger tiers manually.
### Worked Example: Video → Transcription → Text Embeddings → Classifications
```json theme={null}
// Tier 1: raw video objects → segments with transcription
{
"collection_id": "col_segments",
"source_bucket_id": "bkt_videos",
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": { "video": "file" },
"parameters": { "segment_strategy": "scene", "run_transcription": true }
}
}
// Tier 2: segment transcriptions → text chunks + embeddings
{
"collection_id": "col_chunks",
"source_collection_id": "col_segments",
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": { "text": "transcription" },
"field_passthrough": [
{ "source_path": "segment_start_ms" },
{ "source_path": "segment_end_ms" },
{ "source_path": "root_object_id" }
],
"parameters": { "chunk_strategy": "sentences" }
}
}
// Tier 3: text chunks → taxonomy classifications
{
"collection_id": "col_classified",
"source_collection_id": "col_chunks",
"feature_extractor": {
"feature_extractor_name": "passthrough_extractor",
"version": "v1",
"input_mappings": { "text": "text_embedding" },
"parameters": { "taxonomy_id": "tax_topics" }
}
}
```
At query time, a retriever can `feature_search` over `col_classified` (fine-grained, classified chunks) and trace each hit back to the original video segment via `field_passthrough` metadata.
## How the Engine Schedules Tiers
1. **Flatten.** The API flattens the manifest into per-extractor row artifacts (Parquet) and stores them in S3, along with a dependency graph.
2. **Poll.** Ray pollers walk the graph tier-by-tier. A tier's batch is eligible only once every upstream tier has written its MVS rows.
3. **Submit.** When a tier's dependencies are satisfied, the poller submits a Ray job for that tier. Tiers at the same depth can run in parallel.
4. **Write.** Workers run the extractor, emit features and passthrough fields, and `MVSBatchProcessor` writes vectors and payloads to [MVS](/docs/vector-store/overview). Webhooks fire on completion.
5. **Cascade.** Downstream tiers become eligible as soon as their inputs land.
This is fundamentally a pull-based scheduler: nothing runs until the data it needs exists. It means you can re-ingest a single object at the top of the DAG and the entire chain cascades automatically.
## Cross-Tier Lineage
Every document carries the metadata needed to walk back up the chain:
```json theme={null}
{
"document_id": "doc_chunk_42",
"root_object_id": "obj_video_7",
"source_collection_id": "col_segments",
"source_document_id": "doc_segment_3",
"processing_tier": 2,
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
}
```
Use the [Document Lineage API](/docs/api-reference/document-lineage/get-document-lineage) to walk the tree in either direction:
* **Down from an object** — "show me every document derived from `obj_video_7`" returns all tier-1 segments, all tier-2 chunks, all tier-3 classifications in a single tree.
* **Up from a document** — "where did `doc_chunk_42` come from?" returns the source segment, the source object, and the extractors that produced each.
The [decomposition tree visualization](/docs/api-reference/document-lineage/get-decomposition-tree-visualization) renders this as a graph. It's the primary tool for debugging "why is this document showing up in my retriever results?"
<Tip>
When a tier reports `COMPLETED` with fewer documents than expected, the lineage tree tells you exactly where the drop happened — which input documents failed to produce output, and which extractor was running at the time.
</Tip>
## Decomposition Patterns
The shape of the DAG reflects the shape of your data:
| Pattern | Tier 1 | Tier 2+ | When to Use |
| ------------------ | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- |
| **Fan-out** | One extractor decomposes objects into many documents (video → segments, PDF → pages). | Per-segment enrichments (embedding, classification). | Video libraries, document intelligence. |
| **Fan-in** | Multiple source collections. | One extractor joins or aggregates across them. | Multimodal catalogs that combine text + image embeddings into a single document per product. |
| **Sidecar** | Primary extractor runs. | Secondary tier adds a derived feature (e.g., a summary or classification) to the same documents. | Adding enrichments after the fact without reprocessing the primary content. |
| **Versioned swap** | `v1` extractor runs on all objects. | `v2` runs on the same objects in a parallel collection. | A/B testing a new model version before migrating retrievers. |
## Migrations Across a Chain
Upgrading a tier-1 extractor is not a local change — every downstream tier reads tier-1's output, so a schema change ripples through the chain. The safe migration pattern:
1. **Branch.** Create a parallel tier-1 collection (`col_segments_v2`) using the new extractor version. Ingest the same objects from the same bucket.
2. **Rebuild downstream.** Create tier-2 and tier-3 collections pointing at `col_segments_v2` as their source. You now have a second, fully independent chain running in parallel.
3. **Evaluate.** Point an evaluation dataset at both chains (two retrievers referencing the different feature URIs). Compare recall, precision, latency, and cost.
4. **Cut over.** Use a [namespace migration](/docs/api-reference/namespace-migrations/list-migrations) to swap retriever references from the `v1` feature URIs to the `v2` URIs atomically.
5. **Retire.** Once traffic is stable on the new chain, delete the old collections.
This is the same blue/green pattern as service deploys, scaled up to the DAG: the feature URI is the stable address, and the migration flips which version of the chain answers to it.
## Operational Gotchas
* **`field_passthrough` at every tier.** Each tier only carries forward what you explicitly pass through. If you want `metadata.category` visible at tier 3, pass it through at tier 1 and tier 2 as well. Missing passthroughs are the most common source of "why is this field gone?" questions.
* **Tier-0 `input_mappings` always read from the bucket schema; tier-N reads from the previous collection's document schema.** They look the same but the resolution rules differ.
* **Re-ingesting a tier-1 object cascades the whole chain.** Expect tier-2 and tier-3 jobs to fire automatically once tier 1 writes. This is usually what you want, but it's worth knowing before you kick off a 10M-object reprocess.
* **Deleting a tier-1 document tombstones everything downstream.** Lineage is preserved so retrievers can filter out orphans; the actual downstream features are garbage-collected asynchronously.
* **Custom plugins work at any tier.** The same `manifest.py` / `pipeline.py` / `realtime.py` contract applies whether the plugin is reading raw objects or the output of another extractor.
# Retrieval Cookbook
Source: https://docs.mixpeek.com/docs/retrieval/cookbook
Ready-to-copy, runnable multi-stage retriever configurations for common multimodal patterns
<Frame>
<img alt="Retrieval cookbook: ready-to-copy pipeline recipes for RAG, hybrid search, video moments, face search, and dedup" />
</Frame>
Every recipe below is a complete, copy-paste retriever config. Create the retriever once, then execute it with runtime `inputs`.
<Note>
**Stage shape.** Every stage is `{ "stage_name", "stage_type", "config": { "stage_id", "parameters" } }`. `stage_name` is your label; `stage_id` is the implementation (`feature_search`, `rerank`, `attribute_filter`, …); `stage_type` is the category (`filter`, `sort`, `reduce`, `group`, `enrich`, `apply`). See [Retrievers](/docs/retrieval/retrievers) and [Stages](/docs/retrieval/stages/overview).
</Note>
<Warning>
**Point `feature_uri` at *your* collection's index.** Each recipe's `feature_search` uses a `feature_uri` for a specific extractor (e.g. `mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding`). That string must match the collection you built — a `video_search` collection uses `mixpeek://universal_extractor@v1/gemini-embedding-2`, **not** the multimodal one. A mismatched `feature_uri` silently returns **0 results**. Find yours with `GET /v1/collections/{id}` → `vector_indexes[].feature_uri` — see [Find your feature\_uri](/docs/processing/features#find-your-feature_uri-to-search).
</Warning>
```bash Create + execute (every recipe uses this) theme={null}
# Create the retriever (body = one of the recipes below)
curl -sS -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" -d @recipe.json
# -> returns { "retriever_id": "ret_..." }
# Execute it with runtime inputs
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "inputs": { "query": "people discussing electric vehicles" } }'
```
***
## Multimodal RAG
Retrieve context for an LLM across a video's **visual** content and **transcript**, fuse with RRF, rerank, and format into a single prompt-ready context block.
```json theme={null}
{
"retriever_name": "multimodal-rag",
"collection_identifiers": ["col_videos"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"fusion": "rrf",
"final_top_k": 50,
"searches": [
{ "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 },
{ "feature_uri": "mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
]
}
}
},
{ "stage_name": "rerank", "stage_type": "sort",
"config": { "stage_id": "rerank", "parameters": { "inference_name": "BAAI__bge_reranker_v2_m3", "query": "{{INPUT.query}}", "top_k": 10 } } },
{ "stage_name": "prepare", "stage_type": "apply",
"config": { "stage_id": "rag_prepare", "parameters": { "max_tokens": 8000, "output_mode": "single_context", "citation": { "style": "numbered" } } } }
]
}
```
**Result:** the 10 most relevant moments, reranked, formatted into a cited context block ready to paste into an LLM prompt.
***
## Hybrid Search (dense + keyword/BM25)
Fuse semantic recall (dense vectors) with exact-keyword precision (BM25) under RRF — so brand names, SKUs, and prices like `$9.99` still match. Requires a `text` payload index (see [Text Indexes](/docs/vector-store/namespaces#text-indexes-bm25)).
```json theme={null}
{
"retriever_name": "hybrid-search",
"collection_identifiers": ["col_docs"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{
"stage_name": "hybrid",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"fusion": "rrf",
"final_top_k": 25,
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 },
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "lexical": true, "top_k": 100 }
]
}
}
}
]
}
```
**Result:** 25 results ranked by fused semantic + keyword relevance. Use `rrf` here — it ranks by position, immune to the cosine-vs-BM25 score-scale mismatch.
***
## Search Across Languages (cross-lingual)
Mixpeek's text and multimodal embeddings are **multilingual** (E5-Large / Gemini, 100+ languages) and every language lands in the **same** vector space. So a query in one language retrieves semantically matching content in **any** language — no translation step, no per-language index, no language parameter. Query in English, match Farsi, Mandarin, or Spanish transcripts and captions.
```json theme={null}
{
"retriever_name": "cross-lingual-search",
"collection_identifiers": ["col_content"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{ "stage_name": "search", "stage_type": "filter",
"config": { "stage_id": "feature_search", "parameters": {
"final_top_k": 25,
"searches": [ { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } }
]
}
```
**Result:** an English query like `"discussion of sanctions"` returns segments whose transcript or caption is in another language — ranked by meaning, not keywords. For **audio/video** collections, transcription is produced in the **source** language ([universal](/docs/processing/extractors/universal) / [audio](/docs/processing/extractors/audio-sentiment) extractors; Whisper covers 99 languages) and the multilingual embedding is what makes it retrievable cross-lingually — just point `feature_uri` at that extractor's transcript embedding, e.g. `mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1`.
<Note>
**Mixpeek retrieves across languages; it does not translate.** There is no translation output field — matched text comes back in its **original language**. If you need it rendered in the reader's language (e.g. original + English side by side), append an [`llm_enrich`](/docs/retrieval/stages/llm-enrich) stage that translates each result into a new field:
```json theme={null}
{ "stage_name": "translate", "stage_type": "enrich",
"config": { "stage_id": "llm_enrich", "parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Translate the following to English, preserving names and numbers. Return only the translation:\n\n{{DOC.content}}",
"output_field": "translation" } } }
```
Each result then carries both `content` (original) and `translation` (English) for side-by-side display. This is the recommended path — reach for a [custom extractor](/docs/processing/custom-extractors) only if you need translation baked in at ingest time.
</Note>
***
## Video Moment Localization
Search a video and collapse matching segments into a handful of seekable moments with start/end timestamps.
```json theme={null}
{
"retriever_name": "video-moments",
"collection_identifiers": ["col_videos"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{ "stage_name": "search", "stage_type": "filter",
"config": { "stage_id": "feature_search", "parameters": {
"final_top_k": 200,
"searches": [ { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 } ] } } },
{ "stage_name": "moments", "stage_type": "reduce",
"config": { "stage_id": "moment_group", "parameters": {
"parent_field": "source_object_id",
"time_field": "start_time",
"merge_tolerance_ms": 2000,
"max_moments_per_parent": 5,
"score_strategy": "max",
"output_mode": "annotated" } } }
]
}
```
**Result:** per video, up to 5 merged moments with timestamps you can seek to.
<Tip>
`time_field` must point at the field holding each segment's timestamp. For a plain **text** query use the segment's own `start_time`. `query_chunks` is only present when you search with a **file** input via [query preprocessing](/docs/retrieval/stages/feature-search#query-preprocessing).
</Tip>
***
## Face Search (1:N identification)
Find every clip a person appears in by passing a reference face image as a `content` query against the face embedding.
```json theme={null}
{
"retriever_name": "face-search",
"collection_identifiers": ["col_videos"],
"input_schema": { "reference_face_url": { "type": "text", "required": true } },
"stages": [
{ "stage_name": "face", "stage_type": "filter",
"config": { "stage_id": "feature_search", "parameters": {
"final_top_k": 20,
"searches": [ { "feature_uri": "mixpeek://face_identity_extractor@v1/insightface__arcface",
"query": { "input_mode": "content", "value": "{{INPUT.reference_face_url}}" }, "top_k": 50 } ] } } }
]
}
```
**Result:** the 20 closest face matches. A cosine score above \~0.30 typically indicates the same person (see [Face Identity](/docs/processing/extractors/face-identity)).
This recipe chases **one** reference face. To match every new video against a maintained **roster of named identities** (a watchlist you add to and refine over time), build a reference collection instead — see [Bootstrap a Labeled Dataset](/docs/tutorials/bootstrap-labeled-dataset).
***
## Reverse Image Search + Dedup
Find visually similar items from *other* sources, deduplicated.
```json theme={null}
{
"retriever_name": "reverse-image",
"collection_identifiers": ["col_catalog"],
"input_schema": {
"image_url": { "type": "text", "required": true },
"exclude_source": { "type": "text" }
},
"stages": [
{ "stage_name": "similar", "stage_type": "filter",
"config": { "stage_id": "feature_search", "parameters": {
"final_top_k": 100,
"searches": [ { "feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": { "input_mode": "content", "value": "{{INPUT.image_url}}" }, "top_k": 100, "min_score": 0.7 } ] } } },
{ "stage_name": "exclude_self", "stage_type": "filter",
"config": { "stage_id": "attribute_filter", "parameters": {
"field": "metadata.source", "operator": "ne", "value": "{{INPUT.exclude_source}}" } } },
{ "stage_name": "dedupe", "stage_type": "reduce",
"config": { "stage_id": "deduplicate", "parameters": {
"strategy": "field", "fields": ["metadata.source_url"], "keep": "first" } } }
]
}
```
**Result:** unique visually-similar items, excluding the original source.
***
## Search OCR Text from Scanned PDFs
Scanned and archival PDFs are OCR'd — and VLM-corrected for low-confidence blocks — by the [document graph extractor](/docs/processing/extractors/document), which embeds each text block. So OCR'd text is searchable exactly like any other text: no separate OCR index or step. Optionally keep only high-confidence blocks, or target a layout type (e.g. only `table` blocks).
```json theme={null}
{
"retriever_name": "ocr-search",
"collection_identifiers": ["col_scanned_docs"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{ "stage_name": "search", "stage_type": "filter",
"config": { "stage_id": "feature_search", "parameters": {
"final_top_k": 25,
"searches": [ { "feature_uri": "mixpeek://document_graph_extractor@v1/intfloat__multilingual_e5_large_instruct",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } },
{ "stage_name": "high_confidence", "stage_type": "filter",
"config": { "stage_id": "attribute_filter", "parameters": {
"field": "confidence_tag", "operator": "in", "value": ["A", "B"] } } }
]
}
```
**Result:** the 25 best-matching blocks (each with `page_number`, `bbox`, `object_type`), limited to high-confidence OCR. Drop the `high_confidence` stage to include fair/poor blocks, or filter `object_type` to `table`/`form` to target structured regions.
<Tip>
For text burned into **video frames** (not PDFs), enable `run_ocr` on the [multimodal extractor](/docs/processing/extractors/multimodal) — it populates an `ocr_text` payload field you can filter on with [`attribute_filter`](/docs/retrieval/stages/attribute-filter). See also [feature\_search → Lexical (BM25)](/docs/retrieval/stages/feature-search#lexical-bm25-search).
</Tip>
***
## Search + Classify
Search, then attach an LLM/taxonomy label to each result in one pipeline.
```json theme={null}
{
"retriever_name": "search-classify",
"collection_identifiers": ["col_content"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{ "stage_name": "search", "stage_type": "filter",
"config": { "stage_id": "feature_search", "parameters": {
"final_top_k": 25,
"searches": [ { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } },
{ "stage_name": "label", "stage_type": "enrich",
"config": { "stage_id": "llm_enrich", "parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Classify this result: {{DOC.content}}",
"output_field": "label",
"output_schema": { "category": "string", "confidence": "number" } } } }
]
}
```
**Result:** 25 results, each annotated with a structured `label`. For NSFW/safety classification use the [`classify`](/docs/retrieval/stages/classify) stage; for taxonomy-backed labeling see [Taxonomies](/docs/enrichment/taxonomies).
***
## Scan a batch of inputs
Any retriever can run many queries at once — ideal for moderating an upload queue or scanning a catalog:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute/batch" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "queries": [ {"inputs": {"image_url": "https://…/a.jpg"}}, {"inputs": {"image_url": "https://…/b.jpg"}} ] }'
```
The retriever is planned once and reused across the batch ([query optimization](/docs/retrieval/query-optimization)).
## Composing recipes
Stages are independent — add, remove, or reorder them. Common extensions: append a [`rerank`](/docs/retrieval/stages/rerank) for precision, an [`attribute_filter`](/docs/retrieval/stages/attribute-filter) for metadata scoping (the optimizer pushes it down for you), or a [`rag_prepare`](/docs/retrieval/stages/rag-prepare) to format for an LLM.
# Retrievers
Source: https://docs.mixpeek.com/docs/retrieval/retrievers
Compose stage-based search pipelines over your collections
<Frame>
<img alt="Mixpeek Retrievers" />
</Frame>
Retrievers combine feature-aware search stages, structured filters, enrichment joins, and optional LLM post-processing into a single executable pipeline. Each retriever has an input schema, a list of target collections, and a deterministic set of stages executed in order.
<Card title="Build a retriever over your own data" icon="layer-group" href="https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=docs&utm_medium=docs_retrievers&utm_campaign=managed_onramp">
Create a namespace, ingest a few files, and compose your first search pipeline in the Studio — no API key setup required.
</Card>
<Note>
**Multi-stage retrieval is what makes Mixpeek a warehouse, not a database.** No other system offers composable filter → sort → reduce → enrich → apply pipelines over multimodal data. This is the query language for unstructured data — the equivalent of SQL for embeddings across modalities. See the [Retrieval Cookbook](/docs/retrieval/cookbook) for ready-to-copy pipeline configurations.
</Note>
## Anatomy of a Retriever
```json theme={null}
{
"retriever_name": "product_search_v2",
"description": "Product search with enrichment and transformation",
"collection_identifiers": ["col_products"],
"input_schema": {
"query_text": { "type": "text", "required": true },
"max_price": { "type": "number" }
},
"stages": [
{
"stage_name": "enrich_catalog",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_catalog",
"source_field": "metadata.product_id",
"target_field": "product_id",
"fields_to_merge": ["name", "price", "category"],
"output_field": "catalog_data"
}
}
},
{
"stage_name": "fetch_billing",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.stripe.com/v1/customers/{{DOC.metadata.customer_id}}",
"method": "GET",
"allowed_domains": ["api.stripe.com"],
"auth": {
"type": "bearer",
"secret_ref": "stripe_api_key"
},
"output_field": "metadata.billing",
"on_error": "skip"
}
}
},
{
"stage_name": "reshape",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"id\": \"{{DOC.document_id}}\", \"title\": \"{{DOC.metadata.title}}\", \"price\": {{DOC.catalog_data.price}}}",
"fail_on_error": false
}
}
}
],
"cache_config": {
"enabled": true,
"ttl_seconds": 300
}
}
```
## Minimal Working Example
The simplest retriever that performs a feature search. Note the required fields:
* **`collection_identifiers`** is required at the retriever root level when using `feature_search` stages — the search needs to know which collections to query. (Accepts collection names or IDs.)
* **`input_schema`** is a flat map of input field name → type definition (e.g. `{ "query": { "type": "text", "required": true } }`). It's required when your stages use `{{INPUT.*}}` template variables — Mixpeek validates inputs against this schema before execution. It is **not** JSON Schema: map each field name directly to its `{ "type": ..., "required": ... }` definition — do **not** wrap it in `{ "type": "object", "properties": { ... } }`. Field `type` is one of the bucket/input types (`text`, `string`, `number`, `boolean`, `date`, `image`, `video`, `pdf`, `document_reference`, …); `required` defaults to `false`.
* Each stage needs a **`stage_name`** and a **`config`** object containing the **`stage_id`** (e.g. `feature_search`) and its **`parameters`**. `stage_type` (the category, e.g. `filter`) is optional but recommended.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "basic_search",
"collection_identifiers": ["col_my_collection"],
"input_schema": {
"query": { "type": "text", "required": true }
},
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
}
],
"final_top_k": 20
}
}
}
]
}'
```
Execute it:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/<retriever_id>/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{"inputs": {"query": "search terms here"}, "limit": 10}'
```
## Stage Catalog
Stages are the building blocks of retriever pipelines. Each stage belongs to a **category** that defines its behavior:
| Category | Behavior | Example Use Cases |
| ---------- | ------------------------------------------------------ | --------------------------------------------------------------- |
| **filter** | Reduce the number of documents while preserving schema | Attribute filters, semantic search, hybrid search |
| **sort** | Reorder documents without changing the set | Attribute sort, score-based ordering, reranking |
| **reduce** | Collapse results into aggregated values | Top-k selection, deduplication, sampling, summarization |
| **group** | Reshape results by bucketing into logical groups | Group by field, semantic clustering |
| **apply** | Transform or restructure documents | JSON transforms, API calls, code execution, web scrape |
| **enrich** | Add knowledge to documents using AI or joins | LLM enrichment, taxonomy classification, cross-collection joins |
<Note>
Retrieve the live registry with `GET /v1/retrievers/stages`. Each entry includes `stage_id`, category, icon, and parameter schema so you can dynamically build configuration UIs or validations.
**Live stages:** [https://api.mixpeek.com/v1/retrievers/stages](https://api.mixpeek.com/v1/retrievers/stages)
</Note>
<CodeGroup>
```bash theme={null}
curl -s --request GET \
--url "$MP_API_URL/v1/retrievers/stages" \
--header "Authorization: Bearer $MP_API_KEY" \
--header "X-Namespace: $MP_NAMESPACE"
```
```json theme={null}
[
{
"stage_id": "api_call",
"description": "Enrich documents with external API calls",
"category": "apply",
"icon": "external-link"
},
{
"stage_id": "json_transform",
"description": "Transform document structure using Jinja2 templates",
"category": "apply",
"icon": "code"
},
{
"stage_id": "external_web_search",
"description": "Search the web using Exa AI-native search",
"category": "apply",
"icon": "globe"
},
{
"stage_id": "document_enrich",
"description": "Join and enrich documents with data from another collection",
"category": "enrich",
"icon": "link"
},
{
"stage_id": "cross_compare",
"description": "Multi-tier cross-collection comparison with classification",
"category": "apply",
"icon": "code-compare"
}
]
```
</CodeGroup>
### Filter Stages
Filter stages reduce the document set while preserving the document schema. Use these at the start of your pipeline to narrow down candidates.
<Note>
Use `GET /v1/retrievers/stages?category=filter` to retrieve the current list of filter stages and their parameter schemas.
</Note>
### Sort Stages
Sort stages reorder documents without changing the result set. Place these after filters to control ranking.
<Note>
Use `GET /v1/retrievers/stages?category=sort` to retrieve the current list of sort stages and their parameter schemas.
</Note>
### Reduce Stages
Reduce stages collapse results into aggregated values. Use these for deduplication, sampling, or summarization.
<Note>
Use `GET /v1/retrievers/stages?category=reduce` to retrieve the current list of reduce stages and their parameter schemas.
</Note>
### Group Stages
Group stages reshape results by bucketing documents into logical groups or clusters.
<Note>
Use `GET /v1/retrievers/stages?category=group` to retrieve the current list of group stages and their parameter schemas.
</Note>
### Apply Stages
Apply stages transform or restructure documents. Use these to reshape output, call external services, or run custom code.
| Stage ID | Description | Transformation |
| --------------------- | -------------------------------------------------------- | ---------------------------------- |
| `cross_compare` | Multi-tier comparison against a reference collection | N → M (findings) or N → N (enrich) |
| `api_call` | Enrich documents with external API calls | N → N |
| `json_transform` | Transform document structure using Jinja2 templates | N → N |
| `external_web_search` | Search the web using Exa AI-native search | 0 → M (creates documents) |
| `code_execution` | Execute custom Python/TypeScript/JavaScript in sandboxes | N → M (custom logic) |
### Enrich Stages
Enrich stages add knowledge to documents using AI models, taxonomies, or cross-collection joins.
| Stage ID | Description | Transformation |
| ----------------- | ------------------------------------------------ | ---------------------------------- |
| `llm_enrich` | Generate new fields with LLM prompts | N → N (with extracted data added) |
| `taxonomy_enrich` | Classify documents against taxonomy nodes | N → N (with taxonomy labels added) |
| `document_enrich` | Join documents with data from another collection | N → N (LEFT JOIN) |
***
## Enrich Stage Details
### document\_enrich
Joins documents with data from another collection, similar to a SQL LEFT JOIN. Each input document produces exactly one output document with added fields from the target collection.
**When to use:**
* Combine data from multiple collections (e.g., products + catalog info)
* Attach user profiles, metadata, or related entities
* Denormalize data at query time
**Parameters:**
| Parameter | Required | Description |
| ---------------------- | -------- | --------------------------------------------------------------------- |
| `target_collection_id` | Yes | Collection to join with |
| `source_field` | Yes\* | Field in current documents to match |
| `target_field` | Yes\* | Field in target collection to match against |
| `fields_to_merge` | No | Specific fields to merge (or entire document if omitted) |
| `output_field` | No | Where to place enrichment (root or nested path) |
| `retriever_id` | No | Use an existing retriever for lookup instead of direct field matching |
| `retriever_config` | No | Anonymous retriever definition for complex lookups |
| `retriever_inputs` | No | Template inputs when using retriever-based enrichment |
| `strategy` | No | `enrich` (merge fields) or `append` (add as nested object) |
| `allow_missing` | No | Keep documents without matches (default: true) |
| `when` | No | Conditional filter for selective enrichment |
| `cache_behavior` | No | `auto`, `disabled`, or `aggressive` |
| `cache_ttl_seconds` | No | Cache TTL in seconds |
\*Required for direct joins; not needed when using `retriever_id` or `retriever_config`.
**Examples:**
<CodeGroup>
```json Direct Field Join theme={null}
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_products",
"source_field": "metadata.product_id",
"target_field": "product_id",
"fields_to_merge": ["name", "price", "category"],
"output_field": "product_data"
}
}
}
```
```json Retriever-Based Enrichment theme={null}
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_similar_items",
"retriever_id": "ret_find_similar_products",
"retriever_inputs": {
"query": "{{DOC.description}}",
"category": "{{DOC.metadata.category}}"
},
"fields_to_merge": ["name", "price", "image_url"],
"output_field": "similar_products",
"strategy": "append"
}
}
}
```
```json Conditional Enrichment theme={null}
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_specs",
"source_field": "metadata.sku",
"target_field": "sku",
"fields_to_merge": ["specifications", "warranty"],
"output_field": "metadata.technical_details",
"when": {
"field": "metadata.category",
"operator": "eq",
"value": "electronics"
}
}
}
}
```
</CodeGroup>
***
### api\_call
Enriches documents by calling external HTTP APIs. Enables integration with third-party services (Stripe, GitHub, weather APIs, etc.) to augment documents with real-time data.
<Warning>
**Security**: This stage makes external HTTP requests. Always use `allowed_domains` to prevent SSRF attacks. Never store credentials directly—use `auth.secret_ref` to reference vault-stored secrets.
</Warning>
**Parameters:**
| Parameter | Required | Description |
| ------------------- | -------- | ----------------------------------------------------------------------- |
| `url` | Yes | API endpoint URL (supports `{DOC.field}` and `{INPUT.field}` templates) |
| `allowed_domains` | Yes | Domain allowlist for SSRF protection (never use `*`) |
| `output_field` | Yes | Dot-path where API response should be stored |
| `method` | No | HTTP method: GET, POST, PUT, PATCH, DELETE (default: GET) |
| `auth` | No | Authentication configuration (see below) |
| `headers` | No | Additional HTTP headers |
| `body` | No | Request body for POST/PUT/PATCH (JSON, supports templates) |
| `timeout` | No | Request timeout in seconds (1-60, default: 10) |
| `max_response_size` | No | Maximum response size in bytes (default: 10MB) |
| `response_path` | No | JSONPath to extract specific field from response |
| `rate_limit` | No | Rate limiting config (`requests_per_minute`, `requests_per_hour`) |
| `when` | No | Conditional filter for selective enrichment |
| `on_error` | No | Error handling: `skip`, `remove`, or `raise` (default: skip) |
**Authentication Types:**
| Type | Description | Required Fields |
| --------------- | --------------------------------------------- | ---------------------------------------------- |
| `none` | No authentication (public APIs) | — |
| `bearer` | Bearer token (OAuth 2.0, JWT) | `secret_ref` |
| `api_key` | API key in header or query param | `secret_ref`, `key`, `location` (header/query) |
| `basic` | HTTP Basic Auth (username:password in secret) | `secret_ref` |
| `custom_header` | Custom header with arbitrary name | `secret_ref`, `key` |
**Examples:**
<CodeGroup>
```json Stripe Customer Lookup theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.stripe.com/v1/customers/{DOC.metadata.stripe_id}",
"method": "GET",
"allowed_domains": ["api.stripe.com"],
"auth": {
"type": "bearer",
"secret_ref": "stripe_api_key"
},
"output_field": "metadata.stripe_data",
"timeout": 10,
"on_error": "skip"
}
}
}
```
```json GitHub API (Public) theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.github.com/repos/{INPUT.owner}/{INPUT.repo}",
"method": "GET",
"allowed_domains": ["api.github.com"],
"output_field": "metadata.github_info",
"response_path": "$.stargazers_count"
}
}
}
```
```json POST with API Key theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.example.com/v1/analyze",
"method": "POST",
"allowed_domains": ["api.example.com"],
"auth": {
"type": "api_key",
"key": "X-API-Key",
"location": "header",
"secret_ref": "example_api_key"
},
"headers": {
"Content-Type": "application/json"
},
"body": {
"text": "{DOC.text}",
"language": "en"
},
"output_field": "metadata.analysis"
}
}
}
```
</CodeGroup>
***
### json\_transform
Applies a Jinja2 template to each document, rendering the template with full document context and replacing the document with the parsed JSON output. Use this to reformat documents for external APIs or reshape data for downstream consumers.
**Parameters:**
| Parameter | Required | Description |
| --------------- | -------- | ------------------------------------------------------------- |
| `template` | Yes | Jinja2 template string that must render to valid JSON |
| `fail_on_error` | No | Fail entire pipeline on transformation error (default: false) |
**Template Context:**
| Namespace | Description |
| --------------------- | ----------------------------------------------------- |
| `DOC` / `doc` | Current document fields and metadata |
| `INPUT` / `inputs` | Original query inputs from the search request |
| `CONTEXT` / `context` | Execution context (namespace\_id, internal\_id, etc.) |
| `STAGE` / `stage` | Current stage execution data |
**Examples:**
<CodeGroup>
```json Field Selection theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"id\": \"{{ DOC.document_id }}\", \"content\": \"{{ DOC.text }}\", \"score\": {{ DOC.score }}}"
}
}
}
```
```json Conditional Fields theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"workflow_name\": \"process-asset\", \"inputs\": [{\"name\": \"id\", \"value\": \"{{ DOC.id }}\"}{% if DOC.asset_type == \"VIDEO\" %}, {\"name\": \"video\", \"value\": {\"src\": \"{{ DOC.url }}\"}}{% endif %}]}"
}
}
}
```
```json Array Iteration theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"title\": \"{{ DOC.title }}\", \"tags\": [{% for tag in DOC.tags %}\"{{ tag }}\"{% if not loop.last %}, {% endif %}{% endfor %}]}"
}
}
}
```
```json JSON Escaping theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"user_id\": \"{{ DOC.metadata.user_id }}\", \"raw_data\": {{ DOC.metadata.raw | tojson }}}"
}
}
}
```
</CodeGroup>
***
### external\_web\_search
Performs AI-native web search using Exa's neural ranking system. Creates new documents from web search results, enabling retriever pipelines to incorporate real-time internet content.
<Note>
This stage **creates** new documents (0 → M transformation) rather than enriching existing ones. Use it at the start of a pipeline or to augment internal results with external web sources.
</Note>
**Parameters:**
| Parameter | Required | Description |
| ---------------------- | -------- | --------------------------------------------------------------------------------------- |
| `query` | Yes | Search query (supports `{INPUT.field}` and `{DOC.field}` templates) |
| `num_results` | No | Number of results (1-100, default: 10) |
| `use_autoprompt` | No | Enable Exa's query enhancement (default: true) |
| `start_published_date` | No | Filter by publication date (YYYY-MM-DD format) |
| `category` | No | Content category: `research paper`, `news`, `github`, `tweet`, `blog`, `company`, `pdf` |
| `include_text` | No | Include text snippets in results (default: true) |
**Output Schema:**
Each result becomes a document with:
* `metadata.url` – Web page URL
* `metadata.title` – Page title
* `metadata.text` – Text snippet (if `include_text=true`)
* `metadata.published_date` – Publication date (if available)
* `metadata.author` – Author name (if available)
* `metadata.search_query` – Original query used
* `metadata.search_position` – 0-indexed position in results
* `score` – Exa relevance score
**Examples:**
<CodeGroup>
```json Basic Web Search theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{INPUT.query}",
"num_results": 10,
"include_text": true,
"use_autoprompt": true
}
}
}
```
```json Research Papers theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "neural network architectures",
"num_results": 20,
"category": "research paper",
"include_text": true
}
}
}
```
```json Recent News theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{INPUT.company_name} latest product launches",
"num_results": 5,
"category": "news",
"start_published_date": "2024-10-01",
"include_text": true
}
}
}
```
</CodeGroup>
***
### cross\_compare
Compares source documents against a reference collection using a cascading match strategy (exact → fuzzy → semantic → visual). Each comparison produces a classified finding with a score and confidence level. Ideal for drift detection, deduplication, and compliance checking.
<Note>
This stage can output either **findings** (N → M, one document per comparison) or **enriched documents** (N → N, results attached as a field). Set `output_mode` to control this behavior.
</Note>
**Parameters:**
| Parameter | Required | Description |
| --------------------------- | -------- | ------------------------------------------------------------------------------- |
| `reference_collection_id` | Yes | Collection containing reference documents to compare against |
| `source_field` | No | Field on source documents to extract comparison elements (default: `content`) |
| `reference_field` | No | Field on reference documents containing comparison content (default: `content`) |
| `extraction_mode` | No | Element extraction: `raw`, `lines`, `labels`, `list` (default: `raw`) |
| `match_tiers` | No | Ordered matching cascade (default: `["exact", "fuzzy"]`) |
| `fuzzy_threshold` | No | Minimum fuzzy score (default: 0.75) |
| `semantic_threshold` | No | Minimum semantic similarity (default: 0.85) |
| `visual_threshold` | No | Minimum visual similarity (default: 0.55) |
| `classifications` | No | Score-to-label mapping rules (see below) |
| `output_mode` | No | `findings` (N→M) or `enrich` (N→N) (default: `findings`) |
| `output_field` | No | Field name for enrich mode results (default: `comparison_results`) |
| `include_visual_comparison` | No | Enable DINOv2 + SigLIP visual comparison (default: false) |
| `reference_limit` | No | Max reference documents to fetch (default: 200) |
| `source_doc_type_filter` | No | Only process source docs with this doc\_type |
**Examples:**
<CodeGroup>
```json Content Drift Detection theme={null}
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_documentation",
"source_field": "content",
"reference_field": "content",
"extraction_mode": "labels",
"match_tiers": ["exact", "fuzzy", "semantic"],
"include_visual_comparison": true,
"classifications": [
{"min_score": 0.95, "label": "current"},
{"min_score": 0.75, "label": "needs_review"},
{"min_score": 0.0, "label": "outdated"}
]
}
}
}
```
```json Catalog Matching (Enrich Mode) theme={null}
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_internal_catalog",
"source_field": "product_name",
"reference_field": "product_name",
"match_tiers": ["exact", "fuzzy"],
"fuzzy_threshold": 0.80,
"output_mode": "enrich",
"output_field": "catalog_match",
"classifications": [
{"min_score": 0.95, "label": "exact_match"},
{"min_score": 0.80, "label": "likely_match"},
{"min_score": 0.0, "label": "no_match"}
]
}
}
}
```
```json Deduplication Check theme={null}
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_existing_corpus",
"source_field": "content",
"reference_field": "content",
"extraction_mode": "lines",
"match_tiers": ["exact", "fuzzy", "semantic"],
"semantic_threshold": 0.90,
"output_mode": "enrich",
"output_field": "duplication_analysis",
"classifications": [
{"min_score": 0.95, "label": "duplicate"},
{"min_score": 0.80, "label": "near_duplicate"},
{"min_score": 0.0, "label": "unique"}
]
}
}
}
```
</CodeGroup>
***
Call `GET /v1/retrievers/stages` to retrieve the latest stage metadata and parameter schemas.
## Execution Lifecycle
1. **Validate Inputs** – Mixpeek enforces the retriever’s `input_schema`.
2. **Walk Stages** – Each stage receives the current working set, runs, and outputs a new set.
3. **Apply Pagination** – `offset`, `cursor`, `scroll`, or `keyset` pagination is handled after the final stage.
4. **Return Telemetry** – Responses include `status`, `warnings`, `stage_statistics`, and `budget`.
<Warning>
**An empty result can mean success or silent failure. Read `warnings`.**
A true no-match, a mismatched `feature_uri`, and a stage that timed out all
return HTTP 200 with an empty `documents` array. You cannot tell them apart
from `documents` alone.
`warnings` is populated when a search returns zero for a reason the platform
detected, and it is the only such signal in the response. Read it as evidence
that something is off. Confirm the cause yourself before acting on the text,
because a warning can name the wrong cause and prescribe an expensive remedy.
Confirm cheaply first. Re-read your `feature_uri` against
`GET /v1/collections/{id}` → `vector_indexes[].feature_uri`, and run a query
you know works against the same collection. A collection that answers one
query and not another is not missing its vectors.
Read `warnings`, check `status`, and verify before you treat an empty result
as an answer. A wrong `feature_uri` is the most common cause; see
[Retriever returning zero results](/docs/resources/troubleshooting#retriever-returning-zero-results).
</Warning>
<Note>
**Presigned media URLs are short-lived.** Media fields in a result
(`thumbnail_url`, `video_segment_url`, `face_crop_url`, and similar) come back
as signed URLs minted fresh on each request, and they **expire after 24 hours**.
Do not persist or hard-code a URL value. Re-run the query for a fresh one, or
fetch the blob through the document API.
Retriever execute signs these whatever you pass for `return_presigned_urls`,
which is not read on this path. Some values still come back as raw `gs://` or
`s3://`: anything under `metadata`, values nested inside dicts or lists on a
document carrying a `collection_id`, and the fields `original_url`,
`source_blobs` and `presigned_urls`. See
[Media URLs](/docs/vector-store/documents#media-urls) for the rule across endpoints.
</Note>
Response headers include:
* `ETag` – cache validator; pair with `If-None-Match` for 304 responses.
* `Cache-Control` – TTL derived from `cache_config`.
* `X-Cache` – `HIT` or `MISS` for query-level caching.
## Filters & Templates
Structured filters support comparison operators (`eq`, `gt`, `lte`, `in`, etc.) and logical composition (`AND`, `OR`, `NOT`).
### Template Namespaces
Stages support dynamic configuration through template expressions using Jinja2 syntax. Both uppercase and lowercase namespace formats are supported and work identically:
| Namespace | Description | Examples |
| --------------------- | ---------------------------------------------------- | ------------------------------------------------------------- |
| `INPUT` / `inputs` | User-provided query parameters and inputs | `{{INPUT.query_text}}`, `{{inputs.max_price}}` |
| `DOC` / `doc` | Current document fields (for per-document logic) | `{{DOC.metadata.category}}`, `{{doc.content_type}}` |
| `CONTEXT` / `context` | Execution state (budget, timing, retriever metadata) | `{{CONTEXT.budget_remaining}}`, `{{context.time_elapsed_ms}}` |
| `STAGE` / `stage` | Previous stage outputs (for cascading logic) | `{{STAGE.hybrid_search.top_score}}`, `{{stage.filter.count}}` |
<Note>
Mixed usage within the same stage is supported. For example, you can use `{{INPUT.query}}` alongside `{{context.budget_remaining}}` in the same configuration.
</Note>
**Conditional expressions:**
```json theme={null}
{
"batch_size": "{{CONTEXT.budget_remaining > 50 ? 200 : 50}}",
"field": "{{DOC.media_type == 'image' ? 'image_url' : 'video_url'}}"
}
```
**Templated batch size:**
```json theme={null}
{
"batch_size": "{{20 * inputs.page_size}}"
}
```
## Retrievers & Caching
* **Query cache** – caches entire responses keyed by inputs, filters, pagination, and collection index signatures.
* **Stage cache** – reuse outputs of expensive stages by listing them under `cache_stage_names`.
* **Inference cache** – Engine deduplicates identical model calls.
Use `GET /v1/analytics/retrievers/{id}/cache-performance` to monitor hit rates and latency improvements.
## Pagination Options
Pass a `pagination` object. Each method takes its own keys, and `method` is required.
| Method | Keys (default) | Use case |
| -------- | ------------------------------------------------------------------- | --------------------------------------- |
| `offset` | `page_size` (10), `page_number` (1) | Jump to a numbered page |
| `cursor` | `limit` (10), `cursor` (null on first call) | Stable iteration over large result sets |
| `scroll` | `limit` (100), `scroll_id` (null on first call), `scroll_ttl` (300) | Deep pagination for analytics workloads |
| `keyset` | `limit` (10), `after` (null on first call) | Paginated browsing, requires a sort key |
```json theme={null}
{ "inputs": { "query_text": "..." },
"pagination": { "method": "offset", "page_size": 10, "page_number": 1 } }
```
<Warning>
The `offset` method takes `page_size` and `page_number`. It does **not** take
`offset` or `limit`, despite the method name. Unknown keys inside `pagination`
return a 422 that names the valid keys for the method you chose.
A top-level `offset` beside `inputs` is not a request field, so it is ignored
rather than rejected. Every page then returns the default page, which reads as
duplicate documents rather than as an error.
</Warning>
Pagination is only available on the named-retriever path,
`POST /v1/retrievers/{retriever_id}/execute`. The adhoc path,
`POST /v1/retrievers/execute`, accepts `collection_identifiers`, `input_schema`,
`stages`, `inputs`, `budget_limits`, and `stream` — no `pagination` and no `limit`.
Build a named retriever when you need to page.
## Execute a Retriever
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/<retriever_id>/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"inputs": {
"query_text": "wireless earbuds",
"max_price": 150
},
"filters": {
"field": "metadata.category",
"operator": "eq",
"value": "audio"
},
"limit": 10,
"return_urls": true,
"return_vectors": false,
"session_id": "sess_123"
}'
```
Response snippet:
```json theme={null}
{
"execution_id": "exec_b8f31e0c",
"documents": [...],
"stage_statistics": {
"hybrid_search": { "duration_ms": 180, "cache_hit": true },
"filter": { "duration_ms": 8 },
"rerank": { "duration_ms": 120 }
},
"budget": {
"credits_used": 12.4,
"credits_limit": 100,
"time_elapsed_ms": 310
}
}
```
<Note>
`credits_used` and `credits_limit` are internal ledger units (1 credit = \$0.001). Dollar-based usage and the rate card live in [Billing](/docs/platform/billing).
</Note>
## Batch Execution
Execute a retriever against multiple inputs in a single request. The retriever is fetched and optimized once, then executed concurrently across all queries with bounded parallelism.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/<retriever_id>/execute/batch" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"queries": [
{"inputs": {"image": "https://example.com/suspect-1.jpg"}},
{"inputs": {"image": "https://example.com/suspect-2.jpg"}},
{"inputs": {"image": "https://example.com/suspect-3.jpg"}}
],
"concurrency": 5
}'
```
```python Python theme={null}
results = client.retrievers.execute_batch(
retriever_id="ret_abc123",
queries=[
{"inputs": {"image": "https://example.com/suspect-1.jpg"}},
{"inputs": {"image": "https://example.com/suspect-2.jpg"}},
{"inputs": {"image": "https://example.com/suspect-3.jpg"}},
],
concurrency=5,
)
for result in results:
print(f"Query {result.query_index}: {result.status}, {len(result.documents)} docs")
```
</CodeGroup>
| Parameter | Type | Default | Description |
| ------------- | ------- | -------- | -------------------------------------------------------------------- |
| `queries` | array | required | 1–50 query objects, each with `inputs` and optional `filters` |
| `concurrency` | integer | `5` | Max parallel executions (1–20) |
| `settings` | object | `null` | Shared settings applied to every query (e.g., `limit`, `max_chunks`) |
| `stream` | boolean | `false` | Stream results via Server-Sent Events |
Response:
```json theme={null}
{
"retriever_id": "ret_abc123",
"total_queries": 3,
"completed": 3,
"failed": 0,
"results": [
{
"query_index": 0,
"status": "completed",
"documents": [...],
"execution_id": "exec_a1b2c3"
}
],
"total_duration_ms": 61200
}
```
### Streaming
For pipelines with LLM stages that take longer than a few seconds per query, use `"stream": true` to receive results as each query completes:
```bash theme={null}
curl -N -X POST "$MP_API_URL/v1/retrievers/<retriever_id>/execute/batch" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"queries": [
{"inputs": {"image": "https://example.com/suspect-1.jpg"}},
{"inputs": {"image": "https://example.com/suspect-2.jpg"}}
],
"concurrency": 5,
"stream": true
}'
```
The response is an SSE stream:
```
: stream-start
: keepalive
data: {"event_type":"query_complete","query_index":0,"total_queries":2,"status":"completed","documents":[...]}
data: {"event_type":"query_complete","query_index":1,"total_queries":2,"status":"completed","documents":[...]}
data: {"event_type":"batch_complete","retriever_id":"ret_abc123","total_queries":2,"completed":2,"failed":0,"total_duration_ms":61200}
```
Keepalive comments (`: keepalive`) are sent every 15 seconds to keep the connection alive through proxies. Results arrive out of order as each query finishes — use `query_index` to match results to inputs.
### When to use batch
| Scenario | Approach |
| ------------------------------------------- | -------------------------------------------------------------------- |
| Single query | `/execute` |
| 2–50 queries, fast pipeline (under 10s) | `/execute/batch` |
| 2–50 queries, LLM-heavy pipeline (over 10s) | `/execute/batch` with `"stream": true` |
| 50+ queries | Split into batches of 50, call sequentially or from multiple threads |
## Anonymous Retrievers
Anonymous retrievers let you execute a retriever pipeline in a single API call without persisting it. This is ideal for:
* **Prototyping and experimentation** – Test stage configurations quickly without cluttering your retriever list
* **Dynamic pipelines** – Build pipelines on-the-fly based on user context or application logic
* **One-off queries** – Run complex searches that don't need to be reused
* **CI/CD testing** – Validate retriever configurations in automated tests without creating permanent resources
<Tip>
Use anonymous retrievers during development to iterate quickly, then promote working configurations to named retrievers for production use.
</Tip>
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_identifiers": ["col_products"],
"input_schema": {
"query_text": { "type": "text", "required": true }
},
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query_text}}" },
"top_k": 50
}
],
"final_top_k": 50
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3"
}
}
}
],
"inputs": {
"query_text": "noise cancelling headphones"
}
}'
```
The response format is identical to named retriever execution. The key difference: no `retriever_id` is created or stored.
## Publishing & Display Config
Retrievers can be published as public search interfaces hosted at `mxp.co`. Publishing requires a `display_config` that controls how the UI renders inputs, results, and styling.
Set `display_config` when creating or updating a retriever via `PATCH /v1/retrievers/{id}`:
```json theme={null}
{
"display_config": {
"title": "Product Search",
"description": "Search through thousands of products",
"logo_url": "https://example.com/logo.png",
"theme": {
"primary_color": "#4F46E5",
"mode": "light"
},
"inputs": [
{
"field_name": "query_text",
"field_schema": { "type": "text", "required": true }
}
],
"exposed_fields": ["title", "thumbnail_url", "price", "category"],
"layout": {
"results_layout": "grid",
"columns": 3
},
"field_config": {
"thumbnail_url": {
"format": "image",
"format_options": { "width": 400, "height": 300, "aspect_ratio": "16/9" }
},
"price": {
"format": "number",
"format_options": { "label": "Price", "decimals": 2, "prefix": "$" }
}
}
}
}
```
**Key fields:**
| Field | Required | Description |
| ---------------- | -------- | -------------------------------------------------------------------------------------- |
| `title` | Yes | Page heading for the public search interface |
| `description` | No | Subtitle/description text |
| `logo_url` | No | URL to logo image |
| `inputs` | Yes | List of input fields to render (maps to `input_schema`) |
| `exposed_fields` | Yes | Document fields shown in results (at least one required) |
| `layout` | No | Result layout: `grid`, `list`, or `table` with column count |
| `theme` | No | Primary color, dark/light mode |
| `field_config` | No | Per-field display format: `text`, `image`, `date`, `number`, `url`, `boolean`, `array` |
| `seo` | No | SEO metadata (auto-generated from title/description if omitted) |
| `template_type` | No | Built-in template: `portrait-gallery`, `media-search`, `document-search` |
| `field_mappings` | No | Maps template slots (e.g., `thumbnail`, `title`) to actual field names |
Once `display_config` is set, publish the retriever with `POST /v1/retrievers/{id}/publish`.
***
## Maintenance & Versioning
* Use `PATCH /v1/retrievers/{id}` to rename retrievers or adjust cache settings (stages and schema are immutable; create a new retriever for breaking changes).
* List retrievers with filters, search, and sort: `POST /v1/retrievers/list`.
* Retrieve execution history: `GET /v1/retrievers/{id}/executions`.
* Diagnose pipelines without executing: `POST /v1/retrievers/{id}/explain`.
## Interaction Feedback
Capture user feedback with `/v1/retrievers/interactions` to power downstream analytics, learning-to-rank, or personalized retrieval:
```json theme={null}
{
"feature_id": "doc_abc123",
"interaction_type": ["click", "long_view"],
"position": 2,
"metadata": { "duration_ms": 12000 },
"user_id": "user_456",
"session_id": "sess_xyz789"
}
```
## Best Practices
1. **Start narrow** – run a single search stage before adding rerankers or joins.
2. **Push filters early** – stage-level filters shrink the candidate set before expensive operations.
3. **Use JOIN strategies wisely** – `direct` for key-based joins, `retriever` for similarity joins; set `join_strategy` to control merge behavior.
4. **Enable caching** – stage caching plus query caching dramatically reduces latency for repeat queries.
5. **Monitor analytics** – use retriever analytics endpoints to optimize parameters, detect slow stages, and understand cache ROI.
Retrievers turn Mixpeek’s primitives—features, taxonomies, clusters, and models—into end-user search experiences. Configure once, execute anywhere, and evolve the pipeline with confidence.
# Agent Search
Source: https://docs.mixpeek.com/docs/retrieval/stages/agent-search
LLM-driven multi-step retrieval with iterative reasoning and tool orchestration
<Frame>
<img alt="Agent Search stage showing LLM reasoning loop with retriever stages as tools" />
</Frame>
The Agent Search stage uses an LLM reasoning loop to orchestrate other retriever stages as callable tools. Instead of executing a fixed sequence of stages, the LLM dynamically decides which stages to invoke, with what arguments, and how many iterations to perform based on the query and intermediate results.
<Note>
**Stage Category**: FILTER (Adaptive retrieval)
**Transformation**: Query → LLM reasoning (1-N iterations) → Refined documents
</Note>
## When to Use
| Use Case | Description |
| ----------------------------- | ---------------------------------------------------------- |
| **Multi-hop queries** | Questions requiring following references across documents |
| **Iterative refinement** | Broad search followed by intelligent narrowing |
| **Complex filtering logic** | When the right filter conditions depend on query semantics |
| **Tree/hierarchy navigation** | Navigating hierarchical document indexes top-down |
| **Exploratory search** | When the best search strategy isn't known upfront |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------------- | ------------------------ |
| Simple keyword or vector search | `feature_search` |
| Known metadata filter conditions | `attribute_filter` |
| Fixed pipeline with known stages | Chain stages directly |
| Latency-critical applications (\< 1s) | Direct stage execution |
| Cost-sensitive high-volume queries | Pre-configured pipelines |
## Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `strategy` | string | `iterative_refinement` | Reasoning strategy (see Strategies below) |
| `stages` | list\[string] | *from strategy* | Which retriever stages the agent can invoke as tools |
| `system_prompt` | string | *from strategy* | Custom system prompt for the LLM. Supports `{{INPUT.*}}` variables |
| `max_iterations` | integer | `5` | Maximum reasoning iterations (1-20) |
| `timeout_seconds` | float | `60.0` | Total timeout for the reasoning loop (5-300s) |
| `provider` | string | *auto* | LLM provider (openai, google, anthropic) |
| `model_name` | string | *auto* | Specific model to use |
| `feedback` | string | `null` | Feedback from a prior execution to inject into the agent's prompt. Use to correct or refine behavior based on previous results |
| `min_confidence` | float | `0.0` | Minimum confidence (0.0–1.0) required to accept results. Below this threshold, the agent keeps searching even if it signals done |
| `auto_strategy` | boolean | `false` | When true, a lightweight LLM call selects the best strategy automatically before the main loop. Adds \~0.5–1s latency |
## Strategies
| Strategy | Default Tools | Best For |
| ---------------------- | ----------------------------------------------- | -------------------------------------------------------------- |
| `iterative_refinement` | feature\_search, attribute\_filter | Progressive narrowing of results |
| `multi_hop` | feature\_search, attribute\_filter, llm\_filter | Following cross-document references |
| `tree_navigation` | feature\_search, attribute\_filter | Hierarchical index traversal |
| `full_catalog` | *every registered stage* | Open-ended queries where the model should compose the pipeline |
| `custom` | *user-specified* | Full control over tools and prompt |
## Available Tools (Stages)
The agent can invoke any registered retriever stage as a tool. Each stage is presented to the LLM with a simplified parameter schema:
| Tool | What the LLM Sees | Typical Use |
| ------------------ | ------------------------------------------------ | ---------------------- |
| `feature_search` | Semantic similarity search with query and top\_k | Vector search |
| `attribute_filter` | Filter by field, operator, and value | Metadata filtering |
| `llm_filter` | Filter by natural language criteria | Semantic filtering |
| `rerank` | Re-score documents by relevance to query | Result refinement |
| `llm_enrich` | Generate new fields using LLM analysis | Information extraction |
| `taxonomy_enrich` | Classify documents against a taxonomy | Categorization |
## Configuration Examples
<CodeGroup>
```json Iterative Refinement theme={null}
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "iterative_refinement",
"max_iterations": 5,
"timeout_seconds": 30.0
}
}
}
```
```json Custom Strategy with Specific Tools theme={null}
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "custom",
"stages": ["feature_search", "attribute_filter", "rerank"],
"system_prompt": "You are a document retrieval agent. Start with feature_search to find initial matches, then use attribute_filter to narrow by metadata, and rerank for final ordering.",
"max_iterations": 4,
"timeout_seconds": 45.0
}
}
}
```
```json Multi-Hop with LLM Filter theme={null}
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "multi_hop",
"max_iterations": 7,
"timeout_seconds": 60.0
}
}
}
```
```json Tree Navigation theme={null}
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "tree_navigation",
"max_iterations": 10,
"timeout_seconds": 90.0
}
}
}
```
```json With Feedback + Confidence Threshold theme={null}
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "iterative_refinement",
"max_iterations": 5,
"timeout_seconds": 45.0,
"min_confidence": 0.7,
"feedback": "Previous results included marketing documents — focus only on technical specs."
}
}
}
```
```json Auto-Strategy Selection theme={null}
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"auto_strategy": true,
"max_iterations": 5,
"timeout_seconds": 60.0
}
}
}
```
```json Full Catalog (model composes pipeline) theme={null}
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "full_catalog",
"max_iterations": 8,
"timeout_seconds": 90.0
}
}
}
```
</CodeGroup>
<Info>
`full_catalog` hands the LLM every registered filter/sort/rerank/enrich stage as a tool and lets it decide the pipeline shape per query. Higher cost and latency than the pinned strategies — use it when composition quality matters more than cost.
</Info>
<Tip>
Use the `custom` strategy when you need precise control over which tools the agent can access and how it should reason. The built-in strategies provide sensible defaults for common patterns.
</Tip>
<Tip>
Use `feedback` to create a human-in-the-loop refinement cycle: execute a retriever, review the results, then re-execute with feedback describing what was wrong. The agent adjusts its strategy based on your corrections. Pair with `min_confidence` to ensure the agent keeps searching until results meet your quality bar.
</Tip>
## How It Works
1. **Strategy selection**: If `auto_strategy` is enabled, a lightweight LLM call picks the best strategy for the query
2. **Prompt assembly**: The system prompt is built from the strategy defaults, with `feedback` prepended if provided
3. **Reasoning loop**: Each iteration, the LLM receives the query, available tools, and a budget note showing remaining iterations and seconds
4. **Tool execution**: The LLM calls retriever stages as tools. Results are summarized and fed back
5. **Confidence gating**: The LLM calls `finish_search` to declare done with a confidence score (0.0–1.0). If confidence is below `min_confidence`, the loop continues
6. **Context compression**: When the conversation grows long, older messages are replaced with a compact working-memory summary to stay within context limits
7. **Return**: Accumulated results and metadata (confidence, summary, reasoning trace) are returned
Each tool call creates a sub-state execution of the target stage, inheriting namespace and collection context from the parent. Non-empty results from each iteration replace the previous working set. If a refinement query returns zero results, the previous results are preserved.
## Performance
| Metric | Value |
| -------------- | --------------------------------------------- |
| **Latency** | 2-30s (depends on iterations and sub-stages) |
| **Memory** | O(N) per iteration result set |
| **Cost** | LLM API calls + sub-stage costs per iteration |
| **Complexity** | O(iterations \* sub-stage complexity) |
## Common Pipeline Patterns
### Agent as First Stage
```json theme={null}
[
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "iterative_refinement",
"stages": ["feature_search", "attribute_filter"],
"max_iterations": 3
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "content"
}
}
},
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 10
}
}
}
]
```
### Complex Query Decomposition
```json theme={null}
[
{
"stage_name": "agent_search",
"stage_type": "filter",
"config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "custom",
"stages": ["feature_search", "attribute_filter", "llm_filter"],
"system_prompt": "Break down the user's complex query into sub-queries. Use feature_search for each sub-query, attribute_filter to narrow by metadata, and llm_filter to validate relevance. Combine the best results.",
"max_iterations": 5,
"timeout_seconds": 45.0
}
}
},
{
"stage_name": "deduplicate",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "field",
"fields": ["document_id"]
}
}
}
]
```
## Response Metadata
The stage returns execution metadata in `stage_statistics`:
| Field | Description |
| -------------------- | ----------------------------------------------------------- |
| `iterations_used` | Number of reasoning iterations completed |
| `max_iterations` | Maximum iterations that were configured |
| `timeout_hit` | Whether the timeout was reached |
| `strategy` | Strategy that was used |
| `auto_strategy_used` | Whether auto-strategy routing was active |
| `stages_invoked` | List of tool calls made (name + arguments + result count) |
| `total_llm_cost` | Total LLM API cost for the reasoning loop |
| `model_used` | LLM model that was used |
| `final_confidence` | Agent's self-reported confidence in results (0.0–1.0) |
| `final_summary` | Agent's explanation of what was found and why it stopped |
| `reasoning_trace` | Per-iteration trace: agent reasoning, tool calls, durations |
## Error Handling
| Error | Behavior |
| --------------------------- | ---------------------------------------------------------------------------- |
| Sub-stage execution fails | Error message returned to LLM; it can retry or try a different approach |
| Timeout reached | Returns results accumulated so far (graceful degradation) |
| Max iterations reached | Runs a recovery call to capture confidence, then returns accumulated results |
| Unknown stage in tools list | Stage is skipped (logged as warning) |
| LLM returns no tool calls | Runs a recovery call to capture confidence, then returns current results |
| Empty collection | Returns empty result set (no error) |
## Related
* [Feature Search](/docs/retrieval/stages/feature-search) - Vector similarity search (commonly used as agent tool)
* [Attribute Filter](/docs/retrieval/stages/attribute-filter) - Metadata filtering (commonly used as agent tool)
* [LLM Filter](/docs/retrieval/stages/llm-filter) - Natural language filtering
* [Rerank](/docs/retrieval/stages/rerank) - Result re-scoring by relevance
# Aggregate
Source: https://docs.mixpeek.com/docs/retrieval/stages/aggregate
Compute statistical aggregations and metrics across document results
<Frame>
<img alt="Aggregate stage showing statistical computations on search results" />
</Frame>
The Aggregate stage computes statistical aggregations across your search results, including counts, sums, averages, min/max values, and custom metrics. This is useful for analytics, faceted search, and understanding result distributions.
<Note>
**Stage Category**: REDUCE (Aggregates results)
**Transformation**: N documents → aggregation results + optional documents
</Note>
<Info>
**Aggregating a whole collection?** The collection-level endpoint
`POST /v1/collections/{collection_id}/aggregate` runs the same functions over
every matching document (not just the current pipeline's results) and returns
**exact** counts at any collection size — `count`, `count_distinct`, `sum`,
`avg`, `min`, and `max` stream without a row cap. Pair it with the `is_null`
filter operator for field-presence / data-validity checks, e.g. count how many
documents are missing an ancestor (`from_collection`) field across 180k+ assets.
</Info>
## When to Use
| Use Case | Description |
| ------------------ | --------------------------------- |
| **Faceted search** | Count documents by category |
| **Analytics** | Compute metrics across results |
| **Price ranges** | Min/max/avg calculations |
| **Distributions** | Understand result characteristics |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ----------------------- |
| Grouping with full docs | `group_by` |
| Simple counting | Use search facets |
| Per-document stats | `code_execution` |
| LLM-based analysis | `summarize` |
## Parameters
| Parameter | Type | Default | Description |
| ------------------- | ------- | ---------- | ------------------------------------ |
| `aggregations` | array | *Required* | List of aggregation operations |
| `group_by` | string | *none* | Field to group aggregations by |
| `include_documents` | boolean | `false` | Include original documents in output |
## Aggregation Types
| Type | Description | Example |
| --------------- | --------------------------------- | ---------------------- |
| `count` | Number of documents | Total results |
| `sum` | Sum of field values | Total revenue |
| `avg` | Average value | Average price |
| `min` | Minimum value | Lowest price |
| `max` | Maximum value | Highest price |
| `cardinality` | Unique values count | Unique authors |
| `percentile` | Percentile values | P50, P95 |
| `histogram` | Value distribution | Price buckets |
| `stddev` | Standard deviation (sample) | Score spread |
| `variance` | Variance (sample) | Price volatility |
| `frequency` | Value frequency distribution | Top categories |
| `co_occurrence` | Co-occurrence of two fields | Brand + category pairs |
| `correlation` | Pearson correlation of two fields | Price vs. rating |
## Configuration Examples
<CodeGroup>
```json Basic Aggregations theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "count", "name": "total"},
{"type": "avg", "field": "metadata.price", "name": "avg_price"},
{"type": "min", "field": "metadata.price", "name": "min_price"},
{"type": "max", "field": "metadata.price", "name": "max_price"}
]
}
}
}
```
```json Grouped Aggregations theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "count", "name": "count"},
{"type": "avg", "field": "metadata.rating", "name": "avg_rating"}
],
"group_by": "metadata.category"
}
}
}
```
```json Faceted Search theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "count", "name": "total"},
{"type": "cardinality", "field": "metadata.brand", "name": "brand_count"},
{"type": "histogram", "field": "metadata.price", "interval": 50, "name": "price_ranges"}
],
"include_documents": true
}
}
}
```
```json Percentile Analysis theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "percentile", "field": "score", "percentiles": [25, 50, 75, 95], "name": "score_distribution"},
{"type": "avg", "field": "score", "name": "avg_score"}
]
}
}
}
```
```json Multi-Field Aggregations theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "count", "name": "total_docs"},
{"type": "sum", "field": "metadata.views", "name": "total_views"},
{"type": "avg", "field": "metadata.engagement_rate", "name": "avg_engagement"},
{"type": "cardinality", "field": "metadata.author_id", "name": "unique_authors"}
]
}
}
}
```
</CodeGroup>
## Output Schema
### Without Group By
```json theme={null}
{
"aggregations": {
"total": 150,
"avg_price": 49.99,
"min_price": 9.99,
"max_price": 199.99
},
"documents": [] // if include_documents: false
}
```
### With Group By
```json theme={null}
{
"aggregations": {
"electronics": {
"count": 45,
"avg_rating": 4.2
},
"clothing": {
"count": 62,
"avg_rating": 4.5
},
"books": {
"count": 43,
"avg_rating": 4.7
}
}
}
```
### Histogram Output
```json theme={null}
{
"aggregations": {
"price_ranges": {
"buckets": [
{"key": "0-50", "count": 45},
{"key": "50-100", "count": 62},
{"key": "100-150", "count": 28},
{"key": "150-200", "count": 15}
]
}
}
}
```
## Performance
| Metric | Value |
| --------------- | ------------------------------- |
| **Latency** | 5-50ms |
| **Memory** | O(groups × aggregations) |
| **Cost** | Free |
| **Scalability** | Efficient for large result sets |
## Common Pipeline Patterns
### Search with Facets
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 1000 }
],
"final_top_k": 1000
}
}
},
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "count", "name": "total"},
{"type": "histogram", "field": "metadata.category", "name": "categories"},
{"type": "histogram", "field": "metadata.price", "interval": 25, "name": "price_ranges"}
],
"include_documents": true
}
}
}
]
```
### Analytics Pipeline
```json theme={null}
[
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "metadata.date", "operator": "gte", "value": "2024-01-01"},
{"field": "metadata.status", "operator": "eq", "value": "published"}
]
}
}
}
},
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "count", "name": "total_published"},
{"type": "sum", "field": "metadata.views", "name": "total_views"},
{"type": "avg", "field": "metadata.engagement", "name": "avg_engagement"},
{"type": "percentile", "field": "metadata.views", "percentiles": [50, 90, 99], "name": "view_distribution"}
],
"group_by": "metadata.author"
}
}
}
]
```
### E-Commerce Product Analytics
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 500 }
],
"final_top_k": 500
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.in_stock",
"operator": "eq",
"value": true
}
}
}
},
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "count", "name": "available_products"},
{"type": "min", "field": "metadata.price", "name": "lowest_price"},
{"type": "max", "field": "metadata.price", "name": "highest_price"},
{"type": "avg", "field": "metadata.rating", "name": "avg_rating"},
{"type": "cardinality", "field": "metadata.brand", "name": "brand_count"}
],
"group_by": "metadata.category",
"include_documents": true
}
}
}
]
```
## Aggregation Details
### Count
```json theme={null}
{"type": "count", "name": "total"}
```
Counts documents. No field required.
### Sum / Avg / Min / Max
```json theme={null}
{"type": "avg", "field": "metadata.price", "name": "average_price"}
```
Requires numeric field.
### Cardinality
```json theme={null}
{"type": "cardinality", "field": "metadata.author", "name": "unique_authors"}
```
Counts unique values (approximate for large sets).
### Percentile
```json theme={null}
{"type": "percentile", "field": "score", "percentiles": [25, 50, 75, 95], "name": "score_percentiles"}
```
Returns specified percentile values.
### Histogram
```json theme={null}
{"type": "histogram", "field": "metadata.price", "interval": 50, "name": "price_buckets"}
```
Groups values into buckets.
## Statistical Aggregation Examples
<CodeGroup>
```json Percentile Analysis theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"function": "percentile", "field": "score", "alias": "median_score", "percentile_value": 50},
{"function": "percentile", "field": "score", "alias": "p90_score", "percentile_value": 90},
{"function": "stddev", "field": "score", "alias": "score_spread"},
{"function": "variance", "field": "score", "alias": "score_variance"}
]
}
}
}
```
```json Frequency Distribution theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"function": "frequency", "field": "category", "alias": "top_categories", "top_k": 10},
{"function": "frequency", "field": "brand", "alias": "top_brands", "top_k": 5}
]
}
}
}
```
```json Co-occurrence Analysis theme={null}
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"function": "co_occurrence", "field": "brand", "field_b": "category", "alias": "brand_category_pairs"},
{"function": "correlation", "field": "price", "field_b": "rating", "alias": "price_rating_corr"}
]
}
}
}
```
</CodeGroup>
### Statistical Output Examples
**Frequency** returns value counts with percentages:
```json theme={null}
{
"top_categories": [
{"value": "electronics", "count": 45, "percent": 30.0},
{"value": "clothing", "count": 38, "percent": 25.3},
{"value": "books", "count": 22, "percent": 14.7}
]
}
```
**Co-occurrence** returns field pair counts:
```json theme={null}
{
"brand_category_pairs": [
{"field_a": "Nike", "field_b": "footwear", "count": 12, "percent": 8.0},
{"field_a": "Nike", "field_b": "apparel", "count": 8, "percent": 5.3}
]
}
```
**Correlation** returns the Pearson coefficient (-1 to 1):
```json theme={null}
{
"price_rating_corr": 0.342156
}
```
## Error Handling
| Error | Behavior |
| ---------------------------------------- | ---------------------------------- |
| Missing field | Skip document for that aggregation |
| Non-numeric field | Error for numeric aggregations |
| Empty results | Return zero/empty aggregations |
| Invalid type | Stage fails |
| Insufficient data for stddev/correlation | Returns null (need 2+ values) |
## Related
* [Group By](/docs/retrieval/stages/group-by) - Group with full documents
* [Sample](/docs/retrieval/stages/sample) - Statistical sampling
* [Summarize](/docs/retrieval/stages/summarize) - LLM-powered analysis
# API Call
Source: https://docs.mixpeek.com/docs/retrieval/stages/api-call
Enrich documents with external API calls (Stripe, GitHub, weather APIs, etc.)
<Frame>
<img alt="API Call stage showing documents enriched with external HTTP API data" />
</Frame>
The API Call stage enriches documents by calling external HTTP APIs. This enables integration with third-party services like Stripe, GitHub, weather APIs, and more to augment documents with real-time data.
<Note>
**Stage Category**: APPLY (1-1 Enrichment)
**Transformation**: N documents → N documents (same count, expanded schema)
</Note>
## When to Use
| Use Case | Description |
| ------------------------ | ----------------------------------------- |
| **Customer data lookup** | Enrich with Stripe billing data, CRM info |
| **Repository info** | Fetch GitHub commit stats, stars |
| **Real-time data** | Add weather, stock prices, currency rates |
| **Data validation** | Verify addresses, phone numbers |
| **External context** | Lookup additional context from any API |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------------------- | ------------------------------ |
| Untrusted/user-provided URLs | Major security risk (SSRF) |
| API credentials can't be secured | Use organization secrets vault |
| High-volume enrichment | Rate limits apply |
| Time-critical responses | Network latency adds 100-500ms |
| Internal-only APIs behind firewalls | Use `sql_lookup` for databases |
## Parameters
### Required Parameters
| Parameter | Type | Description |
| ----------------- | --------- | ---------------------------------------------------------------------------- |
| `url` | string | API endpoint URL. Supports `{DOC.field}` and `{INPUT.field}` templates. |
| `allowed_domains` | string\[] | Domain allowlist for SSRF protection. **Never use `*`**. |
| `output_field` | string | Dot-path where API response should be stored (e.g., `metadata.stripe_data`). |
### Optional Parameters
| Parameter | Type | Default | Description |
| ------------------- | ------- | ---------- | ---------------------------------------------------- |
| `method` | string | `GET` | HTTP method: `GET`, `POST`, `PUT`, `PATCH`, `DELETE` |
| `auth` | object | `null` | Authentication configuration (see below) |
| `headers` | object | `{}` | Additional HTTP headers (supports templates) |
| `body` | object | `null` | Request body for POST/PUT/PATCH (JSON) |
| `timeout` | integer | `10` | Request timeout in seconds (1-60) |
| `max_response_size` | integer | `10485760` | Maximum response size in bytes (default: 10MB) |
| `response_path` | string | `null` | JSONPath to extract specific field from response |
| `rate_limit` | object | `null` | Rate limiting per domain |
| `when` | object | `null` | Conditional filter for selective enrichment |
| `on_error` | string | `skip` | Error handling: `skip`, `remove`, or `raise` |
## Authentication Types
<Tabs>
<Tab title="Bearer Token">
**OAuth 2.0, JWT tokens** - Most modern APIs (GitHub, OpenAI, Stripe)
```json theme={null}
{
"auth": {
"type": "bearer",
"secret_ref": "stripe_api_key"
}
}
```
Adds header: `Authorization: Bearer {secret_value}`
</Tab>
<Tab title="API Key">
**Header or query parameter** - Weather APIs, Maps, etc.
```json theme={null}
{
"auth": {
"type": "api_key",
"key": "X-API-Key",
"location": "header",
"secret_ref": "weather_api_key"
}
}
```
| Field | Description |
| ------------ | --------------------------------- |
| `key` | Header name or query param name |
| `location` | `header` (recommended) or `query` |
| `secret_ref` | Secret name in vault |
</Tab>
<Tab title="Basic Auth">
**HTTP Basic Authentication** - Legacy systems
```json theme={null}
{
"auth": {
"type": "basic",
"secret_ref": "basic_credentials"
}
}
```
Secret format: `username:password`
Adds header: `Authorization: Basic {base64(username:password)}`
</Tab>
<Tab title="Custom Header">
**Non-standard auth headers**
```json theme={null}
{
"auth": {
"type": "custom_header",
"key": "X-Custom-Auth",
"secret_ref": "custom_token"
}
}
```
Adds header: `{key}: {secret_value}`
</Tab>
</Tabs>
## Error Handling
| Strategy | Behavior | Best For |
| -------- | ---------------------------- | ---------------------------- |
| `skip` | Keep document unchanged | Optional enrichment |
| `remove` | Remove document from results | Mandatory enrichment |
| `raise` | Fail entire pipeline | Debugging, critical failures |
## Configuration Examples
<CodeGroup>
```json Stripe Customer Lookup theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.stripe.com/v1/customers/{DOC.metadata.stripe_id}",
"method": "GET",
"allowed_domains": ["api.stripe.com"],
"auth": {
"type": "bearer",
"secret_ref": "stripe_api_key"
},
"output_field": "metadata.stripe_data",
"timeout": 10,
"on_error": "skip"
}
}
}
```
```json GitHub Repository Stats theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.github.com/repos/{INPUT.owner}/{INPUT.repo}",
"method": "GET",
"allowed_domains": ["api.github.com"],
"output_field": "metadata.github_info",
"response_path": "$.stargazers_count"
}
}
}
```
```json POST with JSON Body theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.example.com/v1/analyze",
"method": "POST",
"allowed_domains": ["api.example.com"],
"auth": {
"type": "api_key",
"key": "X-API-Key",
"location": "header",
"secret_ref": "example_api_key"
},
"headers": {
"Content-Type": "application/json"
},
"body": {
"text": "{DOC.content}",
"language": "en"
},
"output_field": "metadata.analysis"
}
}
}
```
```json Conditional Enrichment theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.stripe.com/v1/customers/{DOC.metadata.customer_id}",
"method": "GET",
"allowed_domains": ["api.stripe.com"],
"auth": {
"type": "bearer",
"secret_ref": "stripe_api_key"
},
"output_field": "metadata.billing",
"when": {
"field": "metadata.is_premium",
"operator": "eq",
"value": true
},
"on_error": "skip"
}
}
}
```
```json Rate Limited with JSONPath theme={null}
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://api.weatherapi.com/v1/current.json?q={DOC.metadata.location}",
"method": "GET",
"allowed_domains": ["api.weatherapi.com"],
"auth": {
"type": "api_key",
"key": "key",
"location": "query",
"secret_ref": "weather_api_key"
},
"output_field": "metadata.weather",
"response_path": "$.current",
"rate_limit": {
"requests_per_minute": 60
},
"timeout": 5
}
}
}
```
</CodeGroup>
## Security
<Warning>
**SSRF Protection Required**
This stage makes external HTTP requests which can be exploited for Server-Side Request Forgery attacks. **Always use `allowed_domains`** to whitelist permitted domains.
</Warning>
### Security Best Practices
1. **Never use `*` in allowed\_domains** - Explicitly list each domain
2. **Never store credentials in configuration** - Always use `auth.secret_ref` to reference vault secrets
3. **Set rate limits** - Prevent abuse and excessive costs
4. **Use HTTPS** - HTTP URLs are automatically upgraded
5. **Audit configurations** - Review before deployment to prevent data exfiltration
### Storing Secrets
```bash theme={null}
# Create a secret in the organization vault
curl -X POST "$MP_API_URL/v1/organizations/secrets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "stripe_api_key",
"secret_value": "mxp_sk_..."
}'
```
Then reference in configuration:
```json theme={null}
{
"auth": {
"type": "bearer",
"secret_ref": "stripe_api_key"
}
}
```
## Performance
| Metric | Value |
| ----------------------- | -------------------------------- |
| **Latency per request** | 100-500ms (network dependent) |
| **Timeout range** | 1-60 seconds |
| **Max response size** | 10MB (configurable) |
| **Parallelization** | Documents processed concurrently |
## Template Variables
URLs, headers, and body values support template variables:
| Namespace | Description | Example |
| --------- | ----------------------- | ---------------------------- |
| `DOC` | Current document fields | `{DOC.metadata.customer_id}` |
| `INPUT` | Query inputs | `{INPUT.api_version}` |
## Related
* [SQL Lookup](/docs/retrieval/stages/sql-lookup) - For database enrichment
* [Document Enrich](/docs/retrieval/stages/document-enrich) - For collection joins
* [Organization Secrets](/docs/api-reference/organization-secrets/create-secret) - Managing API credentials
# Attribute Filter
Source: https://docs.mixpeek.com/docs/retrieval/stages/attribute-filter
Filter documents by metadata field conditions with boolean logic support
<Frame>
<img alt="Attribute Filter stage showing metadata-based document filtering" />
</Frame>
The Attribute Filter stage filters documents based on metadata field conditions. It supports simple single-field filtering and complex boolean logic (AND/OR/NOT). When used as a first stage, it retrieves documents directly from the database; when used after other stages, it filters in-memory results.
<Note>
**Stage Category**: FILTER (Reduces document set)
**Transformation**: N documents → M documents (where M ≤ N, based on conditions)
</Note>
## When to Use
| Use Case | Description |
| -------------------------- | --------------------------------------------- |
| **Metadata filtering** | Filter by status, category, date, etc. |
| **Post-search refinement** | Narrow semantic search results |
| **Access control** | Filter by user permissions |
| **Business logic** | Active items, published content |
| **Initial retrieval** | Fetch documents by attributes (no embeddings) |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ----------------------------------- |
| Semantic similarity | `feature_search` |
| Content-based filtering | `llm_filter` |
| Complex text matching | `feature_search` with text features |
| Scoring/ranking | Use sort stages after filtering |
## Parameters
### Simple Mode
Use for single-condition filtering:
| Parameter | Type | Default | Description |
| ------------------ | ------- | ---------- | -------------------------------- |
| `field` | string | *Required* | Field path to filter on |
| `operator` | string | `eq` | Comparison operator |
| `value` | any | *Required* | Value to compare against |
| `case_insensitive` | boolean | `false` | Case-insensitive string matching |
### Boolean Mode
Use for complex multi-condition filtering:
| Parameter | Type | Default | Description |
| ------------ | ------- | ---------- | -------------------------------------- |
| `conditions` | object | *Required* | Boolean condition object |
| `batch_size` | integer | `100` | Documents per batch (first-stage only) |
### Natural-Language Mode
Supply a plain-English filter and let an LLM compile it to a structured `conditions` tree at runtime. Useful when the filter shape is derived from user input or when field names are unknown in advance.
| Parameter | Type | Default | Description |
| ------------------ | ------------- | ---------- | --------------------------------------------------------------------------------------- |
| `natural_language` | string | *Required* | Plain-English filter description (e.g. `"active premium customers with priority >= 5"`) |
| `field_hints` | list\[string] | `null` | Optional list of allowed field paths to constrain the LLM's output |
<Info>
NL mode rides the wave — better LLMs produce better filter trees. Takes precedence over simple-mode fields; ignored if `conditions` is also set. Adds one LLM call per execution (\~200-500ms).
</Info>
## Supported Operators
| Operator | Description | Example Value |
| ------------------ | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `eq` | Equals | `"active"`, `42`, `true` |
| `ne` | Not equals | `"deleted"` |
| `gt` | Greater than | `100` |
| `gte` | Greater than or equal | `4.5` |
| `lt` | Less than | `50` |
| `lte` | Less than or equal | `10` |
| `in` | In array | `["tech", "science"]` |
| `nin` | Not in array | `["spam", "deleted"]` |
| `contains` | Contains substring | `"guide"` |
| `starts_with` | Starts with | `"intro"` |
| `ends_with` | Ends with | `".pdf"` |
| `regex` | Regular expression | `"^[A-Z].*"` |
| `exists` | Field exists | `true` or `false` |
| `is_null` | Field is null | `true` or `false` |
| `text` | Full-text search (token-based; word order not preserved) | `"machine learning"` |
| `phrase` | Exact phrase — matches the words in order, word-boundary safe | `"machine learning model"` |
| `geo_radius` | Within N meters of a center point | `{"center": {"lat": 40.758, "lon": -73.9855}, "radius": 15000}` |
| `geo_bounding_box` | Inside a lat/lon box (antimeridian-safe) | `{"top_left": {"lat": 42.0, "lon": -74.5}, "bottom_right": {"lat": 40.5, "lon": -72.5}}` |
| `geo_polygon` | Inside an arbitrary polygon (≥3 points) | `{"exterior": {"points": [{"lat": 41.2, "lon": -74.5}, ...]}}` |
<Info>
The three `geo_*` operators take an **object** as their `value`, not a scalar. See [Geospatial filtering](#geospatial-filtering) for the field format and exact value shapes.
</Info>
## Configuration Examples
<CodeGroup>
```json Simple Equality theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.status",
"operator": "eq",
"value": "published"
}
}
}
```
```json Numeric Comparison theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.price",
"operator": "lte",
"value": 100
}
}
}
```
```json In Array theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.category",
"operator": "in",
"value": ["electronics", "computers", "phones"]
}
}
}
```
```json Contains Substring theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.title",
"operator": "contains",
"value": "guide",
"case_insensitive": true
}
}
}
```
```json Field Exists theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.premium_features",
"operator": "exists",
"value": true
}
}
}
```
```json Dynamic Value theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.category",
"operator": "eq",
"value": "{{INPUT.category}}"
}
}
}
```
```json Natural Language theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"natural_language": "published blog posts from 2024 that are not archived",
"field_hints": ["metadata.status", "metadata.published_at", "metadata.archived"]
}
}
}
```
</CodeGroup>
## Boolean Conditions
For complex filtering, use the `conditions` parameter with AND/OR/NOT logic:
<CodeGroup>
```json AND Conditions theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "metadata.status", "operator": "eq", "value": "active"},
{"field": "metadata.in_stock", "operator": "eq", "value": true},
{"field": "metadata.price", "operator": "lte", "value": 500}
]
}
}
}
}
```
```json OR Conditions theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"OR": [
{"field": "metadata.category", "operator": "eq", "value": "featured"},
{"field": "metadata.rating", "operator": "gte", "value": 4.5}
]
}
}
}
}
```
```json NOT Conditions theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"NOT": [
{"field": "metadata.status", "operator": "eq", "value": "deleted"},
{"field": "metadata.spam", "operator": "eq", "value": true}
]
}
}
}
}
```
```json Nested Boolean Logic theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "metadata.status", "operator": "eq", "value": "active"},
{
"OR": [
{"field": "metadata.category", "operator": "eq", "value": "premium"},
{"field": "metadata.featured", "operator": "eq", "value": true}
]
},
{
"NOT": [
{"field": "metadata.region", "operator": "eq", "value": "restricted"}
]
}
]
}
}
}
}
```
</CodeGroup>
## Geospatial filtering
Filter documents by geographic location. Three operators — `geo_radius`,
`geo_bounding_box`, and `geo_polygon` — restrict results to documents whose
location falls within a circle, a box, or an arbitrary polygon. They work in
both simple mode and inside boolean `conditions`, exactly like any other
operator.
### The location field
A document is geo-filterable when it carries a location field. By convention the
field is named `location`, and Mixpeek resolves it wherever it lives — it tries
top-level `location`, then `metadata.location`, then `_internal.metadata.location`
— so you write `"field": "location"` regardless of how the document was ingested.
A location value can be either of two forms:
| Form | Example | Notes |
| ------------- | ---------------------------------- | ---------------------------------------------------------- |
| Object | `{"lat": 40.758, "lon": -73.9855}` | Recommended. `lat` in `[-90, 90]`, `lon` in `[-180, 180]`. |
| GeoJSON array | `[-73.9855, 40.758]` | **Longitude first**, then latitude — the GeoJSON order. |
<Warning>
**GeoJSON arrays are `[lon, lat]`, not `[lat, lon]`.** This is the single most
common geo footgun. `[-73.9855, 40.758]` is Manhattan; `[40.758, -73.9855]` is
an invalid point (latitude 40.758 is fine, but longitude 40.758 places it in
the Indian Ocean, and if either value exceeds its range the point is dropped).
Prefer the `{"lat": ..., "lon": ...}` object form, which is unambiguous.
</Warning>
A document with **no** location field never matches a geo filter (it is treated
as a non-match, not an error). If a document stores a **list** of points, it
matches when **any** point satisfies the filter.
### The three operators
<CodeGroup>
```json geo_radius theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "location",
"operator": "geo_radius",
"value": {
"center": {"lat": 40.758, "lon": -73.9855},
"radius": 15000
}
}
}
}
```
```json geo_bounding_box theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "location",
"operator": "geo_bounding_box",
"value": {
"top_left": {"lat": 42.0, "lon": -74.5},
"bottom_right": {"lat": 40.5, "lon": -72.5}
}
}
}
}
```
```json geo_polygon theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "location",
"operator": "geo_polygon",
"value": {
"exterior": {
"points": [
{"lat": 41.2, "lon": -74.5},
{"lat": 41.2, "lon": -73.5},
{"lat": 40.4, "lon": -73.5},
{"lat": 40.4, "lon": -74.5}
]
}
}
}
}
}
```
</CodeGroup>
| Operator | `value` shape | Semantics |
| ------------------ | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `geo_radius` | `{"center": {"lat", "lon"}, "radius": <meters>}` | Within `radius` **meters** of `center`. Distance is haversine great-circle (mean Earth radius 6,371,000 m). |
| `geo_bounding_box` | `{"top_left": {"lat", "lon"}, "bottom_right": {"lat", "lon"}}` | Inside the box. Antimeridian-safe: if `top_left.lon > bottom_right.lon` the box wraps 180° (e.g. Fiji). |
| `geo_polygon` | `{"exterior": {"points": [{"lat", "lon"}, ...]}}` | Inside the exterior ring by ray-casting. Needs **≥ 3** points. |
<Note>
`radius` is always in **meters** — `15000` is 15 km, not 15,000 km.
</Note>
### Combine with a selective filter (recommended)
Geo operators are evaluated by a payload scan — they are correct but **not
index-accelerated** (there is no geo postings index yet). To keep the candidate
set small and latency low, pair a geo condition with a selective non-geo
condition (a brand, category, or status equality) inside an `AND`:
```json theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "metadata.category", "operator": "eq", "value": "store"},
{
"field": "location",
"operator": "geo_radius",
"value": {"center": {"lat": 40.758, "lon": -73.9855}, "radius": 5000}
}
]
}
}
}
}
```
The selective `eq` narrows the set first; the geo condition is then applied to a
small candidate list. Don't rely on a geo-only filter over a large collection
for sub-millisecond latency.
### Validation
Malformed geo specs fail **loudly at request time** with a message that names
the expected shape — you get a clear `400`, not a silent empty result set.
Common mistakes that are rejected:
* `value` is not an object (e.g. a bare number or string)
* `center` / `top_left` / `bottom_right` missing `lat` or `lon`, or out of range
* `radius` missing or negative
* `geo_polygon` with fewer than 3 points, or a point missing `lat`/`lon`
### Worked example
Six documents, each an office with a `location` (one has none):
| Document | Location | lat, lon |
| ------------ | --------------- | ----------------- |
| Manhattan | NYC | 40.7580, -73.9855 |
| Brooklyn | NYC | 40.6782, -73.9442 |
| Hartford | CT | 41.7658, -72.6734 |
| Boston | MA | 42.3601, -71.0589 |
| Philadelphia | PA | 39.9526, -75.1652 |
| Virtual | *(no location)* | — |
Applying each operator from the examples above:
| Filter | Matches | Why |
| -------------------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `geo_radius` center=Manhattan, 15 km | **Manhattan, Brooklyn** | Brooklyn is \~8 km away; Hartford (\~160 km), Boston (\~306 km), Philadelphia (\~130 km) are outside. |
| `geo_bounding_box` (42.0,-74.5)→(40.5,-72.5) | **Manhattan, Brooklyn, Hartford** | Boston's lon -71.06 is east of -72.5; Philadelphia's lat 39.95 is south of 40.5. |
| `geo_polygon` NYC-metro rectangle | **Manhattan, Brooklyn** | Hartford's lat 41.77 and Philadelphia's lat 39.95 fall outside the ring. |
In every case the Virtual office (no location) is excluded.
## First-Stage vs Later-Stage Behavior
| Position | Behavior |
| --------------- | --------------------------------------------------------------------- |
| **First stage** | Fetches documents directly from database (up to 1,000 per collection) |
| **Later stage** | Filters in-memory results from previous stages |
### First-Stage Example
When no documents exist in the pipeline yet:
```json theme={null}
[
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.status",
"operator": "eq",
"value": "published",
"batch_size": 100
}
}
}
]
```
### Later-Stage Example
After semantic search:
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.query}}"}],
"final_top_k": 100
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.in_stock",
"operator": "eq",
"value": true
}
}
}
]
```
## Performance
| Metric | Value |
| ----------------------- | ------------------------------ |
| **Latency** | 5-20ms |
| **First-stage limit** | 1,000 documents per collection |
| **In-memory filtering** | \< 5ms for 1,000 docs |
| **Index utilization** | Uses indexes when available |
<Tip>
For best performance with `feature_search`, use pre-filters in the search stage instead of a separate `attribute_filter` stage. Pre-filters are applied at the vector index level.
</Tip>
## Common Pipeline Patterns
### Search + Filter + Sort
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"final_top_k": 50
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "metadata.status", "operator": "eq", "value": "active"},
{"field": "metadata.category", "operator": "in", "value": "{{INPUT.categories}}"}
]
}
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.created_at",
"direction": "desc"
}
}
}
]
```
### Attribute-Only Retrieval (No Embeddings)
```json theme={null}
[
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "metadata.type", "operator": "eq", "value": "product"},
{"field": "metadata.featured", "operator": "eq", "value": true}
]
},
"batch_size": 50
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.priority",
"direction": "desc"
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"count": 10
}
}
}
]
```
### Multi-Stage Filtering
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.query}}"}],
"final_top_k": 200
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.category",
"operator": "eq",
"value": "{{INPUT.category}}"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 20
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.in_stock",
"operator": "eq",
"value": true
}
}
}
]
```
## Comparison: attribute\_filter vs feature\_search Pre-Filters
| Aspect | attribute\_filter (stage) | feature\_search pre-filters |
| ---------------- | ------------------------------------- | --------------------------- |
| **When applied** | After search results | During vector search |
| **Performance** | Good | Best (index-level) |
| **Use case** | Post-filtering, first-stage retrieval | Always when possible |
| **Flexibility** | Can be placed anywhere | Only with feature\_search |
**Recommendation:** When filtering during semantic search, prefer pre-filters in `feature_search`. Use `attribute_filter` for:
* Post-search refinement based on previous stage outputs
* First-stage attribute-only retrieval
* Dynamic filters that depend on earlier stage results
## Error Handling
| Error | Behavior |
| ------------------------------ | ----------------------------------------------------------------------------------------------------- |
| Field not found | Document excluded (treated as no match) |
| Invalid operator | Stage fails with error |
| Type mismatch | Attempts type coercion, then excludes |
| Invalid regex | Stage fails with error |
| Malformed geo spec | Request rejected with a clear error naming the expected shape (fails loud, not a silent empty result) |
| Document has no location field | Document excluded from geo filters (treated as no match) |
## Related Stages
* [Feature Search](/docs/retrieval/stages/feature-search) - Semantic vector search
* [LLM Filter](/docs/retrieval/stages/llm-filter) - Content-based LLM filtering
* [Sort Attribute](/docs/retrieval/stages/sort-attribute) - Sort by field values
# Classify
Source: https://docs.mixpeek.com/docs/retrieval/stages/classify
Classify documents at query time using built-in tasks (like NSFW content safety) or a custom model deployed as a plugin
The Classify stage labels documents at query time. It supports two modes:
* **Built-in tasks** (via `task`) — managed classifiers Mixpeek hosts for you. The first available task is `nsfw`, a content-safety classifier for text, image, and video. No plugin to build or deploy.
* **Custom classifiers** (via `feature_uri`) — your own classifier model deployed as a [custom extractor](/docs/processing/custom-extractors). The stage sends document text to your extractor's inference endpoint and attaches predicted labels with confidence scores to each document.
Set exactly one of `task` or `feature_uri`.
<Note>
**Stage Category**: APPLY (Enriches documents)
**Transformation**: N documents → N documents (with classification results added). A built-in task can optionally drop documents (e.g. `drop_if_unsafe`), making it N → ≤N.
</Note>
## When to Use
| Use Case | Description |
| ------------------------- | --------------------------------------------------- |
| **Custom classification** | Apply your own trained classifier to search results |
| **Content labeling** | Tag documents with domain-specific categories |
| **Compliance scoring** | Score documents against compliance criteria |
| **Intent detection** | Classify user queries or document intent |
## When NOT to Use
| Scenario | Recommended Alternative |
| ---------------------------------- | ------------------------------------------------------------------------------- |
| Predefined taxonomy classification | [`taxonomy_enrich`](/docs/retrieval/stages/taxonomy-enrich) (no custom model needed) |
| LLM-based classification | [`llm_enrich`](/docs/retrieval/stages/llm-enrich) with `output_schema` |
| Simple keyword matching | [`attribute_filter`](/docs/retrieval/stages/attribute-filter) |
## Available Classification Tasks
Built-in tasks are managed classifiers Mixpeek hosts — set `task` and skip `feature_uri` entirely. This list grows over time as more built-in tasks ship.
| Task | What it does | Model | Modalities | Outputs |
| ------ | ------------------------------------------------------------------------------------------ | ------------------------------- | ------------------ | -------------------------------------- |
| `nsfw` | Content-safety classification — scores how likely a document is unsafe (not safe for work) | `mixpeek/content-classifier-v1` | Text, image, video | `nsfw_score` (0–1), `label`, `is_nsfw` |
The `mixpeek/content-classifier-v1` model is a CPU-only multimodal classifier: DistilBERT for text, ViT for images, and frame sampling for video — so the `nsfw` task runs without a GPU. By default it annotates each document with the outputs above; set `drop_if_unsafe: true` to filter unsafe documents out of the result set instead.
<Note>
The same `mixpeek/content-classifier-v1` model also powers a **tenant-level upload gate**: when `nsfw_check_enabled` is on for a shared-plane org, NSFW image, video, and text uploads are rejected at upload time. The Classify stage applies the same model at query time over search results. See [Uploads](/docs/ingestion/uploads).
</Note>
## Parameters
| Parameter | Type | Default | Description |
| -------------------- | ------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `task` | string (enum) | `null` | Built-in classification task to run (e.g. `"nsfw"`). Mutually exclusive with `feature_uri`. See [Available Classification Tasks](#available-classification-tasks). |
| `feature_uri` | string | *Required unless `task` is set* | Feature URI of your custom classifier plugin. Mutually exclusive with `task`. |
| `document_field` | string | `"content"` | Document field path containing text to classify |
| `image_field` | string | `null` | Document field path containing an image URL to classify instead of text (built-in tasks only) |
| `video_field` | string | `null` | Document field path containing a video URL to classify instead of text (built-in tasks only) |
| `output_field` | string | `"classification"` | Field path to store classification results |
| `nsfw_threshold` | float | `0.7` | Score at or above which a document is flagged `is_nsfw` (`nsfw` task only) |
| `drop_if_unsafe` | boolean | `false` | Drop documents flagged unsafe instead of annotating them (`nsfw` task only) |
| `max_document_chars` | integer | `5000` | Maximum characters sent for classification (100–50000) |
| `top_k_labels` | integer | `null` | Keep only the top-k labels by confidence (custom classifiers) |
| `min_confidence` | float | `null` | Minimum confidence threshold (0.0–1.0, custom classifiers) |
| `batch_size` | integer | `10` | Documents per inference call (1–100) |
| `max_concurrency` | integer | `5` | Maximum concurrent inference requests (1–20) |
## Plugin Contract
Your classifier plugin must accept `{text: str}` and return `{labels: [{label: str, confidence: float}]}`.
```python theme={null}
# In your plugin's realtime.py
class ClassifierService(BaseInferenceService):
def _process_single(self, inputs: dict, parameters: dict) -> dict:
text = inputs["text"]
# Your classification logic here
return {
"labels": [
{"label": "technology", "confidence": 0.92},
{"label": "business", "confidence": 0.78},
{"label": "science", "confidence": 0.45},
]
}
```
<Tip>
Set `inference_type: "classify"` in your plugin's manifest to declare compatibility with the classify stage.
</Tip>
## Configuration Examples
### Built-in NSFW Task
<CodeGroup>
```json Annotate (flag NSFW) theme={null}
{
"stage_name": "nsfw_check",
"config": {
"stage_id": "classify",
"parameters": {
"task": "nsfw",
"document_field": "content",
"output_field": "safety",
"nsfw_threshold": 0.7
}
}
}
```
```json Filter (drop unsafe docs) theme={null}
{
"stage_name": "nsfw_filter",
"config": {
"stage_id": "classify",
"parameters": {
"task": "nsfw",
"document_field": "content",
"drop_if_unsafe": true,
"nsfw_threshold": 0.7
}
}
}
```
```json Classify a Media URL theme={null}
{
"stage_name": "nsfw_image_check",
"config": {
"stage_id": "classify",
"parameters": {
"task": "nsfw",
"image_field": "metadata.thumbnail_url",
"output_field": "safety"
}
}
}
```
</CodeGroup>
The annotate example writes a result like `{"nsfw_score": 0.03, "label": "safe", "is_nsfw": false}` to `output_field` on every document. The filter example drops any document whose `nsfw_score` meets `nsfw_threshold` instead of annotating it.
### Custom Classifier
<CodeGroup>
```json Basic Classification theme={null}
{
"stage_name": "my_classifier",
"config": {
"stage_id": "classify",
"parameters": {
"feature_uri": "mixpeek://my_classifier@1.0.0/classify",
"document_field": "content",
"output_field": "classification"
}
}
}
```
```json With Confidence Filtering theme={null}
{
"stage_name": "my_classifier",
"config": {
"stage_id": "classify",
"parameters": {
"feature_uri": "mixpeek://my_classifier@1.0.0/classify",
"document_field": "metadata.description",
"output_field": "labels",
"min_confidence": 0.5,
"top_k_labels": 3
}
}
}
```
```json High-Throughput theme={null}
{
"stage_name": "my_classifier",
"config": {
"stage_id": "classify",
"parameters": {
"feature_uri": "mixpeek://my_classifier@1.0.0/classify",
"batch_size": 20,
"max_concurrency": 10,
"max_document_chars": 2000
}
}
}
```
</CodeGroup>
## Output
For a **custom classifier**, each document gets the predicted labels added at `output_field`:
```json theme={null}
{
"document_id": "doc_123",
"content": "Apple Inc. announced new AI features...",
"classification": [
{"label": "technology", "confidence": 0.95},
{"label": "business", "confidence": 0.82}
]
}
```
For the built-in **`nsfw` task**, each document gets a safety result at `output_field` (unless `drop_if_unsafe` removed it):
```json theme={null}
{
"document_id": "doc_123",
"content": "Apple Inc. announced new AI features...",
"safety": {
"nsfw_score": 0.03,
"label": "safe",
"is_nsfw": false
}
}
```
## Performance
| Metric | Value |
| ---------------------- | ---------------------------- |
| **Latency** | Depends on your plugin model |
| **Batch size** | 10 documents default |
| **Concurrency** | 5 parallel requests default |
| **Max document chars** | 5000 default |
## Common Pipeline Patterns
### Search + Classify + Filter by Label
```json theme={null}
[
{
"stage_name": "search",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": {"input_mode": "text", "value": "{{INPUT.query}}"},
"top_k": 100
}],
"final_top_k": 25
}
}
},
{
"stage_name": "classify",
"config": {
"stage_id": "classify",
"parameters": {
"feature_uri": "mixpeek://my_classifier@1.0.0/classify",
"min_confidence": 0.7,
"top_k_labels": 1
}
}
},
{
"stage_name": "filter_by_label",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "classification.0.label",
"operator": "eq",
"value": "{{INPUT.target_category}}"
}
}
}
]
```
## Related
* [Taxonomy Enrich](/docs/retrieval/stages/taxonomy-enrich) - Predefined taxonomy classification (no custom model)
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - LLM-based enrichment and classification
* [Custom Extractors](/docs/processing/custom-extractors) - Build and deploy custom inference models
* [Uploads](/docs/ingestion/uploads) - Tenant-level NSFW upload gate using the same `mixpeek/content-classifier-v1` model
# Cluster
Source: https://docs.mixpeek.com/docs/retrieval/stages/cluster
Group documents by embedding similarity into semantic clusters
<Frame>
<img alt="Cluster stage showing semantic grouping of search results" />
</Frame>
The Cluster stage groups documents based on embedding similarity, creating semantic clusters of related content. This helps organize search results into meaningful groups and discover themes within your results.
<Note>
**Stage Category**: GROUP (Groups documents)
**Transformation**: N documents → K clusters with documents
</Note>
## When to Use
| Use Case | Description |
| ----------------------- | --------------------------------- |
| **Theme discovery** | Find topics within search results |
| **Result organization** | Group similar items together |
| **Deduplication** | Find near-duplicate content |
| **Exploration** | Understand result diversity |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ----------------------- |
| Grouping by field value | `group_by` |
| Removing duplicates | `deduplicate` |
| Pre-defined categories | `taxonomy_enrich` |
| Single representative | `sample` per group |
## Parameters
| Parameter | Type | Default | Description |
| ------------------ | ------- | ---------- | ------------------------------------------------------------------ |
| `n_clusters` | integer | `5` | Number of clusters (kmeans/agglomerative) |
| `feature_uri` | string | *auto* | Embedding to cluster; auto-inherited from a prior `feature_search` |
| `output_mode` | string | `clusters` | `clusters`, `labeled`, or `representatives` |
| `algorithm` | string | `kmeans` | Clustering algorithm |
| `min_cluster_size` | integer | `2` | Minimum documents per cluster |
| `include_outliers` | boolean | `true` | Include documents that don't fit clusters |
| `label_clusters` | boolean | `false` | Generate cluster labels with LLM |
## Clustering Algorithms
| Algorithm | Description | Best For |
| --------------- | ------------------------------------------ | ------------------------------------------ |
| `auto` | LLM picks the algorithm from dataset shape | You don't know the right algorithm upfront |
| `kmeans` | K-means clustering | Fixed number of clusters |
| `hdbscan` | Density-based clustering | Unknown cluster count |
| `dbscan` | Density-based with fixed epsilon | Noisy data with clear density separation |
| `agglomerative` | Hierarchical clustering | Nested clusters |
| `spectral` | Graph-based clustering | Non-convex cluster shapes |
<Info>
`auto` runs one lightweight LLM call on the first batch — given sample size, dimensionality, variance, and intended cluster count — and locks in a concrete algorithm. Rides the wave: upgrading the LLM improves selection without code changes. Falls back to `kmeans` if the LLM call fails.
</Info>
## Configuration Examples
<CodeGroup>
```json Basic Clustering theme={null}
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"n_clusters": 5
}
}
}
```
```json KMeans With Known K theme={null}
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"algorithm": "kmeans",
"n_clusters": 8
}
}
}
```
```json Density-Based Clustering theme={null}
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"algorithm": "hdbscan",
"min_cluster_size": 3
}
}
}
```
```json Custom Embedding Field theme={null}
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"n_clusters": 6,
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
}
}
}
```
```json Fine-Grained Clustering theme={null}
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"n_clusters": 15,
"min_cluster_size": 2,
"algorithm": "kmeans"
}
}
}
```
```json Representative Documents theme={null}
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"algorithm": "kmeans",
"n_clusters": 8,
"output_mode": "representatives"
}
}
}
```
</CodeGroup>
## How Clustering Works
1. **Extract Embeddings**: Get embedding vectors from each document
2. **Apply Algorithm**: Run clustering algorithm (e.g., k-means)
3. **Assign Documents**: Each document assigned to nearest cluster
4. **Compute Centroids**: Calculate cluster centers
5. **Label (optional)**: Generate human-readable cluster names
## Output Schema
```json theme={null}
{
"clusters": [
{
"cluster_id": 0,
"label": "Machine Learning Tutorials",
"centroid": [0.12, -0.34, ...],
"size": 12,
"documents": [
{
"document_id": "doc_123",
"content": "Introduction to neural networks...",
"score": 0.95,
"cluster": {
"cluster_id": 0,
"distance_to_centroid": 0.15
}
}
]
},
{
"cluster_id": 1,
"label": "Data Engineering",
"centroid": [0.45, 0.23, ...],
"size": 8,
"documents": [...]
}
],
"outliers": [
{
"document_id": "doc_789",
"content": "Unrelated content...",
"outlier_reason": "distance_threshold_exceeded"
}
],
"metadata": {
"algorithm": "kmeans",
"n_clusters": 5,
"total_documents": 50,
"clustered_documents": 48,
"outlier_count": 2
}
}
```
## Performance
| Metric | Value |
| --------------- | ----------------------------- |
| **Latency** | 50-200ms |
| **Memory** | O(N × embedding\_dim) |
| **Cost** | Free (+ LLM cost if labeling) |
| **Scalability** | Up to \~10K documents |
<Warning>
Clustering large document sets (10K+) can be slow. Consider pre-filtering or sampling before clustering.
</Warning>
## Common Pipeline Patterns
### Search + Cluster
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"n_clusters": 5
}
}
}
]
```
### Cluster + Sample Representatives
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 }
],
"final_top_k": 200
}
}
},
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"n_clusters": 10
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "stratified",
"stratify_by": "cluster.cluster_id",
"min_per_stratum": 2
}
}
}
]
```
### Theme Discovery Pipeline
```json theme={null}
[
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.date",
"operator": "gte",
"value": "2024-01-01"
}
}
}
},
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"algorithm": "hdbscan",
"min_cluster_size": 5
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Summarize the main themes found in these clusters"
}
}
}
]
```
### Diverse Results Pipeline
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"n_clusters": 5
}
}
},
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n # Select top doc from each cluster\n clusters = doc.get('clusters', [])\n results = []\n for c in clusters:\n if c['documents']:\n results.append(c['documents'][0])\n doc['diverse_results'] = results\n return doc"
}
}
}
]
```
## Cluster Labeling
When `label_clusters: true`, an LLM generates descriptive labels:
| Cluster Documents | Generated Label |
| ---------------------- | -------------------------- |
| Docs about Python ML | "Python Machine Learning" |
| Docs about cloud infra | "Cloud Infrastructure" |
| Docs about API design | "REST API Design Patterns" |
## Choosing num\_clusters
| Result Size | Recommended Clusters |
| ------------ | -------------------- |
| \< 50 docs | 3-5 clusters |
| 50-200 docs | 5-10 clusters |
| 200-500 docs | 8-15 clusters |
| 500+ docs | 10-20 clusters |
<Tip>
Start with fewer clusters and increase if clusters are too broad. Use HDBSCAN if you don't know the optimal number.
</Tip>
## Error Handling
| Error | Behavior |
| ------------------ | ----------------------- |
| Missing embeddings | Skip document |
| Too few documents | Reduce num\_clusters |
| Clustering fails | Return unclustered docs |
| Labeling fails | Use numeric labels |
## Related
* [Group By](/docs/retrieval/stages/group-by) - Group by field values
* [Sample](/docs/retrieval/stages/sample) - Select representatives
* [MMR](/docs/retrieval/stages/mmr) - Diversity in ranking
* [Deduplicate](/docs/retrieval/stages/deduplicate) - Remove duplicates
# Code Execution
Source: https://docs.mixpeek.com/docs/retrieval/stages/code-execution
Execute custom code to transform, filter, or enrich documents
<Frame>
<img alt="Code Execution stage showing custom code transformations" />
</Frame>
The Code Execution stage allows you to run custom Python code to transform, filter, or enrich documents. This provides maximum flexibility for complex logic that can't be expressed with other stages.
<Note>
**Stage Category**: APPLY (Transforms documents)
**Transformation**: N documents → M documents (custom logic)
</Note>
## When to Use
| Use Case | Description |
| -------------------------- | ------------------------------- |
| **Custom transformations** | Complex field calculations |
| **Business logic** | Domain-specific rules |
| **Data normalization** | Custom parsing/formatting |
| **Advanced filtering** | Logic beyond structured\_filter |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ----------------------- |
| Simple field transforms | `json_transform` |
| LLM-based enrichment | `llm_enrich` |
| Standard filtering | `attribute_filter` |
| External API calls | `api_call` |
## Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------- | ---------- | ----------------------------------------------------------- |
| `code` | string | *Required* | Code to execute |
| `language` | string | `python` | Execution language (`python`, `typescript`, `javascript`) |
| `output_field` | string | `computed` | Document field path where results are merged |
| `result_variable` | string | `result` | Variable name containing the output list |
| `timeout_ms` | integer | `5000` | Execution timeout in milliseconds (100-30000) |
| `max_output_size` | integer | `100000` | Max output size in bytes (1024-1000000) |
| `env` | object | `{}` | Environment variables (supports `INPUT`/`SECRET` templates) |
| `on_error` | string | `skip` | Error handling strategy (`skip`, `raise`) |
## Configuration Examples
<CodeGroup>
```json Basic Transformation theme={null}
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n doc['word_count'] = len(doc.get('content', '').split())\n return doc"
}
}
}
```
```json Custom Scoring theme={null}
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n base_score = doc.get('score', 0)\n recency_boost = 1.0 if doc.get('metadata', {}).get('is_recent') else 0.8\n doc['adjusted_score'] = base_score * recency_boost\n return doc"
}
}
}
```
```json Filtering Logic theme={null}
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n content = doc.get('content', '')\n # Filter out short or low-quality content\n if len(content) < 100:\n return None\n if content.count('http') > 5:\n return None # Too many links\n return doc"
}
}
}
```
```json With External Packages theme={null}
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "import dateutil.parser\n\ndef transform(doc):\n date_str = doc.get('metadata', {}).get('date')\n if date_str:\n parsed = dateutil.parser.parse(date_str)\n doc['metadata']['year'] = parsed.year\n doc['metadata']['month'] = parsed.month\n return doc"
}
}
}
```
```json Text Processing theme={null}
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "import re\n\ndef transform(doc):\n content = doc.get('content', '')\n # Extract emails\n emails = re.findall(r'[\\w.-]+@[\\w.-]+', content)\n doc['extracted_emails'] = emails\n # Clean content\n doc['clean_content'] = re.sub(r'\\s+', ' ', content).strip()\n return doc"
}
}
}
```
</CodeGroup>
## Code Structure
Your code must define a `transform` function:
```python theme={null}
def transform(doc):
"""
Transform a single document.
Args:
doc: Dictionary containing document fields
Returns:
- Modified doc dict to keep document
- None to filter out document
"""
# Your logic here
return doc
```
### Available in Scope
| Variable | Type | Description |
| --------- | ---- | ------------------------- |
| `doc` | dict | Current document |
| `INPUT` | dict | Pipeline input parameters |
| `CONTEXT` | dict | Pipeline context |
## Input Document Structure
```python theme={null}
doc = {
"document_id": "doc_123",
"content": "Document text content...",
"score": 0.85,
"metadata": {
"title": "Document Title",
"author": "John Doe",
"date": "2024-01-15"
}
}
```
## Output Options
| Return Value | Effect |
| ------------------ | -------------------------- |
| `doc` (modified) | Keep document with changes |
| `doc` (unmodified) | Keep document as-is |
| `None` | Filter out document |
## Performance
| Metric | Value |
| --------------- | ---------------------------- |
| **Latency** | 5-50ms per document |
| **Timeout** | Configurable (default 5s) |
| **Memory** | Configurable (default 128MB) |
| **Concurrency** | Parallel execution |
<Warning>
Code execution adds latency. Keep transformations simple and avoid heavy computation. For complex processing, consider pre-computing during ingestion.
</Warning>
## Common Pipeline Patterns
### Custom Scoring Pipeline
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n score = doc.get('score', 0)\n # Boost verified sources\n if doc.get('metadata', {}).get('verified'):\n score *= 1.2\n # Penalize old content\n if doc.get('metadata', {}).get('year', 2024) < 2020:\n score *= 0.8\n doc['custom_score'] = score\n return doc"
}
}
},
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": {
"score_field": "custom_score"
}
}
}
]
```
### Data Normalization Pipeline
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 30 }
],
"final_top_k": 30
}
}
},
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n meta = doc.get('metadata', {})\n # Normalize price to USD\n price = meta.get('price', 0)\n currency = meta.get('currency', 'USD')\n rates = {'EUR': 1.1, 'GBP': 1.27, 'USD': 1.0}\n doc['metadata']['price_usd'] = price * rates.get(currency, 1.0)\n return doc"
}
}
}
]
```
### Advanced Filtering
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n content = doc.get('content', '')\n # Complex filtering logic\n word_count = len(content.split())\n if word_count < 50:\n return None\n # Check for required sections\n required = ['introduction', 'conclusion']\n content_lower = content.lower()\n if not all(section in content_lower for section in required):\n return None\n doc['metadata']['word_count'] = word_count\n return doc"
}
}
}
]
```
## Security
| Restriction | Description |
| ----------- | ------------------------------ |
| Network | No outbound network access |
| Filesystem | No file system access |
| Imports | Limited to approved packages |
| Resources | Memory and CPU limits enforced |
## Allowed Packages
Built-in packages available:
* `json`, `re`, `math`, `datetime`, `collections`
* `itertools`, `functools`, `operator`
The sandbox provides a standard runtime environment for the selected `language`.
## Error Handling
| Error | Behavior |
| ----------------- | ---------------- |
| Syntax error | Stage fails |
| Runtime exception | Document skipped |
| Timeout | Document skipped |
| Memory exceeded | Stage fails |
<Tip>
Always handle missing fields gracefully using `.get()` with defaults to avoid runtime errors.
</Tip>
## Debugging
Enable debug mode to see execution details:
```json theme={null}
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n print(f'Processing: {doc.get(\"document_id\")}')\n return doc"
}
}
}
```
## Related
* [JSON Transform](/docs/retrieval/stages/json-transform) - Template-based transforms
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - AI-powered enrichment
* [Attribute Filter](/docs/retrieval/stages/attribute-filter) - Standard filtering
# Cross Compare
Source: https://docs.mixpeek.com/docs/retrieval/stages/cross-compare
Multi-tier cross-collection content matching with configurable classification
<Frame>
<img alt="Cross Compare stage showing multi-tier matching cascade between source and reference collections" />
</Frame>
The Cross Compare stage compares source documents against a reference collection using a cascading match strategy: exact → fuzzy → semantic → visual. Each match is classified using configurable rules, enabling drift detection, deduplication, and compliance checking workflows.
<Note>
**Stage Category**: APPLY (Cross-collection comparison)
**Transformation**: N documents → M finding documents (`findings` mode) or N documents → N enriched documents (`enrich` mode)
</Note>
## When to Use
| Use Case | Description |
| ------------------------------ | --------------------------------------------------------------- |
| **Content drift detection** | Compare video UI against documentation to find outdated content |
| **Product catalog matching** | Match supplier products against internal catalog |
| **Content deduplication** | Check new content against existing corpus |
| **Compliance checking** | Verify content against requirements or standards |
| **Cross-reference validation** | Validate labels, features, or terms across sources |
## When NOT to Use
| Scenario | Recommended Alternative |
| --------------------------- | -------------------------------------- |
| Simple field joins | `document_enrich` |
| External API enrichment | `api_call` |
| Single-collection filtering | `attribute_filter` or `feature_search` |
| Semantic similarity search | `feature_search` |
## Parameters
### Core Parameters
| Parameter | Type | Default | Description |
| ------------------------- | ------ | ---------- | ------------------------------------------------------------- |
| `reference_collection_id` | string | *Required* | Collection containing reference documents to compare against |
| `source_field` | string | `content` | Field on source documents to extract comparison elements from |
| `reference_field` | string | `content` | Field on reference documents containing comparison content |
| `extraction_mode` | string | `raw` | How to extract elements: `raw`, `lines`, `labels`, or `list` |
### Matching Configuration
| Parameter | Type | Default | Description |
| -------------------- | --------- | -------------------- | ---------------------------------------------------------- |
| `match_tiers` | string\[] | `["exact", "fuzzy"]` | Ordered matching cascade. Stops at first successful match. |
| `fuzzy_threshold` | float | `0.75` | Minimum fuzzy score to accept a match |
| `semantic_threshold` | float | `0.85` | Minimum semantic similarity to accept |
| `visual_threshold` | float | `0.55` | Minimum visual similarity to accept |
### Classification
| Parameter | Type | Default | Description |
| ----------------- | --------- | ---------- | ------------------------------------------------- |
| `classifications` | object\[] | See below | Score-to-label mapping rules (evaluated in order) |
| `no_match_label` | string | `no_match` | Label when no tier matches |
Default classification rules:
```json theme={null}
[
{"min_score": 0.95, "label": "exact_match"},
{"min_score": 0.85, "label": "close_match"},
{"min_score": 0.65, "label": "partial_match"},
{"min_score": 0.0, "label": "no_match"}
]
```
### Output Configuration
| Parameter | Type | Default | Description |
| -------------- | ------ | -------------------- | ---------------------------------------- |
| `output_mode` | string | `findings` | `findings` (N-to-M) or `enrich` (1-to-1) |
| `output_field` | string | `comparison_results` | Field name for results in `enrich` mode |
### Visual Comparison
| Parameter | Type | Default | Description |
| --------------------------- | ------- | ------------------------------------------ | ------------------------------------------ |
| `include_visual_comparison` | boolean | `false` | Enable visual embedding comparison |
| `text_vector_index` | string | `intfloat__multilingual_e5_large_instruct` | Vector index for semantic matching |
| `image_vector_index` | string | `google__siglip_base_patch16_224` | SigLIP vector index |
| `structure_vector_index` | string | `facebook__dinov2_base` | DINOv2 vector index |
| `dinov2_weight` | float | `0.7` | Weight for DINOv2 in combined visual score |
| `siglip_weight` | float | `0.3` | Weight for SigLIP in combined visual score |
### Reference & Source Configuration
| Parameter | Type | Default | Description |
| ------------------------ | ------- | ------------ | ----------------------------------------------------- |
| `reference_limit` | integer | `200` | Max reference documents to fetch |
| `reference_doc_type` | string | `null` | Filter reference docs by doc\_type |
| `source_location_field` | string | `start_time` | Field containing location reference (timestamp, page) |
| `source_doc_type_filter` | string | `null` | Only process source docs with this doc\_type |
| `filter_generic_labels` | boolean | `true` | Filter generic UI labels in `labels` mode |
## Extraction Modes
<Tabs>
<Tab title="raw">
Use the field value as a single element. Best for comparing whole content blocks.
```json theme={null}
{"extraction_mode": "raw"}
```
</Tab>
<Tab title="lines">
Split by newlines. Each line becomes a comparison element. Useful for step-by-step instructions or structured text.
```json theme={null}
{"extraction_mode": "lines"}
```
</Tab>
<Tab title="labels">
Extract UI/feature labels via pattern matching. Identifies instruction patterns ("Click **Settings**"), em-dash separators ("Label — description"), and action labels ("Configure X").
Generic labels like "Save", "Cancel", "Next" are filtered by default.
```json theme={null}
{"extraction_mode": "labels", "filter_generic_labels": true}
```
</Tab>
<Tab title="list">
Field is already a list of elements. Used directly without extraction.
```json theme={null}
{"extraction_mode": "list"}
```
</Tab>
</Tabs>
## Matching Cascade
The matching cascade tries each tier in order and stops at the first successful match:
```
For each source element:
├─ exact: Case-insensitive string match → score = 1.0
├─ fuzzy: SequenceMatcher ratio ≥ fuzzy_threshold
├─ semantic: Vector similarity ≥ semantic_threshold
└─ visual: DINOv2 + SigLIP similarity ≥ visual_threshold
```
If no tier matches, the element receives `match_tier: "none"` and the `no_match_label` classification.
## Configuration Examples
<CodeGroup>
```json Content Drift Detection theme={null}
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_documentation",
"source_field": "content",
"reference_field": "content",
"extraction_mode": "labels",
"match_tiers": ["exact", "fuzzy", "semantic"],
"include_visual_comparison": true,
"source_doc_type_filter": "scene",
"source_location_field": "start_time",
"classifications": [
{"min_score": 0.95, "label": "current"},
{"min_score": 0.75, "label": "needs_review"},
{"min_score": 0.0, "label": "outdated"}
]
}
}
}
```
```json Product Catalog Matching theme={null}
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_internal_catalog",
"source_field": "product_name",
"reference_field": "product_name",
"extraction_mode": "raw",
"match_tiers": ["exact", "fuzzy"],
"fuzzy_threshold": 0.80,
"output_mode": "enrich",
"output_field": "catalog_match",
"classifications": [
{"min_score": 0.95, "label": "exact_match"},
{"min_score": 0.80, "label": "likely_match"},
{"min_score": 0.0, "label": "no_match"}
]
}
}
}
```
```json Content Deduplication theme={null}
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_existing_corpus",
"source_field": "content",
"reference_field": "content",
"extraction_mode": "lines",
"match_tiers": ["exact", "fuzzy", "semantic"],
"semantic_threshold": 0.90,
"output_mode": "enrich",
"output_field": "duplication_analysis",
"classifications": [
{"min_score": 0.95, "label": "duplicate"},
{"min_score": 0.80, "label": "near_duplicate"},
{"min_score": 0.0, "label": "unique"}
]
}
}
}
```
```json Compliance Checking theme={null}
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_requirements",
"source_field": "content",
"reference_field": "requirement_text",
"extraction_mode": "lines",
"match_tiers": ["exact", "fuzzy", "semantic"],
"fuzzy_threshold": 0.70,
"semantic_threshold": 0.80,
"output_mode": "findings",
"classifications": [
{"min_score": 0.90, "label": "compliant"},
{"min_score": 0.70, "label": "partial"},
{"min_score": 0.0, "label": "non_compliant"}
]
}
}
}
```
</CodeGroup>
## Output Schema
### Findings Mode
Each comparison produces a finding document:
```json theme={null}
{
"element_type": "text",
"source_content": "Configure API Keys",
"source_location": "00:01:23",
"reference_match": "API Key Configuration",
"reference_url": "https://docs.example.com/api-keys",
"match_tier": "fuzzy",
"match_score": 0.87,
"classification": "close_match",
"confidence": 0.92,
"signals": {
"context_match": true,
"workflow_match": false,
"transcript_match": true
}
}
```
### Enrich Mode
Comparison results attached as a field on source documents:
```json theme={null}
{
"document_id": "doc_source_123",
"content": "...",
"comparison_results": [
{
"element_type": "text",
"source_content": "Configure API Keys",
"match_tier": "fuzzy",
"match_score": 0.87,
"classification": "close_match",
"confidence": 0.92
}
]
}
```
### Finding Fields
| Field | Type | Description |
| ----------------- | ------ | --------------------------------------------------------- |
| `element_type` | string | Type of element: `text`, `code`, `visual`, or custom |
| `source_content` | string | Content from the source document |
| `source_location` | string | Location reference (timestamp, page number) |
| `reference_match` | string | Best matching content from reference |
| `reference_url` | string | URL or ID of matched reference document |
| `match_tier` | string | Tier used: `exact`, `fuzzy`, `semantic`, `visual`, `none` |
| `match_score` | float | Match score (0.0 - 1.0) |
| `classification` | string | Label from classification rules |
| `confidence` | float | Multi-signal confidence (0.0 - 1.0) |
| `signals` | object | Corroborating signals used in confidence |
## Performance
| Scenario | Expected Latency | Notes |
| -------------------------------- | ---------------- | -------------------------- |
| Exact + fuzzy only (50 docs) | 50-200ms | In-memory string matching |
| With semantic tier (50 docs) | 200-500ms | MVS vector queries |
| With visual comparison (50 docs) | 500-1500ms | Multiple vector queries |
| Large reference set (200 docs) | 300-800ms | More candidates to compare |
<Tip>
Reference documents are fetched once and reused across all source documents. The matching cascade short-circuits at the first successful tier, so ordering `match_tiers` from fastest to slowest (exact → fuzzy → semantic → visual) is optimal.
</Tip>
**Limits:**
* Max source documents per execution: 50
* Max reference documents fetched: 200 (configurable via `reference_limit`)
## Common Pipeline Patterns
### Drift Detection Pipeline
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
}],
"final_top_k": 50
}
}
},
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_documentation",
"source_field": "content",
"reference_field": "content",
"extraction_mode": "labels",
"match_tiers": ["exact", "fuzzy", "semantic"],
"include_visual_comparison": true,
"classifications": [
{"min_score": 0.95, "label": "current"},
{"min_score": 0.75, "label": "needs_review"},
{"min_score": 0.0, "label": "outdated"}
]
}
}
}
]
```
### Catalog Match + Transform
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 30
}],
"final_top_k": 30
}
}
},
{
"stage_name": "cross_compare",
"stage_type": "apply",
"config": {
"stage_id": "cross_compare",
"parameters": {
"reference_collection_id": "col_reference_catalog",
"source_field": "product_name",
"reference_field": "product_name",
"match_tiers": ["exact", "fuzzy"],
"output_mode": "enrich",
"output_field": "match_result"
}
}
},
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"product\": \"{{ DOC.product_name }}\", \"match_status\": \"{{ DOC.match_result[0].classification }}\", \"score\": {{ DOC.match_result[0].match_score }}}"
}
}
}
]
```
## Error Handling
| Error | Behavior |
| -------------------------------- | ---------------------------------------- |
| Reference collection not found | Stage fails with error |
| No reference documents found | All elements classified as `no_match` |
| Vector index not available | Semantic/visual tiers skipped silently |
| Source field missing on document | Document skipped |
| Exceeds max\_working\_documents | Extra documents passed through unchanged |
## vs Other Enrichment Stages
| Feature | cross\_compare | document\_enrich | api\_call |
| --------------- | -------------------------------------------- | ------------------------- | ----------------------- |
| **Purpose** | Multi-tier comparison with classification | Simple field join/lookup | External API enrichment |
| **Data source** | Internal MVS namespaces | Internal MVS namespaces | External HTTP APIs |
| **Matching** | Cascading: exact → fuzzy → semantic → visual | Top-1 vector or key match | N/A |
| **Output** | Classified findings with scores | Joined fields | API response |
| **Latency** | 50-1500ms | 5-20ms | 100-500ms |
| **Best for** | Drift detection, dedup, compliance | Cross-collection joins | Third-party data |
## Related
* [Document Enrich](/docs/retrieval/stages/document-enrich) - Simple cross-collection joins
* [Feature Search](/docs/retrieval/stages/feature-search) - Vector search (often used before cross\_compare)
* [JSON Transform](/docs/retrieval/stages/json-transform) - Transform comparison output
* [Taxonomy Enrich](/docs/retrieval/stages/taxonomy-enrich) - Classification enrichment
# Deduplicate
Source: https://docs.mixpeek.com/docs/retrieval/stages/deduplicate
Remove duplicate documents by field match or content similarity
<Frame>
<img alt="Deduplicate stage showing removal of duplicate documents" />
</Frame>
The Deduplicate stage removes duplicate documents from the result set based on exact field matching or content similarity. This is analogous to SQL's `DISTINCT`, MongoDB's `$group` with `$first`, and Elasticsearch's field collapsing.
<Note>
**Stage Category**: REDUCE (Removes duplicates)
**Transformation**: N documents → M documents (M ≤ N, duplicates removed)
</Note>
## When to Use
| Use Case | Description |
| --------------------------- | ------------------------------------------------- |
| **URL deduplication** | One result per source URL after web enrichment |
| **Author collapse** | Keep one result per author |
| **Content dedup** | Remove near-identical text chunks |
| **Multi-source merge** | Remove overlapping results from multiple searches |
| **Query expansion cleanup** | Remove duplicates from expanded query results |
## When NOT to Use
| Scenario | Recommended Alternative |
| -------------------------- | ------------------------ |
| Grouping with aggregation | `group_by` stage |
| Sampling unique categories | `sample` with stratified |
| Limiting result count | `limit` stage |
| Filtering by criteria | `attribute_filter` |
## Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ------------- | -------------------- | ------------------------------------------------------------- |
| `strategy` | string | `field` | Dedup method: `field` (exact match) or `content` (similarity) |
| `fields` | list\[string] | *required for field* | Field paths to compare for deduplication |
| `content_field` | string | `content` | Text field for content-based dedup |
| `similarity_threshold` | float | `0.95` | Similarity threshold for content dedup (0.0-1.0) |
| `keep` | string | `first` | Which duplicate to keep: `first` or `last` |
| `case_sensitive` | boolean | `true` | Whether string comparisons are case-sensitive |
## Deduplication Strategies
| Strategy | Performance | Best For |
| --------- | --------------- | ------------------------------------- |
| `field` | O(N) hash-based | Exact field matching (URL, ID, title) |
| `content` | O(N²) pairwise | Near-duplicate text detection |
## Configuration Examples
<CodeGroup>
```json Deduplicate by URL theme={null}
{
"stage_name": "deduplicate",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "field",
"fields": ["metadata.source_url"],
"keep": "first"
}
}
}
```
```json Case-Insensitive Author Dedup theme={null}
{
"stage_name": "deduplicate",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "field",
"fields": ["metadata.author"],
"case_sensitive": false
}
}
}
```
```json Multi-Field Dedup theme={null}
{
"stage_name": "deduplicate",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "field",
"fields": ["metadata.author", "metadata.title"]
}
}
}
```
```json Content Similarity Dedup theme={null}
{
"stage_name": "deduplicate",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "content",
"content_field": "content",
"similarity_threshold": 0.9,
"keep": "first"
}
}
}
```
</CodeGroup>
<Tip>
For best results, place deduplicate after sorting/reranking so that `keep: "first"` retains the highest-scored duplicate. This ensures you keep the most relevant version of each document.
</Tip>
## Performance
| Metric | Value |
| -------------- | ---------------------------------------------------- |
| **Latency** | \< 5ms (field) / 10-100ms (content) |
| **Memory** | O(N) hash set (field) / O(N) content cache (content) |
| **Cost** | Free |
| **Complexity** | O(N) field / O(N²) content |
## Common Pipeline Patterns
### Web Search Deduplication
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 50}],
"final_top_k": 50
}
}
},
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 10
}
}
},
{
"stage_name": "deduplicate",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "field",
"fields": ["metadata.source_url"]
}
}
}
]
```
### Cross-Collection Dedup
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 100}],
"final_top_k": 100
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "content"
}
}
},
{
"stage_name": "deduplicate",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "content",
"content_field": "content",
"similarity_threshold": 0.85
}
}
}
]
```
## Error Handling
| Error | Behavior |
| -------------------- | ------------------------------------------------------ |
| Field doesn't exist | Documents with missing fields have `None` as key value |
| All unique documents | Returns all documents unchanged |
| Empty input | Returns empty result set |
| Single document | Returned as-is (no duplicates possible) |
## Related
* [Group By](/docs/retrieval/stages/group-by) - Group documents with aggregation
* [Limit](/docs/retrieval/stages/limit) - Truncate results after deduplication
* [Sample](/docs/retrieval/stages/sample) - Random sampling (different from dedup)
* [Unwind](/docs/retrieval/stages/unwind) - Inverse: expand grouped items
# External Web Search
Source: https://docs.mixpeek.com/docs/retrieval/stages/external-web-search
Augment results with real-time web search using Exa's neural search API
<Frame>
<img alt="External Web Search stage showing Exa API integration for web results" />
</Frame>
The External Web Search stage integrates Exa's neural search API to augment your results with real-time web content. This enables hybrid retrieval combining your indexed documents with fresh web results.
<Note>
**Stage Category**: APPLY (Enriches pipeline with web results)
**Transformation**: N documents → N + M documents (web results added)
</Note>
## When to Use
| Use Case | Description |
| ---------------------------- | ----------------------------------------- |
| **Knowledge augmentation** | Supplement internal docs with web content |
| **Real-time information** | Access current events, news, updates |
| **Research expansion** | Broaden search beyond your corpus |
| **Competitive intelligence** | Include competitor content in results |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------ | ------------------------- |
| Internal-only search | Skip this stage |
| Sensitive/confidential queries | Use only indexed content |
| Low-latency requirements | Web search adds 200-500ms |
## Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ------- | ----------------- | ----------------------------------------- |
| `query` | string | `{{INPUT.query}}` | Search query (supports templates) |
| `num_results` | integer | `10` | Number of web results to retrieve (1-100) |
| `start_published_date` | string | `null` | Filter by publish date (`YYYY-MM-DD`) |
| `category` | string | `null` | Content category filter |
| `use_autoprompt` | boolean | `true` | Let Exa optimize the query |
| `include_text` | boolean | `true` | Include text snippets in results |
## Available Categories
| Category | Description |
| ---------------- | ----------------------------- |
| `company` | Company websites and profiles |
| `research_paper` | Academic and research content |
| `news` | News articles |
| `pdf` | PDF documents |
| `github` | GitHub repositories |
| `tweet` | Twitter/X content |
| `personal_site` | Personal websites and blogs |
## Configuration Examples
<CodeGroup>
```json Basic Web Search theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 10
}
}
}
```
```json Category-Filtered Search theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 5,
"category": "research_paper"
}
}
}
```
```json Recent News Search theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 10,
"category": "news",
"start_published_date": "2024-01-01"
}
}
}
```
```json Research Papers theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 20,
"category": "research_paper",
"include_text": true
}
}
}
```
```json GitHub Code Search theme={null}
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}} implementation",
"num_results": 10,
"category": "github",
"use_autoprompt": false
}
}
}
```
</CodeGroup>
## Content Extraction
Set `include_text` to `true` (default) to include text snippets in each result. Disable it to reduce API costs and response size.
## Output Schema
Web results are added to the document set with a `source: "web"` marker:
```json theme={null}
{
"document_id": "web_abc123",
"source": "web",
"url": "https://example.com/article",
"title": "Article Title",
"content": "Full extracted text content...",
"published_date": "2024-03-15T10:30:00Z",
"author": "John Doe",
"score": 0.95,
"metadata": {
"domain": "example.com",
"category": "news"
}
}
```
## Exa Neural Search
Exa uses neural search rather than keyword matching:
| Feature | Description |
| -------------------------- | ------------------------------------- |
| **Semantic understanding** | Understands query intent |
| **Neural ranking** | ML-based relevance scoring |
| **Content extraction** | Automatic text extraction |
| **Autoprompt** | Query optimization for better results |
<Tip>
Enable `use_autoprompt` (default) for natural language queries. Disable it when you need exact phrase matching or have already optimized your query.
</Tip>
## Performance
| Metric | Value |
| ---------------------- | ------------------------- |
| **Latency** | 200-500ms |
| **Rate limits** | Based on Exa plan |
| **Parallel execution** | Concurrent with pipeline |
| **Caching** | Results cached for 1 hour |
## Common Pipeline Patterns
### Internal + Web Hybrid Search
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 20 }
],
"final_top_k": 20
}
}
},
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 10
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
}
]
```
### Web-Augmented RAG
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 30 }
],
"final_top_k": 30
}
}
},
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 5,
"category": "news",
"start_published_date": "{{INPUT.date_filter}}"
}
}
},
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 8000,
"output_mode": "single_context"
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ---------------- | -------------------------------------- |
| API rate limit | Retry with backoff |
| Network timeout | Stage fails gracefully, no web results |
| Invalid domain | Ignored, other domains searched |
| No results found | Empty web result set |
<Warning>
Web search results may include content from untrusted sources. Consider filtering or validating web content before using in sensitive applications.
</Warning>
## Related
* [Web Scrape](/docs/retrieval/stages/web-scrape) - Extract content from specific URLs
* [Feature Search](/docs/retrieval/stages/feature-search) - Search your indexed content
* [Rerank](/docs/retrieval/stages/rerank) - Combine and rank mixed results
# Feature Search
Source: https://docs.mixpeek.com/docs/retrieval/stages/feature-search
Unified semantic and hybrid search across multiple embedding features with configurable fusion strategies
<Frame>
<img alt="Feature Search stage showing multi-vector semantic search with fusion" />
</Frame>
The Feature Search stage is the **primary search stage** for retrieval pipelines. It performs vector similarity search across one or more embedding features, supporting single-modal, multimodal, and hybrid search patterns. Results from multiple searches are fused using configurable strategies (RRF, DBSF, weighted, max, or learned).
<Note>
**Stage Category**: FILTER (Retrieves documents)
**Transformation**: 0 documents → N documents (retrieves from collection based on vector similarity)
</Note>
<Card title="Run feature search on your own data" icon="magnifying-glass" href="https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=docs&utm_medium=docs_feature_search&utm_campaign=managed_onramp">
Create a managed namespace, extract embeddings from your own files, then compose this stage into a retriever.
</Card>
## When to Use
| Use Case | Description |
| ----------------------- | -------------------------------------------- |
| **Semantic search** | Find documents similar in meaning to a query |
| **Image search** | Search by image embeddings |
| **Video search** | Search by video frame embeddings |
| **Multimodal search** | Combine text + image + video in one query |
| **Hybrid search** | Fuse results from multiple embedding types |
| **Decompose/recompose** | Group results by parent document |
| **Faceted search** | Get result counts by field values |
## When NOT to Use
| Scenario | Recommended Alternative |
| --------------------------- | -------------------------- |
| Exact field matching | `attribute_filter` |
| Full-text keyword search | Combine with text features |
| No embeddings in collection | `attribute_filter` |
| Post-search filtering only | Use after `feature_search` |
## Core Concepts
### Feature URIs
Feature URIs identify which embedding index to search. They follow the pattern:
```
mixpeek://{extractor_name}@{version}/{output_name}
```
**Examples:**
* `mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding` - Multimodal text/image/video embeddings
* `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1` - Text-only embeddings
* `mixpeek://image_extractor@v1/google_siglip_base_v1` - Image embeddings
* `mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1` - Speech/transcript embeddings (from video/audio)
### Fusion Strategies
When searching multiple features, results are combined using fusion:
| Strategy | Description | Best For |
| ---------- | ------------------------------- | --------------------------------------------------------------------------- |
| `rrf` | Reciprocal Rank Fusion | General purpose, balanced results |
| `dbsf` | Distribution-Based Score Fusion | When scores have different distributions |
| `weighted` | Weighted combination | When you know relative importance |
| `max` | Maximum score wins | When any match is sufficient |
| `learned` | Thompson Sampling bandit | Automatically adapts per-user from [interaction data](/docs/retrieval/auto-tune) |
### Learned Fusion Configuration
When `fusion` is set to `"learned"`, you can provide a `learning_config` object to control how the bandit adapts. See [Auto-Tune](/docs/retrieval/auto-tune) for a full walkthrough.
<CodeGroup>
```json Learned Fusion with learning_config theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
}
],
"fusion": "learned",
"learning_config": {
"context_features": ["INPUT.user_id"],
"demographic_features": ["INPUT.user_segment"],
"reward_map": {
"click": 1.0,
"purchase": 3.0,
"add_to_cart": 2.0,
"bookmark": 1.5,
"positive_feedback": 2.0,
"negative_feedback": -2.0,
"skip": -0.5
},
"min_interactions": 5,
"exploration_bonus": 1.0,
"exploration_decay": 0.99,
"exploration_floor": 0.1,
"decay_factor": 0.995,
"decay_window_days": 365,
"min_weight": 0.05,
"max_weight": 0.95,
"rollout_pct": 100.0,
"shadow_mode": false
},
"final_top_k": 25
}
}
}
```
</CodeGroup>
#### learning\_config Fields
| Field | Type | Default | Description |
| ---------------------- | ---------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `context_features` | `string[]` | `["INPUT.user_id"]` | Input fields for personal-level learning. References `INPUT.*` fields from the retriever's `input_schema`. |
| `demographic_features` | `string[]` | `[]` | Input fields for segment-level fallback (e.g., `"INPUT.user_segment"`). |
| `reward_signal` | `string` | `"click"` | *Deprecated.* Use `reward_map` instead. |
| `reward_map` | `object` | [See defaults](/docs/retrieval/reward-signals) | Maps interaction types to reward magnitudes. Positive values reinforce the associated feature; negative values penalize it. |
| `min_interactions` | `integer` | `5` | Minimum interactions before personal-level weights are used. Below this, falls back to demographic or global. |
| `exploration_bonus` | `float` | `1.0` | Initial multiplier for weight distribution variance. |
| `exploration_decay` | `float` | `0.99` | Per-interaction decay of `exploration_bonus`. |
| `exploration_floor` | `float` | `0.1` | Minimum exploration bonus (prevents full exploitation). |
| `decay_factor` | `float` | `0.995` | Per-day exponential decay on older interactions. `1.0` = no decay. |
| `decay_window_days` | `integer` | `365` | Interactions older than this are excluded entirely. |
| `min_weight` | `float` | `0.05` | Floor for any feature's weight after sampling. |
| `max_weight` | `float` | `0.95` | Ceiling for any feature's weight after sampling. |
| `rollout_pct` | `float` | `100.0` | Percentage of requests using learned weights (0-100). |
| `shadow_mode` | `boolean` | `false` | Compute learned weights but serve static results. |
<Info>
See [Auto-Tune](/docs/retrieval/auto-tune) for the full concept overview, [Reward Signals](/docs/retrieval/reward-signals) for reward map customization, and [Rollout & Safety](/docs/retrieval/auto-tune-rollout) for traffic splitting and kill switch details.
</Info>
## Parameters
| Parameter | Type | Default | Description |
| ------------- | ------- | ---------- | ------------------------------------ |
| `searches` | array | *Required* | Array of search configurations |
| `final_top_k` | integer | `25` | Total results to return after fusion |
| `fusion` | string | `rrf` | Fusion strategy for multi-search |
| `group_by` | object | `null` | Group results by field |
| `facets` | array | `null` | Fields to compute facet counts |
### Search Object Parameters
Each item in the `searches` array supports:
| Parameter | Type | Default | Description |
| --------------------- | ------------- | ---------- | ----------------------------------------------------------------------------------------------------- |
| `feature_uri` | string | *Required* | Embedding index to search |
| `query` | string/object | *Required* | Query text or embedding |
| `top_k` | integer | `100` | Candidates per search |
| `filters` | object | `null` | Pre-filter conditions |
| `weight` | number | `1.0` | Weight for fusion (weighted strategy) |
| `lexical` | boolean | `false` | Run this search as keyword/BM25 instead of vector (see [Lexical (BM25) Search](#lexical-bm25-search)) |
| `query_preprocessing` | object | `null` | Large file decomposition config |
### Query Input Modes
The `query` field on each search object accepts either a plain string (shorthand for `text` mode) or an object with an explicit `input_mode`:
| Mode | `input_mode` | Value | Supported by |
| ----------------- | ----------------- | --------------------------------------- | --------------------------------- |
| **Text** | `"text"` | Plain text string | All text-capable extractors |
| **Content** | `"content"` | Single URL or base64 data URI | All multimodal extractors |
| **Document** | `"document"` | Reference to an existing document | All extractors |
| **Vector** | `"vector"` | Pre-computed embedding (list of floats) | All extractors |
| **Multi-content** | `"multi_content"` | List of URLs and/or text strings | `gemini_multifile_extractor` only |
**Text** — embed a string and search:
```json theme={null}
{"input_mode": "text", "value": "{{INPUT.query}}"}
```
**Content** — fetch a URL and embed it:
```json theme={null}
{"input_mode": "content", "value": "{{INPUT.image_url}}"}
```
**Vector** — use a pre-computed embedding directly (no inference at query time):
```json theme={null}
{"input_mode": "vector", "value": "{{INPUT.embedding}}"}
```
**Multi-content** — embed multiple files together in one API call. Only valid when the `feature_uri` points to an extractor whose vector index has `supports_multi_query=True` (currently: `gemini_multifile_extractor`). Attempting this with any other feature URI returns a `400` error.
```json theme={null}
{
"input_mode": "multi_content",
"values": ["{{INPUT.image_url}}", "{{INPUT.description}}"]
}
```
Each item in `values` is auto-detected: URLs (`http://`, `https://`, `s3://`) are fetched and embedded as files; all other strings are embedded as text. All items are passed to the underlying model in one call, producing a single query vector that mirrors how objects were indexed.
### Lexical (BM25) Search
Set `lexical: true` on a search to run **keyword/BM25** matching instead of vector similarity. The query text is matched against the namespace's [full-text index](/docs/vector-store/namespaces#text-indexes-bm25) — it is **not** embedded into a vector. BM25 catches exact tokens that dense embeddings routinely miss: brand names, SKUs, prices like `$9.99`, promo codes, error strings, and CTAs.
| Behavior | Detail |
| ----------------- | ---------------------------------------------------------------------------------------------------------- |
| **Input** | Must be text (`input_mode: "text"`). The query string is used verbatim. |
| **Matching** | Across **all** `text`-indexed string payload fields — not a single field. |
| **`feature_uri`** | Used only for collection scoping; no vector index is queried. |
| **Prerequisite** | A `text` payload index must exist (see [Text Indexes (BM25)](/docs/vector-store/namespaces#text-indexes-bm25)). |
<Note>
**Searching only one field (e.g. OCR text).** BM25 matches across *all* `text`-indexed string fields — it can't be scoped to a single field like `ocr_text`. To make one field independently searchable, give it its own dense index by running a `text_extractor` over it (map the extractor's input to `ocr_text`), then `feature_search` that feature URI directly. For coarse exact-substring filtering on a single field, an [`attribute_filter`](/docs/retrieval/stages/attribute-filter) with the `contains` operator works but is not relevance-ranked.
</Note>
The real power is **hybrid retrieval** — fuse a dense (vector) search with a lexical (BM25) search under `rrf` so semantic recall and exact-keyword precision reinforce each other:
```json Dense + Lexical Hybrid (RRF) theme={null}
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"fusion": "rrf",
"final_top_k": 25,
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
},
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"lexical": true,
"top_k": 100
}
]
}
}
}
```
<Tip>
Use `rrf` fusion for dense+lexical hybrid — it ranks by position, so it is immune to the score-scale mismatch between cosine similarity and BM25. Avoid `weighted`/`max` here unless you have a specific reason.
</Tip>
### Query Preprocessing
When searching with large files (videos, PDFs, long documents) as input, `query_preprocessing` decomposes the file into chunks using the same extractor pipeline that indexed your data, runs parallel searches for each chunk, and fuses the results.
This is **ingestion applied to the query** — same decomposition and embedding, but vectors are used for search instead of storage.
| Parameter | Type | Default | Description |
| ------------- | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `feature_uri` | string | `null` | Extractor pipeline for decomposition (inherits from search `feature_uri` if not set) |
| `params` | object | `null` | Extractor parameters — **identical schema to the collection's extractor config for that `feature_uri`** |
| `max_chunks` | integer | `20` | Max chunks to search (1-100). Each chunk runs its own search and adds query cost — reads are metered (see [Billing](/docs/platform/billing)) |
| `aggregation` | string | `rrf` | Fusion strategy: `rrf`, `max`, or `avg` |
| `dedup_field` | string | `null` | Field to deduplicate results by |
<Info>
**`params` uses the extractor's own parameter schema.** Whatever parameters the extractor accepts during ingestion (e.g. `split_method`, `time_split_interval` for video; `chunk_size`, `chunk_overlap` for text) are the same parameters you pass here. There is no separate preprocessing-specific schema — the extractor drives the decomposition exactly as it would during collection processing. Refer to the extractor's own documentation for valid parameter names.
</Info>
<Info>
You can also set `query_preprocessing` at the **stage level** (on `parameters`) to apply it to all searches as a default. Per-search settings override the stage default.
</Info>
**Aggregation strategies:**
| Strategy | Best For | How It Works |
| -------- | ------------------------------ | -------------------------------------------------------- |
| `rrf` | General purpose (recommended) | Rank-based fusion, immune to score magnitude differences |
| `max` | "Find this exact moment" | Keeps highest score per document across chunks |
| `avg` | "Find similar overall content" | Averages scores — consistent matches win |
## Configuration Examples
<CodeGroup>
```json Basic Text Search theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"final_top_k": 25
}
}
}
```
```json Image Search theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": "{{INPUT.image_url}}",
"top_k": 50
}
],
"final_top_k": 20
}
}
}
```
```json Multimodal Hybrid Search theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": "{{INPUT.image_url}}",
"top_k": 100
}
],
"fusion": "rrf",
"final_top_k": 25
}
}
}
```
```json Weighted Fusion theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100,
"weight": 0.7
},
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": "{{INPUT.image_url}}",
"top_k": 100,
"weight": 0.3
}
],
"fusion": "weighted",
"final_top_k": 20
}
}
}
```
```json With Pre-Filters theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100,
"filters": {
"AND": [
{"field": "metadata.status", "operator": "eq", "value": "published"},
{"field": "metadata.category", "operator": "in", "value": ["tech", "science"]}
]
}
}
],
"final_top_k": 25
}
}
}
```
```json With Grouping (Decompose/Recompose) theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 200
}
],
"group_by": {
"field": "metadata.parent_id",
"limit": 10,
"group_size": 3
},
"final_top_k": 30
}
}
}
```
```json With Facets theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"facets": ["metadata.category", "metadata.author", "metadata.year"],
"final_top_k": 25
}
}
}
```
```json Multi-file Query (Gemini Multifile) theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://gemini_multifile_extractor@v1/gemini-embedding-exp-03-07",
"query": {
"input_mode": "multi_content",
"values": [
"{{INPUT.image_url}}",
"{{INPUT.spec_sheet_url}}",
"{{INPUT.description}}"
]
},
"top_k": 20
}
],
"final_top_k": 10
}
}
}
```
</CodeGroup>
## Query Preprocessing Examples
Search with a large video — decompose it into 10-second segments, search each, and fuse:
<CodeGroup>
```json Video Search with Preprocessing theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": {"input_mode": "content", "value": "{{INPUT.video}}"},
"top_k": 100,
"query_preprocessing": {
"params": {"split_method": "time", "time_split_interval": 10},
"max_chunks": 20,
"aggregation": "max"
}
}
],
"final_top_k": 25
}
}
}
```
```json PDF Search (Page-by-Page) theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://image_extractor@v1/siglip_embedding",
"query": {"input_mode": "content", "value": "{{INPUT.pdf_document}}"},
"top_k": 100,
"query_preprocessing": {
"max_chunks": 50,
"aggregation": "rrf"
}
}
],
"final_top_k": 25
}
}
}
```
```json Mixed: Video (Preprocessed) + Text (Normal) theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": {"input_mode": "content", "value": "{{INPUT.video}}"},
"query_preprocessing": {
"params": {"time_split_interval": 10},
"aggregation": "max"
}
},
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": {"input_mode": "text", "value": "{{INPUT.text_query}}"}
}
],
"final_top_k": 25
}
}
}
```
```json Stage-Level Default (All Searches) theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"query_preprocessing": {
"max_chunks": 15,
"aggregation": "rrf"
},
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": {"input_mode": "content", "value": "{{INPUT.video1}}"}
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": {"input_mode": "content", "value": "{{INPUT.video2}}"}
}
],
"final_top_k": 25
}
}
}
```
</CodeGroup>
<Note>
Preprocessing uses the same extractor pipeline that indexed your data. The `params` accept the same fields you configured on your collection's feature extractor (e.g., `split_method`, `chunk_size`). If you don't specify `params`, extractor defaults are used.
</Note>
The response includes preprocessing metadata showing what happened:
```json theme={null}
{
"metadata": {
"preprocessing": {
"content_type": "video/mp4",
"extractor": "multimodal_extractor@v1",
"chunks_generated": 18,
"chunks_searched": 18,
"aggregation": "rrf",
"preprocessing_ms": 12450
}
}
}
```
Each result also includes `query_chunks` showing which parts of your query matched:
```json theme={null}
{
"document_id": "doc_abc123",
"score": 0.89,
"query_chunks": [
{"chunk_index": 0, "start_ms": 0, "end_ms": 10000, "score": 0.92},
{"chunk_index": 2, "start_ms": 20000, "end_ms": 30000, "score": 0.87}
]
}
```
## Grouping (Decompose/Recompose)
When documents are decomposed into chunks (e.g., video frames, document pages), use `group_by` to recompose results by parent:
```json theme={null}
{
"group_by": {
"field": "metadata.parent_id",
"limit": 10,
"group_size": 3
}
}
```
| Parameter | Description |
| ------------ | -------------------------------------------- |
| `field` | Field to group by (e.g., parent document ID) |
| `limit` | Maximum number of groups to return |
| `group_size` | Maximum documents per group |
**Use cases:**
* Video search: Group frames by video, return top 3 frames per video
* Document search: Group chunks by document, return best chunks per doc
* Product search: Group variants by product family
## Faceted Search
Get counts of results by field values for building filter UIs:
```json theme={null}
{
"facets": ["metadata.category", "metadata.brand", "metadata.price_range"]
}
```
**Response includes:**
```json theme={null}
{
"facets": {
"metadata.category": [
{"value": "electronics", "count": 45},
{"value": "clothing", "count": 23}
],
"metadata.brand": [
{"value": "Apple", "count": 12},
{"value": "Samsung", "count": 8}
]
}
}
```
## Filter Syntax
<Info>
Filtered fields must have [payload indexes](/docs/retrieval/filters#payload-indexes) on your namespace. Without indexes, filtering is slow and the response includes warnings about unindexed fields.
</Info>
Pre-filters use boolean logic with AND/OR/NOT:
```json theme={null}
{
"filters": {
"AND": [
{"field": "metadata.status", "operator": "eq", "value": "active"},
{
"OR": [
{"field": "metadata.category", "operator": "eq", "value": "tech"},
{"field": "metadata.category", "operator": "eq", "value": "science"}
]
}
]
}
}
```
### Supported Operators
| Operator | Description | Example |
| ---------- | --------------------- | -------------------------------------------------------------------------- |
| `eq` | Equals | `{"field": "status", "operator": "eq", "value": "active"}` |
| `ne` | Not equals | `{"field": "status", "operator": "ne", "value": "deleted"}` |
| `gt` | Greater than | `{"field": "price", "operator": "gt", "value": 100}` |
| `gte` | Greater than or equal | `{"field": "rating", "operator": "gte", "value": 4}` |
| `lt` | Less than | `{"field": "age", "operator": "lt", "value": 30}` |
| `lte` | Less than or equal | `{"field": "count", "operator": "lte", "value": 10}` |
| `in` | In array | `{"field": "category", "operator": "in", "value": ["a", "b"]}` |
| `nin` | Not in array | `{"field": "status", "operator": "nin", "value": ["deleted", "archived"]}` |
| `contains` | Contains substring | `{"field": "title", "operator": "contains", "value": "guide"}` |
| `exists` | Field exists | `{"field": "metadata.optional", "operator": "exists", "value": true}` |
## Performance
| Metric | Value |
| ------------------- | ---------------------------------- |
| **Latency** | 10-50ms (single search) |
| **Latency** | 20-80ms (multi-search with fusion) |
| **Optimal top\_k** | 100-500 per search |
| **Maximum top\_k** | 10,000 per search |
| **Fusion overhead** | \< 5ms |
<Tip>
For best performance, use pre-filters to reduce the search space. Filtering at the vector index level is much faster than post-filtering in later stages.
</Tip>
## Common Pipeline Patterns
### Basic Search + Rerank
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"final_top_k": 50
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
}
]
```
### Multimodal Search + Filter + Limit
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": "{{INPUT.image}}",
"top_k": 100
}
],
"fusion": "rrf",
"final_top_k": 50
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.in_stock",
"operator": "eq",
"value": true
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"count": 20
}
}
}
]
```
### Video Search with Frame Grouping
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 500
}
],
"group_by": {
"field": "metadata.video_id",
"limit": 10,
"group_size": 5
},
"final_top_k": 50
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Summarize why these video segments match the query"
}
}
}
]
```
### E-commerce Search with Facets
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 200,
"filters": {
"AND": [
{"field": "metadata.in_stock", "operator": "eq", "value": true},
{"field": "metadata.price", "operator": "lte", "value": "{{INPUT.max_price}}"}
]
}
}
],
"facets": ["metadata.category", "metadata.brand", "metadata.color"],
"final_top_k": 50
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "{{INPUT.sort_by}}",
"direction": "{{INPUT.sort_order}}"
}
}
}
]
```
## Output Schema
Each result includes:
| Field | Type | Description |
| ------------- | ------ | ---------------------------------- |
| `document_id` | string | Unique document identifier |
| `score` | float | Combined similarity score |
| `content` | string | Document content |
| `metadata` | object | Document metadata |
| `features` | object | Feature data and scores per search |
**Example output:**
```json theme={null}
{
"document_id": "doc_abc123",
"score": 0.892,
"content": "Document content here...",
"metadata": {
"title": "Example Document",
"category": "tech",
"created_at": "2024-01-15T10:30:00Z"
},
"features": {
"mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding": {
"score": 0.91
},
"mixpeek://image_extractor@v1/google_siglip_base_v1": {
"score": 0.87
}
}
}
```
## Comparison: feature\_search vs attribute\_filter
| Aspect | feature\_search | attribute\_filter |
| ------------ | ----------------------- | --------------------- |
| **Purpose** | Semantic similarity | Exact matching |
| **Input** | Query text/embedding | Field conditions |
| **Scoring** | Vector similarity | Binary match |
| **Speed** | 10-50ms | 5-20ms |
| **Use when** | Finding similar content | Filtering by metadata |
## Error Handling
| Error | Behavior |
| --------------------- | ---------------------- |
| Invalid feature\_uri | Stage fails with error |
| Empty query | Returns empty results |
| Filter syntax error | Stage fails with error |
| No matching documents | Returns empty results |
## Creating a Retriever with feature\_search
The following is a **complete working example** of creating a retriever that uses the `feature_search` stage, then executing it. Pay close attention to the field names — several are easy to confuse.
<Warning>
**Common mistakes:**
* Use `collection_identifiers` (not `collection_ids`) in the retriever body.
* `input_schema` is a **flat map keyed by field name** (`{"query": {"type": "text"}}`) — do **not** wrap it in a JSON Schema object (`{"properties": {...}, "type": "object"}`).
* Use `type: "text"` (not `"string"`) in `input_schema` values.
* `stage_type` at the outer level must be `"filter"` — passing `stage_type: "feature_search"` is rejected (`feature_search` is a `stage_id`, not a `stage_type`).
* `stage_id: "feature_search"` lives inside the `config` object, **not** at the outer `stage_id`.
* Inside each search, the query value uses `{"input_mode": "text", "value": "..."}` — the `value` key, not a bare `text` key.
* `final_top_k` lives inside `config.parameters`, not at the top level.
* If a `feature_uri` is wrong, the error lists the `available_feature_uris` for your target collections — copy the exact URI (e.g. `multilingual_e5_large_instruct_v1`, not `embedding`).
</Warning>
### Step 1 — Create the Retriever
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "my-semantic-retriever",
"collection_identifiers": ["col_abc123"],
"input_schema": {
"query": {
"type": "text",
"description": "Search query",
"required": true
}
},
"stages": [
{
"stage_name": "Semantic Search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": {
"input_mode": "text",
"value": "{{INPUT.query}}"
},
"top_k": 50
}
],
"final_top_k": 10
}
}
}
]
}'
```
```python Python theme={null}
import httpx
response = httpx.post(
"https://api.mixpeek.com/v1/retrievers",
headers={
"Authorization": f"Bearer {api_key}",
"X-Namespace": namespace_id,
},
json={
"retriever_name": "my-semantic-retriever",
# ✅ correct field: collection_identifiers
"collection_identifiers": ["col_abc123"],
"input_schema": {
"query": {
"type": "text", # ✅ "text", not "string"
"description": "Search query",
"required": True,
}
},
"stages": [
{
"stage_name": "Semantic Search",
"stage_type": "filter", # ✅ required at outer stage
"config": {
"stage_id": "feature_search", # ✅ inside config
"parameters": {
"searches": [
{
"feature_uri": (
"mixpeek://text_extractor@v1/"
"multilingual_e5_large_instruct_v1"
),
"query": {
"input_mode": "text",
"value": "{{INPUT.query}}",
},
"top_k": 50,
}
],
"final_top_k": 10, # ✅ inside parameters
},
},
}
],
},
)
data = response.json()
retriever_id = data["retriever_id"] # retriever_id is top-level on the response
```
</CodeGroup>
### Step 2 — Execute the Retriever
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers/$RETRIEVER_ID/execute" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"inputs": {
"query": "machine learning for fashion brand compliance"
}
}'
```
```python Python theme={null}
response = httpx.post(
f"https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute",
headers={
"Authorization": f"Bearer {api_key}",
"X-Namespace": namespace_id,
},
json={
"inputs": {"query": "machine learning for fashion brand compliance"}
},
)
results = response.json()
```
</CodeGroup>
### Finding the Right feature\_uri
The `feature_uri` must match an embedding index that exists in your namespace. To discover available feature URIs, list the vector indexes in a collection:
```bash theme={null}
curl "https://api.mixpeek.com/v1/collections/$COLLECTION_ID" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
Each item in the `vector_indexes` array has a `feature_uri` field — use that value directly in your retriever stage.
## Embedding Task Conditioning
Feature search automatically applies **task-aware embedding conditioning** to instruction-aware models (E5, Gemini) at query time. This means query embeddings are optimized for asymmetric retrieval without any configuration.
**How it works:**
* **Index time:** Extractors embed documents with `retrieval_document` task (configurable via `embedding_task` on the extractor — see [Text Extractor](/docs/processing/extractors/text#embedding-task))
* **Query time:** Feature search automatically uses `retrieval_query` task for all query embeddings
This asymmetric pairing (document vs. query) improves retrieval quality by \~10% for instruction-aware models like E5-Large.
The `embedding_task` used is included in the stage response metadata:
```json theme={null}
{
"metadata": {
"embedding_task": "retrieval_query",
"num_features": 1,
"fusion_strategy": "rrf",
"total_results": 25
}
}
```
**Task-aware models:**
| Model | Task Support | Used By |
| ------------------------------------------ | ------------------------------------------ | ------------------------------------------------------ |
| E5-Large (`intfloat_e5_large_instruct_v1`) | Prefix-based (`"query: "` / `"passage: "`) | `text_extractor`, `multimodal_extractor` transcription |
| Gemini Embedding 2 | Instruction-based | `universal_extractor` |
| Vertex Multimodal | Not task-aware (ignored) | `multimodal_extractor` visual |
| SigLIP / CLIP | Not task-aware (ignored) | `image_extractor` |
## Related Stages
* [Attribute Filter](/docs/retrieval/stages/attribute-filter) - Metadata-based filtering
* [Rerank](/docs/retrieval/stages/rerank) - Neural re-ranking
* [Query Expand](/docs/retrieval/stages/query-expand) - Query expansion before search
* [MMR](/docs/retrieval/stages/mmr) - Diversity-optimized selection
# Group By
Source: https://docs.mixpeek.com/docs/retrieval/stages/group-by
Aggregate documents by shared field values into logical groups
<Frame>
<img alt="Group By stage showing document aggregation by field values" />
</Frame>
The Group By stage aggregates documents that share the same value for a specified field, creating logical groups. This is useful for organizing results by category, author, date, or any other attribute.
<Note>
**Stage Category**: GROUP (Groups documents)
**Transformation**: N documents → G groups (where G = unique field values)
</Note>
## When to Use
| Use Case | Description |
| ----------------------- | -------------------------- |
| **Category grouping** | Group products by category |
| **Author aggregation** | Group articles by author |
| **Date grouping** | Group by day/month/year |
| **Source organization** | Group by data source |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------------- | ----------------------- |
| Semantic similarity grouping | `cluster` |
| Statistical aggregations only | `aggregate` |
| Removing duplicates | `deduplicate` |
| Top-N per group | Use with `sample` |
## Parameters
| Parameter | Type | Default | Description |
| ---------------- | ------- | ------------------ | ------------------------------------------------------------------- |
| `group_by_field` | string | `source_object_id` | Field to group by (dot notation supported) |
| `max_per_group` | integer | `10` | Maximum documents to keep per group |
| `output_mode` | string | `all` | `first` (top doc per group), `all` (grouped), `flatten` (flat list) |
<Warning>
Use `group_by_field`. This stage ignores unknown keys, so a wrong key name
returns HTTP 200 and wrong groups.
Send `field` here and the stage groups by the `source_object_id` default
instead. `field` belongs to a different model: the `group_by` object nested
inside `feature_search` parameters, which requires it.
</Warning>
## Configuration Examples
<CodeGroup>
```json Basic Group By theme={null}
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.category"
}
}
}
```
```json Limited Docs Per Group theme={null}
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.author",
"max_per_group": 5
}
}
}
```
```json Deduplicate (top doc per group) theme={null}
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.brand",
"max_per_group": 1,
"output_mode": "first"
}
}
}
```
```json Date Grouping theme={null}
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.publish_date"
}
}
}
```
```json Nested Field Grouping theme={null}
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.source.type",
"max_per_group": 3
}
}
}
```
</CodeGroup>
## Output Schema
```json theme={null}
{
"groups": [
{
"key": "electronics",
"count": 25,
"documents": [
{
"document_id": "doc_123",
"content": "Latest smartphone review...",
"score": 0.95,
"metadata": {"category": "electronics", "price": 999}
},
{
"document_id": "doc_456",
"content": "Laptop comparison guide...",
"score": 0.89,
"metadata": {"category": "electronics", "price": 1299}
}
]
},
{
"key": "clothing",
"count": 18,
"documents": [...]
}
],
"metadata": {
"total_groups": 5,
"total_documents": 100,
"field": "metadata.category"
}
}
```
## Performance
| Metric | Value |
| --------------- | --------- |
| **Latency** | 5-20ms |
| **Memory** | O(N) |
| **Cost** | Free |
| **Scalability** | Efficient |
## Common Pipeline Patterns
### Search + Group by Category
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.category",
"max_per_group": 5
}
}
}
]
```
### Grouped Results with Aggregations
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 }
],
"final_top_k": 200
}
}
},
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.brand",
"max_per_group": 10
}
}
},
{
"stage_name": "aggregate",
"stage_type": "reduce",
"config": {
"stage_id": "aggregate",
"parameters": {
"aggregations": [
{"type": "avg", "field": "metadata.price", "name": "avg_price"},
{"type": "avg", "field": "metadata.rating", "name": "avg_rating"}
],
"group_by": "metadata.brand"
}
}
}
]
```
### Author-Grouped Search
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "authors",
"target_field": "author_id",
"source_field": "metadata.author_id",
"output_field": "author"
}
}
},
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "author.name",
"max_per_group": 3
}
}
}
]
```
### Time-Based Grouping
```json theme={null}
[
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.date",
"operator": "gte",
"value": "2024-01-01"
}
}
}
},
{
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"code": "def transform(doc):\n date = doc.get('metadata', {}).get('date', '')\n doc['metadata']['month'] = date[:7] # YYYY-MM\n return doc"
}
}
},
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.month"
}
}
}
]
```
## Document Sorting Within Groups
Documents within each group are automatically sorted by relevance `score` (highest first), then limited to `max_per_group`.
## Output Modes
| `output_mode` | Description |
| --------------- | -------------------------------------------------------------- |
| `all` (default) | Return all documents (up to `max_per_group`) grouped by field |
| `first` | Return only the top-scoring document per group (deduplication) |
| `flatten` | Return all documents as a flat list (drops group structure) |
## Handling Missing Values
Documents missing the `group_by_field` value are grouped under a `null` key.
## Error Handling
| Error | Behavior |
| ------------------ | ---------------------------------- |
| Missing field | Documents grouped under "null" key |
| Empty results | Return empty groups array |
| Invalid field path | Stage fails |
## Group By vs Cluster
| Aspect | Group By | Cluster |
| -------------- | --------------------- | -------------------- |
| Grouping basis | Field value | Embedding similarity |
| Groups known | Yes (field values) | No (discovered) |
| Speed | Fast | Slower |
| Use case | Category organization | Theme discovery |
## Related
* [Aggregate](/docs/retrieval/stages/aggregate) - Statistical aggregations
* [Cluster](/docs/retrieval/stages/cluster) - Semantic grouping
* [Sample](/docs/retrieval/stages/sample) - Select from groups
* [Sort Attribute](/docs/retrieval/stages/sort-attribute) - Simple sorting
# JSON Transform
Source: https://docs.mixpeek.com/docs/retrieval/stages/json-transform
Transform document structure using Jinja2 templates for API payloads or custom schemas
<Frame>
<img alt="JSON Transform stage showing Jinja2 template transformation of documents" />
</Frame>
The JSON Transform stage applies a Jinja2 template to each document, rendering the template with full document context and replacing the document with the parsed JSON output. Use this to reformat documents for external APIs or reshape data for downstream consumers.
<Note>
**Stage Category**: APPLY (1-1 Transformation)
**Transformation**: N documents → N documents (or fewer with `fail_on_error=False`)
</Note>
## When to Use
| Use Case | Description |
| --------------------------- | ------------------------------------------------- |
| **External API formatting** | Format documents for webhook payloads |
| **Response optimization** | Remove unused fields to reduce bandwidth |
| **Schema adaptation** | Convert internal format to client-specific format |
| **Conditional outputs** | Include fields based on document properties |
| **Array flattening** | Transform nested structures to flat arrays |
| **Field renaming** | Rename or reorganize document fields |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ---------------------------------- |
| Filtering documents | `attribute_filter` or `llm_filter` |
| Sorting documents | `sort_by_field` or `rerank` |
| Enriching with new data | `document_enrich` or `api_call` |
| Joining external data | `taxonomy_enrich` |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------- | ---------- | ---------------------------------------------- |
| `template` | string | *Required* | Jinja2 template that must render to valid JSON |
| `fail_on_error` | boolean | `false` | Fail entire pipeline on transformation error |
## Template Context
Templates have access to the full retriever execution context:
| Namespace | Description | Example |
| --------------------- | ----------------------------------------- | ---------------------------- |
| `DOC` / `doc` | Current document fields and metadata | `{{ DOC.document_id }}` |
| `INPUT` / `inputs` | Original query inputs from search request | `{{ INPUT.query }}` |
| `CONTEXT` / `context` | Execution context (namespace\_id, etc.) | `{{ CONTEXT.namespace_id }}` |
| `STAGE` / `stage` | Current stage execution data | `{{ STAGE.name }}` |
| `SECRET` / `secret` | Vault secrets (API keys, credentials) | `{{ SECRET.api_key }}` |
<Note>
Both uppercase and lowercase namespace formats work identically (`DOC` == `doc`).
</Note>
### Built-in Functions
These numeric helpers are available directly in templates — useful for clamping
and rounding computed values (e.g. ranking scores):
| Function | Description | Example |
| ---------------- | ----------------------------------- | --------------------------------- |
| `max` / `min` | Largest / smallest of the arguments | `{{ max(0, 1 - days_old / 14) }}` |
| `abs` | Absolute value | `{{ abs(DOC.delta) }}` |
| `round` | Round to nearest (optional digits) | `{{ round(score, 4) }}` |
| `ceil` / `floor` | Round up / down to an integer | `{{ ceil(DOC.count / 10) }}` |
Read tunable inputs with a default using `INPUT.get`, so callers can override a
weight per request but omit it otherwise: `{{ INPUT.get('recency_weight', 0.3) }}`.
## Template Features
### Jinja2 Syntax
| Feature | Syntax | Description |
| ------------ | ------------------------- | ------------------- |
| Variables | `{{ DOC.field }}` | Output field values |
| Conditionals | `{% if %}...{% endif %}` | Conditional content |
| Loops | `{% for item in items %}` | Iterate over arrays |
| Filters | `{{ value \| tojson }}` | Transform values |
| Comments | `{# comment #}` | Template comments |
### Useful Filters
| Filter | Description | Example |
| ---------------- | ----------------------- | -------------------------------------- |
| `tojson` | JSON-safe encoding | `{{ DOC.data \| tojson }}` |
| `length` | Get array/string length | `{{ DOC.tags \| length }}` |
| `default` | Fallback value | `{{ DOC.optional \| default('N/A') }}` |
| `first` / `last` | Array element | `{{ DOC.items \| first }}` |
| `join` | Join array | `{{ DOC.tags \| join(', ') }}` |
## Configuration Examples
<CodeGroup>
```json Simple Field Selection theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"id\": \"{{ DOC.document_id }}\", \"content\": \"{{ DOC.text }}\", \"score\": {{ DOC.score }}}"
}
}
}
```
```json With JSON Escaping theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"id\": \"{{ DOC.document_id }}\", \"content\": {{ DOC.text | tojson }}, \"metadata\": {{ DOC.metadata | tojson }}}"
}
}
}
```
```json Conditional Field Inclusion theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"workflow_name\": \"process-asset\", \"inputs\": [{\"name\": \"id\", \"value\": \"{{ DOC.id }}\"}{% if DOC.asset_type == \"VIDEO\" %}, {\"name\": \"video\", \"value\": {\"src\": \"{{ DOC.url }}\"}}{% endif %}]}"
}
}
}
```
```json Array Iteration theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"title\": \"{{ DOC.title }}\", \"tags\": [{% for tag in DOC.tags %}\"{{ tag }}\"{% if not loop.last %}, {% endif %}{% endfor %}]}"
}
}
}
```
```json Nested Field Access theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"user_id\": \"{{ DOC.metadata.user_id }}\", \"category\": \"{{ DOC.metadata.category }}\", \"raw_data\": {{ DOC.metadata.raw | tojson }}}"
}
}
}
```
```json Strict Mode (Fail on Error) theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"required_field\": \"{{ DOC.must_exist }}\", \"value\": {{ DOC.number }}}",
"fail_on_error": true
}
}
}
```
```json External Workflow API Format theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"workflow\": \"{{ DOC.workflow_name }}\", \"inputs\": [{\"name\": \"variant_id\", \"value\": \"{{ DOC.variant_id }}\"}{% if DOC.asset_type == \"VIDEO\" %}, {\"name\": \"video\", \"value\": {\"src\": \"{{ DOC.asset_url }}\"}}{% endif %}]}"
}
}
}
```
</CodeGroup>
## Error Handling
| Setting | Behavior |
| -------------------------------- | ------------------------------------------------------- |
| `fail_on_error: false` (default) | Skip failed documents with warning, continue processing |
| `fail_on_error: true` | Fail entire retrieval on first transformation error |
**Common failure causes:**
* Invalid template syntax
* Template rendering errors (missing fields)
* Invalid JSON output from template
* Document missing required fields
<Tip>
Use `fail_on_error: false` for public APIs where partial results are acceptable. Use `fail_on_error: true` for internal workflows where data integrity is critical.
</Tip>
## Performance
| Metric | Value |
| -------------- | ------------------------------------- |
| **Latency** | \< 1ms per document |
| **Processing** | Sequential (fast, no caching needed) |
| **Schema** | Output completely defined by template |
## Multi-line Templates
For complex templates, use HEREDOC syntax in the API call:
```bash theme={null}
curl -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-d '{
"stages": [{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\n \"id\": \"{{ DOC.document_id }}\",\n \"title\": {{ DOC.title | tojson }},\n \"items\": [\n {% for item in DOC.items %}{\n \"name\": \"{{ item.name }}\",\n \"value\": {{ item.value }}\n }{% if not loop.last %},{% endif %}\n {% endfor %}\n ]\n}"
}
}
}]
}'
```
## Common Patterns
### Drop Unused Fields
```json theme={null}
{
"template": "{\"id\": \"{{ DOC.document_id }}\", \"title\": \"{{ DOC.title }}\", \"url\": \"{{ DOC.url }}\"}"
}
```
### Flatten Nested Metadata
```json theme={null}
{
"template": "{\"doc_id\": \"{{ DOC.document_id }}\", \"user_id\": \"{{ DOC.metadata.user_id }}\", \"category\": \"{{ DOC.metadata.category }}\", \"score\": {{ DOC.score }}}"
}
```
### Add Query Context
```json theme={null}
{
"template": "{\"query\": \"{{ INPUT.query }}\", \"result_id\": \"{{ DOC.document_id }}\", \"score\": {{ DOC.score }}}"
}
```
### Compute a custom ranking signal (recency boost + used-demotion)
`json_transform` is the general-purpose way to compute a **custom ranking score**
at query time: read the relevance score and any document fields, blend in your own
signals, emit the result as a `score`, then order by it with
[`sort_relevance`](/docs/retrieval/stages/sort-relevance). Because every weight is read
from `INPUT`, the same retriever is tunable per request — including an instant
off-switch — with no redeploy.
The pipeline is three stages:
```
feature_search → json_transform → sort_relevance
(relevance) (blend the score) (order by "score")
```
A worked example: boost fresh documents, and demote any document that already
carries a `used_in_ad` edge. The blended score is
```
score = relevance × (1 + recency_weight × recency_factor) × (1 − used_demotion if used)
```
where `recency_factor` is a gentle linear decay `max(0, 1 − days_old / freshness_days)`.
The template reads each weight from `INPUT` with a default, so callers override only
what they want. This is the exact template that runs in production, prettified for
readability (the verbatim single-line version is in the block below):
```jinja theme={null}
{% set md = DOC['_internal']['metadata'] %}
{% set d = (md.get('date_created') or '1970-01-01')[:10] %}
{# pseudo-ordinal day count: Y*372 + M*31 + D — monotonic, months≈31d, ±1d vs true days #}
{% set ord_doc = (d[:4]|int)*372 + (d[5:7]|int)*31 + (d[8:10]|int) %}
{% set asof = INPUT.get('as_of') or '2026-07-15' %}
{% set ord_asof = (asof[:4]|int)*372 + (asof[5:7]|int)*31 + (asof[8:10]|int) %}
{% set days_old = ord_asof - ord_doc %}
{# tunables — `x if x is not none else default` also catches a client-passed null #}
{% set fw = (INPUT.get('freshness_days') or 14) | int %}
{% set rw = (INPUT.get('recency_weight') if INPUT.get('recency_weight') is not none else 0.3) | float %}
{% set ud = (INPUT.get('used_demotion') if INPUT.get('used_demotion') is not none else 0.4) | float %}
{% set on = INPUT.get('boost_enabled') if INPUT.get('boost_enabled') is not none else true %}
{# edge-driven, not a flag: does this doc carry a used_in_ad edge? #}
{% set used = ((DOC.get('edges') or []) | selectattr('type', 'equalto', 'used_in_ad') | list | length) > 0 %}
{% set rf = max(0.0, 1.0 - (days_old / fw)) %}
{% set base = DOC.get('score') or 0 %}
{% set final = (base * (1 + rw * rf) * (1 - (ud if used else 0))) if on else base %}
{
"document_id": {{ DOC.document_id | tojson }},
"title": {{ md.get('title') | tojson }},
"brand": {{ md.get('brand') | tojson }},
"job_id": {{ md.get('job_id') | tojson }},
"date_created": {{ d | tojson }},
"days_old": {{ days_old }},
"base_score": {{ base | round(4) }},
"recency_factor": {{ rf | round(4) }},
"is_used_in_ad": {{ used | tojson }},
"new_badge": {{ (days_old <= fw) | tojson }},
"score": {{ final | round(4) }}
}
```
A few things worth noticing in the real template:
* **User metadata is read from `_internal.metadata`** here (`md`), the path managed-pipeline
documents use. If your documents carry metadata at the top level or under `metadata`
instead, adjust the path to match how they were ingested.
* **The default idiom guards `null`, not just missing.** `INPUT.get('recency_weight') if
INPUT.get('recency_weight') is not none else 0.3` falls back both when the caller omits
the input **and** when it passes an explicit `null` — `INPUT.get('x', 0.3)` alone would
keep a client-sent `null`.
* **`days_old` is a pseudo-ordinal** (`Y*372 + M*31 + D`), monotonic and accurate to about
±1 day versus true calendar days — deliberately cheap, since exact date math isn't needed
to rank by freshness.
<Accordion title="Verbatim single-line template (paste as the template string)">
The production template is authored on one line (newlines are optional in Jinja).
This is byte-for-byte what ran in prod:
```jinja theme={null}
{% set md = DOC['_internal']['metadata'] %}{% set d = (md.get('date_created') or '1970-01-01')[:10] %}{% set ord_doc = (d[:4]|int)*372 + (d[5:7]|int)*31 + (d[8:10]|int) %}{% set asof = INPUT.get('as_of') or '2026-07-15' %}{% set ord_asof = (asof[:4]|int)*372 + (asof[5:7]|int)*31 + (asof[8:10]|int) %}{% set days_old = ord_asof - ord_doc %}{% set fw = (INPUT.get('freshness_days') or 14) | int %}{% set rw = (INPUT.get('recency_weight') if INPUT.get('recency_weight') is not none else 0.3) | float %}{% set ud = (INPUT.get('used_demotion') if INPUT.get('used_demotion') is not none else 0.4) | float %}{% set on = INPUT.get('boost_enabled') if INPUT.get('boost_enabled') is not none else true %}{% set used = ((DOC.get('edges') or []) | selectattr('type','equalto','used_in_ad') | list | length) > 0 %}{% set rf = max(0.0, 1.0 - (days_old / fw)) %}{% set base = DOC.get('score') or 0 %}{% set final = (base * (1 + rw * rf) * (1 - (ud if used else 0))) if on else base %}{"document_id": {{ DOC.document_id | tojson }}, "title": {{ md.get('title') | tojson }}, "brand": {{ md.get('brand') | tojson }}, "job_id": {{ md.get('job_id') | tojson }}, "date_created": {{ d | tojson }}, "days_old": {{ days_old }}, "base_score": {{ base | round(4) }}, "recency_factor": {{ rf | round(4) }}, "is_used_in_ad": {{ used | tojson }}, "new_badge": {{ (days_old <= fw) | tojson }}, "score": {{ final | round(4) }}}
```
</Accordion>
Then sort by the computed field (it is the default, shown here for clarity):
```json theme={null}
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": { "score_field": "score", "direction": "desc" }
}
}
```
<Note>
`json_transform` **replaces** each document with the template's JSON output, so
emit every field you want downstream — including the `score` that `sort_relevance`
reads. Pass the template to the stage as its `template` string (escape it as JSON,
or use the heredoc form shown under [Multi-line Templates](#multi-line-templates)).
</Note>
Tunable inputs (declare them in the retriever's `input_schema` as optional):
| Input | Default | Effect |
| ---------------- | ------- | ------------------------------------------------------------------------------------------- |
| `recency_weight` | `0.3` | How strongly freshness lifts the score. Raise to let recent docs dominate. |
| `used_demotion` | `0.4` | Fraction to subtract when the doc has a `used_in_ad` edge. |
| `freshness_days` | `14` | Decay window, and the cutoff for the `new_badge` flag. |
| `as_of` | today | Reference date the age is measured from. |
| `boost_enabled` | `true` | Set `false` to bypass all boosting and return pure relevance order — an instant off-switch. |
```bash theme={null}
# default ranking
curl -X POST "$MP_API_URL/v1/retrievers/$RET_ID/execute" -H "Authorization: Bearer $MP_API_KEY" \
-d '{"inputs": {"query": "footage scene"}}'
# heavier recency, wider window
curl ... -d '{"inputs": {"query": "footage scene", "recency_weight": 2.0, "freshness_days": 30}}'
# off-switch — pure relevance
curl ... -d '{"inputs": {"query": "footage scene", "boost_enabled": false}}'
```
## Related
* [Sort Relevance](/docs/retrieval/stages/sort-relevance) - Order by a score, including one computed here
* [RAG Prepare](/docs/retrieval/stages/rag-prepare) - Format documents for LLM context
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - Extract structured data with LLMs
* [API Call](/docs/retrieval/stages/api-call) - Format for external API calls
# Limit
Source: https://docs.mixpeek.com/docs/retrieval/stages/limit
Truncate results to a maximum count with optional offset for pagination
<Frame>
<img alt="Limit stage showing result truncation to top-N documents" />
</Frame>
The Limit stage truncates the document set to a maximum number of results, optionally with an offset for pagination-style behavior. This is the retriever pipeline equivalent of SQL's `LIMIT/OFFSET` clause.
<Note>
**Stage Category**: REDUCE (Truncates documents)
**Transformation**: N documents → min(N, limit) documents
</Note>
## When to Use
| Use Case | Description |
| --------------------- | ---------------------------------------------------- |
| **Top-K results** | Return only the best N results after reranking |
| **Pagination** | Implement page-based result access with offset |
| **Cost control** | Cap document count before expensive LLM stages |
| **Fixed output** | Guarantee exactly N results for downstream consumers |
| **Mid-pipeline trim** | Reduce candidates between expensive stages |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ---------------------------------------- |
| Random sampling | `sample` stage |
| Filtering by criteria | `attribute_filter` or `llm_filter` |
| Initial retrieval limit | Set `limit` in `feature_search` directly |
| Statistical reduction | `aggregate` stage |
| Grouping results | `group_by` stage |
## Parameters
| Parameter | Type | Default | Description |
| --------- | ------- | ------- | -------------------------------------------------------- |
| `limit` | integer | `10` | Maximum number of documents to return (1-10000) |
| `offset` | integer | `0` | Number of documents to skip from the beginning (0-10000) |
## Configuration Examples
<CodeGroup>
```json Top 10 Results theme={null}
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 10
}
}
}
```
```json Pagination (Page 3) theme={null}
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 10,
"offset": 20
}
}
}
```
```json Single Best Result theme={null}
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 1
}
}
}
```
```json Cap Before LLM Processing theme={null}
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 25
}
}
}
```
</CodeGroup>
<Tip>
Place the limit stage after sorting/reranking to ensure you're keeping the highest-quality results. Limiting before reranking loses potentially relevant documents.
</Tip>
## Performance
| Metric | Value |
| -------------- | ----------------- |
| **Latency** | \< 1ms |
| **Memory** | O(1) |
| **Cost** | Free |
| **Complexity** | O(1) list slicing |
## Common Pipeline Patterns
### Rerank Then Limit
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 100}],
"final_top_k": 100
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "content"
}
}
},
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 10
}
}
}
]
```
### Cost-Controlled LLM Pipeline
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 200}],
"final_top_k": 200
}
}
},
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 20
}
}
},
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Summarize: {{DOC.content}}",
"output_field": "summary"
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ---------------------- | ------------------------------------ |
| Limit > input count | Returns all available documents |
| Offset > input count | Returns empty result set |
| Empty input | Returns empty result set |
| Offset + Limit > count | Returns documents from offset to end |
## Related
* [Sample](/docs/retrieval/stages/sample) - Random or stratified sampling
* [Deduplicate](/docs/retrieval/stages/deduplicate) - Remove duplicates before limiting
* [Rerank](/docs/retrieval/stages/rerank) - Re-score before limiting to ensure best results
# LLM Enrich
Source: https://docs.mixpeek.com/docs/retrieval/stages/llm-enrich
Extract structured data from documents using language model analysis
<Frame>
<img alt="LLM Enrich stage showing structured data extraction with language models" />
</Frame>
The LLM Enrich stage uses language models to extract structured data from document content. It can identify entities, classify content, extract key information, and generate structured outputs.
<Note>
**Stage Category**: ENRICH (Enriches documents)
**Transformation**: N documents → N documents (with extracted data added)
</Note>
## When to Use
| Use Case | Description |
| -------------------------------- | ------------------------------------------ |
| **Entity extraction** | Extract names, dates, amounts from text |
| **Content classification** | Categorize documents by topic/type |
| **Key information extraction** | Pull specific facts from unstructured text |
| **Structured output generation** | Convert prose to structured data |
## When NOT to Use
| Scenario | Recommended Alternative |
| ---------------------------------- | --------------------------- |
| Simple field transformation | `json_transform` |
| Predefined taxonomy classification | `taxonomy_enrich` |
| Large-scale processing | Pre-process during indexing |
| Real-time low-latency | Use cached extractions |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------- | ---------------- | --------------------------------------------------------------------------------------------- |
| `provider` | string | `google` | LLM provider: `openai`, `google`, or `anthropic` (auto-inferred from `model_name` if omitted) |
| `model_name` | string | provider default | Specific model, e.g. `gpt-4o-mini`, `gemini-2.5-flash-lite`, `claude-3-5-haiku` |
| `prompt` | string | *Required* | Extraction instructions (template — supports `{{DOC.field}}` / `{{INPUT.field}}`) |
| `content_field` | string | `content` | Field to analyze |
| `output_field` | string | `extracted` | Field for extracted data |
| `output_schema` | object | `null` | JSON schema for structured output |
| `batch_size` | integer | `5` | Documents per LLM call |
## Available Models
| Model | Speed | Quality | Best For |
| ----------------- | ------ | --------- | ------------------ |
| `gpt-4o-mini` | Fast | Good | Simple extractions |
| `gpt-4o` | Medium | Excellent | Complex analysis |
| `claude-3-haiku` | Fast | Good | Quick processing |
| `claude-3-sonnet` | Medium | Excellent | Nuanced extraction |
## Configuration Examples
<CodeGroup>
```json Basic Entity Extraction theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Extract all company names and person names mentioned in this document.",
"output_field": "entities"
}
}
}
```
```json Structured Output with Schema theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Extract the following information from this product review.",
"output_field": "review_analysis",
"output_schema": {
"type": "object",
"properties": {
"sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
"rating_mentioned": {"type": "number", "minimum": 1, "maximum": 5},
"pros": {"type": "array", "items": {"type": "string"}},
"cons": {"type": "array", "items": {"type": "string"}},
"would_recommend": {"type": "boolean"}
}
}
}
}
}
```
```json Key Facts Extraction theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Extract key facts from this news article: main event, date, location, people involved, and outcome.",
"output_field": "key_facts",
"output_schema": {
"type": "object",
"properties": {
"main_event": {"type": "string"},
"date": {"type": "string"},
"location": {"type": "string"},
"people": {"type": "array", "items": {"type": "string"}},
"outcome": {"type": "string"}
}
}
}
}
}
```
```json Topic Classification theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "anthropic",
"model_name": "claude-3-haiku",
"prompt": "Classify this document into primary and secondary topics. Be specific.",
"output_field": "topics",
"output_schema": {
"type": "object",
"properties": {
"primary_topic": {"type": "string"},
"secondary_topics": {"type": "array", "items": {"type": "string"}},
"confidence": {"type": "number", "minimum": 0, "maximum": 1}
}
}
}
}
}
```
```json Contact Information Extraction theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Extract all contact information (emails, phone numbers, addresses) from this document.",
"output_field": "contacts",
"output_schema": {
"type": "object",
"properties": {
"emails": {"type": "array", "items": {"type": "string", "format": "email"}},
"phones": {"type": "array", "items": {"type": "string"}},
"addresses": {"type": "array", "items": {"type": "string"}}
}
}
}
}
}
```
</CodeGroup>
## Output Schema
Define structured output using JSON Schema:
```json theme={null}
{
"output_schema": {
"type": "object",
"properties": {
"field_name": {"type": "string"},
"numeric_field": {"type": "number"},
"boolean_field": {"type": "boolean"},
"array_field": {"type": "array", "items": {"type": "string"}},
"enum_field": {"type": "string", "enum": ["option1", "option2", "option3"]}
},
"required": ["field_name"]
}
}
```
### Supported Types
| Type | Description |
| --------- | -------------- |
| `string` | Text values |
| `number` | Numeric values |
| `boolean` | True/false |
| `array` | Lists of items |
| `object` | Nested objects |
## Output Examples
### Without Schema
```json theme={null}
{
"document_id": "doc_123",
"content": "Apple Inc. announced...",
"entities": "Companies: Apple Inc., Microsoft\nPeople: Tim Cook, Satya Nadella"
}
```
### With Schema
```json theme={null}
{
"document_id": "doc_123",
"content": "Great product, 5 stars!...",
"review_analysis": {
"sentiment": "positive",
"rating_mentioned": 5,
"pros": ["easy to use", "great value", "fast shipping"],
"cons": ["packaging could be better"],
"would_recommend": true
}
}
```
## Performance
| Metric | Value |
| --------------- | ------------------------------ |
| **Latency** | 300-800ms per batch |
| **Batch size** | 5 documents default |
| **Token usage** | \~200 tokens per document |
| **Parallel** | Batches processed concurrently |
<Warning>
LLM enrichment is expensive. Consider pre-computing extractions during indexing for frequently accessed data.
</Warning>
## Common Pipeline Patterns
### Search + Extract + Filter
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 20
}
],
"final_top_k": 20
}
}
},
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Extract the main topic and sentiment.",
"output_field": "analysis",
"output_schema": {
"type": "object",
"properties": {
"topic": {"type": "string"},
"sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]}
}
}
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "analysis.sentiment",
"operator": "eq",
"value": "positive"
}
}
}
]
```
### Entity Extraction Pipeline
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 10
}
],
"final_top_k": 10
}
}
},
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Extract all entities with their types and relationships.",
"output_field": "entities",
"output_schema": {
"type": "object",
"properties": {
"people": {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string"}, "role": {"type": "string"}}}},
"organizations": {"type": "array", "items": {"type": "string"}},
"locations": {"type": "array", "items": {"type": "string"}},
"dates": {"type": "array", "items": {"type": "string"}}
}
}
}
}
}
]
```
## Writing Effective Prompts
### Good Prompts
```
✓ "Extract the product name, price, and key features from this product listing."
✓ "Identify all dates mentioned and their associated events."
✓ "Classify the sentiment as positive, neutral, or negative, and explain why."
```
### Poor Prompts
```
✗ "Analyze this document" (too vague)
✗ "Get the data" (not specific)
✗ "Tell me about it" (unclear output)
```
<Tip>
Be specific about what to extract and in what format. When using `output_schema`, the LLM will conform to the structure.
</Tip>
## Multimodal Query Inputs
Pass images from your query inputs to the LLM alongside document content. This enables visual comparison, brand matching, and cross-modal analysis.
### Parameters
| Parameter | Type | Default | Description |
| ------------------- | ------ | ------- | -------------------------------------------------------------- |
| `multimodal_inputs` | object | `null` | Map of INPUT field names to media types (`"image"`, `"video"`) |
### How It Works
1. Declare which INPUT fields carry multimodal content via `multimodal_inputs`
2. At runtime, the stage extracts URLs from `{{INPUT.field_name}}`
3. Images are sent to the LLM alongside the prompt and document content
4. Works with providers that support vision (Google Gemini, OpenAI GPT-4o, Anthropic Claude)
### Configuration Examples
<CodeGroup>
```json Brand Logo Comparison theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Compare this brand logo to the document. Rate alignment 1-10: {{DOC.brand_name}}",
"output_field": "brand_alignment",
"multimodal_inputs": {
"query_image": "image"
}
}
}
}
```
```json Visual Product Match theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Does this product image match the description? {{DOC.description}}",
"output_field": "visual_match",
"output_schema": {
"type": "object",
"properties": {
"matches": {"type": "boolean"},
"confidence": {"type": "number"},
"differences": {"type": "array", "items": {"type": "string"}}
}
},
"multimodal_inputs": {
"product_photo": "image"
}
}
}
}
```
```json Multi-Image Reference theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Compare the reference image against this document's content: {{DOC.title}}",
"output_field": "comparison",
"multimodal_inputs": {
"reference_image": "image",
"style_guide": "image"
}
}
}
}
```
</CodeGroup>
### Calling with Multimodal Inputs
When using `multimodal_inputs`, pass the image URLs in the retriever's `inputs`:
```json theme={null}
{
"inputs": {
"query": "Find products similar to this",
"query_image": "https://storage.example.com/brand-logo.png"
},
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": { "..." : "..." }
}
},
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"prompt": "Compare this image to: {{DOC.product_name}}",
"output_field": "match_score",
"multimodal_inputs": {"query_image": "image"}
}
}
}
]
}
```
<Note>
If a declared multimodal input field is missing from the query inputs at runtime, the stage proceeds without it (text-only mode). No error is raised.
</Note>
***
## Bring Your Own Key (BYOK)
Use your own LLM API keys instead of Mixpeek's default keys. This gives you control over costs, rate limits, and API usage.
### Why Use BYOK?
| Benefit | Description |
| ---------------- | --------------------------------------------- |
| **Cost Control** | Use your own LLM provider account and billing |
| **Rate Limits** | Use your own rate limits instead of shared |
| **Compliance** | Keep API calls under your own account |
| **Key Rotation** | Rotate keys without changing retrievers |
### Setup
<Steps>
<Step title="Store your API key as a secret">
Store your LLM provider API key in the organization secrets vault:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "openai_api_key",
"secret_value": "sk-proj-abc123..."
}'
```
</Step>
<Step title="Reference the secret in your stage">
Use the `api_key` parameter with template syntax:
```json theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Extract key entities from this document.",
"output_field": "entities",
"api_key": "{{secrets.openai_api_key}}"
}
}
}
```
</Step>
</Steps>
### BYOK Configuration Example
<CodeGroup>
```json OpenAI BYOK theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Summarize this document in 2-3 sentences.",
"output_field": "summary",
"api_key": "{{secrets.openai_api_key}}"
}
}
}
```
```json Anthropic BYOK theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "anthropic",
"provider": "anthropic",
"model_name": "claude-3-haiku-20240307",
"prompt": "Extract the main topics from this content.",
"output_field": "topics",
"api_key": "{{secrets.anthropic_api_key}}"
}
}
}
```
```json Google BYOK theme={null}
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "google",
"provider": "google",
"model_name": "gemini-3.1-flash-lite",
"prompt": "Classify the sentiment of this text.",
"output_field": "sentiment",
"api_key": "{{secrets.google_api_key}}"
}
}
}
```
</CodeGroup>
### Supported Providers
| Provider | Secret Name Example | Models |
| --------- | ------------------- | ---------------------------------------------- |
| OpenAI | `openai_api_key` | gpt-4o, gpt-4o-mini |
| Anthropic | `anthropic_api_key` | claude-3-haiku, claude-3-sonnet, claude-3-opus |
| Google | `google_api_key` | gemini-3.1-flash-lite, gemini-2.5-pro |
<Note>
When `api_key` is not specified, the stage uses Mixpeek's default API keys and usage is charged to your Mixpeek account.
</Note>
## Custom Enrichment Model (BYO Plugin)
Use your own enrichment model deployed as a [custom extractor](/docs/processing/custom-extractors) instead of a hosted LLM provider. Set `feature_uri` to route enrichment through your extractor's inference endpoint.
### Parameters
| Parameter | Type | Default | Description |
| ------------- | ------ | ------- | --------------------------------------------------------------------------------- |
| `feature_uri` | string | `null` | Feature URI of a custom enrichment plugin. Overrides `provider`/`model` when set. |
Your plugin must accept `{prompt: str, document: dict}` and return `{text: str}`.
### Configuration Example
```json theme={null}
{
"stage_name": "my_enricher",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"feature_uri": "mixpeek://my_summarizer@1.0.0/summarize",
"prompt": "Summarize: {{DOC.content}}",
"output_field": "summary"
}
}
}
```
<Tip>
Set `inference_type: "generate"` in your plugin's manifest to declare compatibility with LLM stages.
</Tip>
## Error Handling
| Error | Behavior |
| ---------------------- | -------------------------------- |
| LLM timeout | Retry once, then null result |
| Schema validation fail | Raw text in output\_field |
| Rate limit | Automatic backoff |
| Empty content | Skip enrichment |
| Invalid API key | Error returned with auth failure |
## Related
* [LLM Filter](/docs/retrieval/stages/llm-filter) - Filter using LLM evaluation
* [Taxonomy Enrich](/docs/retrieval/stages/taxonomy-enrich) - Predefined classification
* [JSON Transform](/docs/retrieval/stages/json-transform) - Template-based transformation
# LLM Filter
Source: https://docs.mixpeek.com/docs/retrieval/stages/llm-filter
Filter documents using LLM-based content evaluation and criteria matching
<Frame>
<img alt="LLM Filter stage showing content-based filtering with language models" />
</Frame>
The LLM Filter stage uses language models to evaluate document content against specified criteria, filtering based on semantic understanding rather than metadata fields.
<Note>
**Stage Category**: FILTER (Reduces document set)
**Transformation**: N documents → M documents (where M ≤ N, based on LLM evaluation)
</Note>
## When to Use
| Use Case | Description |
| ----------------------------- | ---------------------------------------- |
| **Content quality filtering** | Remove low-quality or irrelevant content |
| **Semantic criteria** | Filter by meaning, not just keywords |
| **Complex requirements** | "Only technical documentation" |
| **Subjective evaluation** | Tone, style, or sentiment filtering |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------- | --------------------------- |
| Simple metadata filtering | `attribute_filter` (faster) |
| Large result sets (100+) | Too slow, pre-filter first |
| Deterministic rules | `attribute_filter` |
| Low latency requirements | Use metadata filters |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------- | ---------- | -------------------------------- |
| `model` | string | *Required* | LLM model to use |
| `criteria` | string | *Required* | Natural language filter criteria |
| `content_field` | string | `content` | Field to evaluate |
| `explanation` | boolean | `false` | Include filtering explanation |
| `batch_size` | integer | `10` | Documents per LLM call |
## Available Models
| Model | Speed | Quality | Cost |
| ----------------- | ------ | --------- | ------ |
| `gpt-4o-mini` | Fast | Good | Low |
| `gpt-4o` | Medium | Excellent | Medium |
| `claude-3-haiku` | Fast | Good | Low |
| `claude-3-sonnet` | Medium | Excellent | Medium |
## Configuration Examples
<CodeGroup>
```json Basic Content Filter theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"criteria": "Keep only documents that contain technical information about software development"
}
}
}
```
```json Quality Filter theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"criteria": "Filter out documents that are: incomplete, contain mostly ads/spam, or are not in English",
"explanation": true
}
}
}
```
```json Topic Relevance theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "anthropic",
"model_name": "claude-3-haiku",
"criteria": "Keep documents specifically about {{INPUT.topic}}. Exclude tangentially related or off-topic content."
}
}
}
```
```json Professional Tone Filter theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"criteria": "Include only documents with professional, formal tone suitable for business communication. Exclude casual, informal, or inappropriate content.",
"content_field": "content"
}
}
}
```
```json Factual Content Only theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"criteria": "Keep only factual, informative content. Remove: opinions without evidence, speculation, promotional content, and entertainment-focused material.",
"explanation": true
}
}
}
```
</CodeGroup>
## Writing Effective Criteria
### Good Criteria Examples
```
✓ "Keep documents about machine learning algorithms and their implementations"
✓ "Filter out content that is primarily promotional or marketing material"
✓ "Include only documents written in the last 5 years about cloud computing"
✓ "Keep technical documentation; remove blog posts and news articles"
```
### Poor Criteria Examples
```
✗ "Good documents" (too vague)
✗ "Relevant content" (not specific)
✗ "High quality" (subjective without definition)
```
<Tip>
Be specific about what to include AND exclude. The LLM makes a binary keep/discard decision for each document.
</Tip>
## Output Schema
### Without Explanation
Documents that pass the filter are returned unchanged:
```json theme={null}
{
"document_id": "doc_123",
"content": "Technical documentation about...",
"metadata": {...}
}
```
### With Explanation
```json theme={null}
{
"document_id": "doc_123",
"content": "Technical documentation about...",
"metadata": {...},
"llm_filter": {
"passed": true,
"explanation": "Document contains detailed technical information about API implementation."
}
}
```
### Filtered Out (not in results)
Documents that don't match criteria are removed from the result set.
## Performance
| Metric | Value |
| --------------- | ------------------------------ |
| **Latency** | 200-500ms per batch |
| **Batch size** | 10 documents default |
| **Token usage** | \~100 tokens per document |
| **Parallel** | Batches processed concurrently |
<Warning>
LLM filtering is expensive and slow. Always apply `attribute_filter` or use search `top_k` limits to reduce the document set before LLM filtering.
</Warning>
## Common Pipeline Patterns
### Search + Metadata Filter + LLM Filter
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.type",
"operator": "eq",
"value": "documentation"
}
}
}
},
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"criteria": "Keep only documents that provide actionable, step-by-step instructions"
}
}
},
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 5
}
}
}
]
```
### Quality + Relevance Pipeline
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 30 }
],
"final_top_k": 30
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 15
}
}
},
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"criteria": "Keep only high-quality, authoritative sources. Remove: user-generated content without verification, outdated information (pre-2020), and incomplete documents."
}
}
}
]
```
## Cost Optimization
| Strategy | Impact |
| ------------------------ | --------------------------- |
| Pre-filter with metadata | Reduce documents before LLM |
| Use cheaper models | `gpt-4o-mini` vs `gpt-4o` |
| Increase batch size | Fewer API calls |
| Limit input documents | Use `top_k` in search |
## Bring Your Own Key (BYOK)
Use your own LLM API keys instead of Mixpeek's default keys. This gives you control over costs, rate limits, and API usage.
### Why Use BYOK?
| Benefit | Description |
| ---------------- | --------------------------------------------- |
| **Cost Control** | Use your own LLM provider account and billing |
| **Rate Limits** | Use your own rate limits instead of shared |
| **Compliance** | Keep API calls under your own account |
| **Key Rotation** | Rotate keys without changing retrievers |
### Setup
<Steps>
<Step title="Store your API key as a secret">
Store your LLM provider API key in the organization secrets vault:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "openai_api_key",
"secret_value": "sk-proj-abc123..."
}'
```
</Step>
<Step title="Reference the secret in your stage">
Use the `api_key` parameter with template syntax:
```json theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"criteria": "Keep only technical documentation",
"api_key": "{{secrets.openai_api_key}}"
}
}
}
```
</Step>
</Steps>
### BYOK Configuration Example
<CodeGroup>
```json OpenAI BYOK theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "openai",
"provider": "openai",
"model_name": "gpt-4o-mini",
"criteria": "Keep only high-quality, professional content",
"api_key": "{{secrets.openai_api_key}}"
}
}
}
```
```json Anthropic BYOK theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "anthropic",
"provider": "anthropic",
"model_name": "claude-3-haiku-20240307",
"criteria": "Filter out promotional or marketing content",
"api_key": "{{secrets.anthropic_api_key}}"
}
}
}
```
```json Google BYOK theme={null}
{
"stage_name": "llm_filter",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"provider": "google",
"provider": "google",
"model_name": "gemini-3.1-flash-lite",
"criteria": "Keep only factual, informative documents",
"api_key": "{{secrets.google_api_key}}"
}
}
}
```
</CodeGroup>
### Supported Providers
| Provider | Secret Name Example | Models |
| --------- | ------------------- | ---------------------------------------------- |
| OpenAI | `openai_api_key` | gpt-4o, gpt-4o-mini |
| Anthropic | `anthropic_api_key` | claude-3-haiku, claude-3-sonnet, claude-3-opus |
| Google | `google_api_key` | gemini-3.1-flash-lite, gemini-2.5-pro |
<Note>
When `api_key` is not specified, the stage uses Mixpeek's default API keys and usage is charged to your Mixpeek account.
</Note>
## Custom Filter Model (BYO Plugin)
Use your own filter model deployed as a [custom extractor](/docs/processing/custom-extractors) instead of a hosted LLM provider. Set `feature_uri` to route filtering through your extractor's inference endpoint.
### Parameters
| Parameter | Type | Default | Description |
| ------------- | ------ | ------- | ----------------------------------------------------------------------------- |
| `feature_uri` | string | `null` | Feature URI of a custom filter plugin. Overrides `provider`/`model` when set. |
Your plugin must accept `{prompt: str, document: dict}` and return `{keep: bool, reason: str}`.
### Configuration Example
```json theme={null}
{
"stage_name": "my_filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"feature_uri": "mixpeek://my_filter_model@1.0.0/filter",
"criteria": "Keep only documents relevant to the query"
}
}
}
```
<Tip>
Set `inference_type: "generate"` in your plugin's manifest to declare compatibility with LLM stages.
</Tip>
## Error Handling
| Error | Behavior |
| --------------- | -------------------------------- |
| LLM timeout | Retry once, then fail |
| Rate limit | Automatic backoff |
| Invalid model | Stage fails |
| Empty criteria | All documents pass |
| Invalid API key | Error returned with auth failure |
## Related
* [Attribute Filter](/docs/retrieval/stages/attribute-filter) - Metadata-based filtering
* [Rerank](/docs/retrieval/stages/rerank) - Relevance-based ordering
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - Extract data with LLM
# MMR (Maximal Marginal Relevance)
Source: https://docs.mixpeek.com/docs/retrieval/stages/mmr
Diversify search results by balancing relevance with result variety
<Frame>
<img alt="MMR stage showing diversity-aware result selection" />
</Frame>
The MMR (Maximal Marginal Relevance) stage diversifies search results by iteratively selecting documents that are both relevant to the query and different from already-selected documents. This prevents redundant results and surfaces a broader range of relevant content.
<Note>
**Stage Category**: SORT (Reorders with diversity)
**Transformation**: N documents → top\_k documents (diverse selection)
</Note>
## When to Use
| Use Case | Description |
| ---------------------------- | -------------------------------------- |
| **Reduce redundancy** | Avoid showing near-duplicate results |
| **Exploration** | Surface different aspects of a topic |
| **Coverage** | Ensure results span multiple subtopics |
| **Recommendation diversity** | Show varied options |
## When NOT to Use
| Scenario | Recommended Alternative |
| ---------------------- | ----------------------- |
| Pure relevance ranking | `rerank` |
| Simple sorting | `sort_by_field` |
| Already diverse corpus | Skip MMR |
| Very small result sets | Not enough to diversify |
## Parameters
| Parameter | Type | Default | Description |
| ----------------------- | ------- | ------- | ----------------------------------------------- |
| `lambda` | float | `0.7` | Balance: 0 = max diversity, 1 = max relevance |
| `top_k` | integer | `25` | Number of results to return |
| `diversity_feature_uri` | string | *auto* | Feature URI of the embedding used for diversity |
## Lambda Parameter
The `lambda` parameter controls the relevance-diversity trade-off:
| Lambda | Behavior |
| ------ | ----------------------------------- |
| `1.0` | Pure relevance (no diversification) |
| `0.7` | Slightly favor relevance |
| `0.5` | Balanced (default) |
| `0.3` | Favor diversity |
| `0.0` | Maximum diversity |
## Configuration Examples
<CodeGroup>
```json Balanced MMR theme={null}
{
"stage_name": "mmr",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": {
"lambda": 0.5,
"top_k": 10
}
}
}
```
```json High Diversity theme={null}
{
"stage_name": "mmr",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": {
"lambda": 0.3,
"top_k": 15
}
}
}
```
```json Relevance-Focused theme={null}
{
"stage_name": "mmr",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": {
"lambda": 0.8,
"top_k": 10
}
}
}
```
```json Custom Diversity Feature theme={null}
{
"stage_name": "mmr",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": {
"lambda": 0.5,
"top_k": 20,
"diversity_feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
}
}
}
```
</CodeGroup>
## How MMR Works
The MMR algorithm iteratively selects documents using:
```
MMR = argmax[λ × Sim(doc, query) - (1-λ) × max(Sim(doc, selected_docs))]
```
1. **First Selection**: Choose the most relevant document
2. **Subsequent Selections**: Balance relevance against similarity to already-selected documents
3. **Repeat**: Until top\_k documents are selected
### Example Selection Process
| Iteration | Selected | Reason |
| --------- | ------------ | -------------------------------------- |
| 1 | Doc A (0.95) | Highest relevance |
| 2 | Doc C (0.82) | High relevance, different from A |
| 3 | Doc E (0.78) | Good relevance, different from A & C |
| 4 | Doc B (0.90) | Skipped earlier due to similarity to A |
## Output Schema
```json theme={null}
{
"document_id": "doc_123",
"content": "Document content...",
"score": 0.85,
"mmr": {
"relevance_score": 0.92,
"diversity_penalty": 0.07,
"mmr_score": 0.85,
"selection_order": 3
}
}
```
## Performance
| Metric | Value |
| -------------- | ------------------- |
| **Latency** | 10-50ms |
| **Complexity** | O(N × top\_k) |
| **Memory** | O(N) embeddings |
| **Cost** | Free (no API calls) |
## Common Pipeline Patterns
### Search + MMR
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "mmr",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": {
"lambda": 0.5,
"top_k": 10
}
}
}
]
```
### Search + Filter + MMR + Enrich
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.category",
"operator": "in",
"value": ["tech", "science", "business"]
}
}
}
},
{
"stage_name": "mmr",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": {
"lambda": 0.6,
"top_k": 15
}
}
},
{
"stage_name": "llm_enrichment",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"model": "gpt-4o-mini",
"prompt": "Summarize in one sentence",
"output_field": "summary"
}
}
}
]
```
### Diverse RAG Pipeline
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "mmr",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": {
"lambda": 0.4,
"top_k": 8
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Synthesize diverse perspectives on: {{INPUT.query}}\n\n{{DOCUMENTS}}"
}
}
}
]
```
## MMR vs Rerank
| Aspect | MMR | Rerank |
| -------- | --------------------- | --------------------- |
| Goal | Diversity + relevance | Maximum relevance |
| Method | Embedding similarity | Cross-encoder scoring |
| Speed | Fast (10-50ms) | Slower (50-100ms) |
| Best for | Exploration, coverage | Precision, accuracy |
## Tuning Lambda
| Use Case | Recommended Lambda |
| ---------------- | ------------------------- |
| News aggregation | 0.3-0.4 (high diversity) |
| Product search | 0.5-0.6 (balanced) |
| Technical docs | 0.7-0.8 (relevance focus) |
| Legal/compliance | 0.8-0.9 (high precision) |
<Tip>
Start with lambda=0.5 and adjust based on user feedback. If users complain about redundant results, lower lambda. If they miss relevant results, raise it.
</Tip>
## Error Handling
| Error | Behavior |
| ------------------- | --------------------------- |
| Missing embeddings | Fall back to relevance sort |
| Empty input | Return empty |
| top\_k > input size | Return all documents |
| Invalid lambda | Clamp to \[0, 1] |
## Related
* [Rerank](/docs/retrieval/stages/rerank) - Pure relevance re-scoring
* [Sort Relevance](/docs/retrieval/stages/sort-relevance) - Simple score sorting
* [Sample](/docs/retrieval/stages/sample) - Random diversity
# Moment Group
Source: https://docs.mixpeek.com/docs/retrieval/stages/moment-group
Merge contiguous temporal intervals into consolidated video moments, grouped by parent object
<Frame>
<img alt="Moment Group stage merging frame-level intervals into consolidated video moments" />
</Frame>
The Moment Group stage takes frame-level or chunk-level documents with temporal metadata and merges contiguous intervals into consolidated video moments (time ranges), grouped by parent object. Instead of returning individual frames that matched a query, you get precise start/end time ranges pinpointing where in a video the match occurs.
<Note>
**Stage Category**: REDUCE (Merges intervals into moments)
**Transformation**: N frame/chunk documents → M consolidated moments per parent video
</Note>
## When to Use
| Use Case | Description |
| -------------------------- | ------------------------------------------------------ |
| **Sub-scene localization** | Pinpoint exactly where in a video a query matches |
| **Moment extraction** | Return time-range clips instead of individual frames |
| **Video search results** | Provide start/end timestamps for player seek |
| **Highlight reels** | Extract the top-scoring moments across a video library |
## When NOT to Use
| Scenario | Recommended Alternative |
| ---------------------------------------- | ----------------------- |
| Grouping by non-temporal fields | `group_by` |
| Time-window aggregations (hour/day/week) | `temporal` |
| Results without temporal metadata | `aggregate` |
| Semantic clustering of results | `cluster` |
## Requirements
* Input documents must have temporal intervals — either `query_chunks` with `start_ms`/`end_ms` (attached by `feature_search` preprocessing) or document-level time fields.
* The `parent_field` must exist on the documents so results can be grouped by source video/object.
<Warning>
**Text query vs file query — pick the right `time_field`.** The default `time_field: "query_chunks"` is only populated when you search with a **file** input (via [query preprocessing](/docs/retrieval/stages/feature-search#query-preprocessing)). For a plain **text** query, the matched video segments carry top-level `start_time`/`end_time` — set `time_field: "start_time"` (the stage derives the end field by swapping `start`→`end`). Leaving `query_chunks` with a text query yields **empty moments** — the most common silent failure here. See the [Video Moment Localization recipe](/docs/retrieval/cookbook#video-moment-localization).
</Warning>
## Parameters
| Parameter | Type | Default | Description |
| ------------------------ | ------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `parent_field` | string | `"source_object_id"` | Field to group results by parent video/object. Common values: `source_object_id` (standard ingest lineage), `root_object_id` (top-level ancestor), or any custom field. |
| `time_field` | string | `"query_chunks"` | Source of temporal intervals. `"query_chunks"` reads the array `feature_search` attaches (each entry has `start_ms`, `end_ms`, `score`). For document-level timestamps, use a dot-path like `"metadata.start_ms"` — the stage derives the end field by replacing `"start"` with `"end"`. |
| `merge_tolerance_ms` | integer | `2000` | Maximum gap in milliseconds between intervals before they are split into separate moments. `0` = exact-overlap only. Range: 0–60000. |
| `max_moments_per_parent` | integer | `10` | Maximum moments returned per parent, sorted by score (highest first). Range: 1–100. |
| `score_strategy` | string | `"max"` | How to aggregate scores across merged intervals: `"max"`, `"avg"`, or `"sum"`. |
| `min_score` | float | `null` | Drop moments scoring below this threshold. Range: 0.0–1.0. |
| `output_mode` | string | `"annotated"` | `"annotated"` keeps the best-scoring document per parent and attaches a `moments` array (preserves all original document fields). `"moments_only"` emits standalone moment documents with IDs like `moment_{parent}_{start_ms}`. |
## Configuration Examples
<CodeGroup>
```json Basic Moment Grouping theme={null}
{
"stage_name": "moment_group",
"stage_type": "reduce",
"config": {
"stage_id": "moment_group",
"parameters": {
"parent_field": "source_object_id",
"time_field": "query_chunks",
"merge_tolerance_ms": 2000,
"max_moments_per_parent": 5,
"score_strategy": "max",
"output_mode": "annotated"
}
}
}
```
```json Tight Scene Localization theme={null}
{
"stage_name": "moment_group",
"stage_type": "reduce",
"config": {
"stage_id": "moment_group",
"parameters": {
"parent_field": "source_object_id",
"time_field": "query_chunks",
"merge_tolerance_ms": 1500,
"max_moments_per_parent": 3,
"score_strategy": "max",
"min_score": 0.6,
"output_mode": "annotated"
}
}
}
```
```json Standalone Moment Documents theme={null}
{
"stage_name": "moment_group",
"stage_type": "reduce",
"config": {
"stage_id": "moment_group",
"parameters": {
"parent_field": "root_object_id",
"time_field": "query_chunks",
"merge_tolerance_ms": 3000,
"max_moments_per_parent": 10,
"score_strategy": "avg",
"output_mode": "moments_only"
}
}
}
```
```json Document-Level Timestamps theme={null}
{
"stage_name": "moment_group",
"stage_type": "reduce",
"config": {
"stage_id": "moment_group",
"parameters": {
"parent_field": "source_object_id",
"time_field": "metadata.start_ms",
"merge_tolerance_ms": 2000,
"max_moments_per_parent": 5,
"score_strategy": "max",
"output_mode": "annotated"
}
}
}
```
</CodeGroup>
## Output Schema
### Annotated Mode (default)
In `annotated` mode, the best-scoring document per parent is preserved with a `moments` array attached:
```json theme={null}
{
"document_id": "doc_abc123",
"source_object_id": "video_001",
"score": 0.92,
"content": "...",
"metadata": { "..." : "..." },
"moments": [
{
"start_ms": 12000,
"end_ms": 18500,
"score": 0.92,
"match_count": 4,
"document_ids": ["doc_abc123", "doc_abc124", "doc_abc125", "doc_abc126"]
},
{
"start_ms": 45000,
"end_ms": 51000,
"score": 0.78,
"match_count": 2,
"document_ids": ["doc_abc130", "doc_abc131"]
}
]
}
```
### Moments-Only Mode
In `moments_only` mode, standalone moment documents are emitted:
```json theme={null}
{
"document_id": "moment_video_001_12000",
"start_ms": 12000,
"end_ms": 18500,
"score": 0.92,
"match_count": 4,
"document_ids": ["doc_abc123", "doc_abc124", "doc_abc125", "doc_abc126"]
}
```
### Moment Object Fields
| Field | Type | Description |
| -------------- | ------- | ------------------------------------------------------------- |
| `start_ms` | integer | Start of the moment in milliseconds |
| `end_ms` | integer | End of the moment in milliseconds |
| `score` | float | Aggregated score (per `score_strategy`) |
| `match_count` | integer | Number of source intervals merged into this moment |
| `document_ids` | array | IDs of the original documents that contributed to this moment |
## Performance
| Metric | Value |
| --------------- | ------------------------------------------------- |
| **Latency** | 5-50ms |
| **Memory** | O(N) where N = input documents |
| **Cost** | Free |
| **Scalability** | Efficient — runs in the API layer, no engine call |
## Common Pipeline Patterns
### Video Search with Moment Localization
The canonical use case: search across video frames, then consolidate matches into seekable moments.
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 200
}
],
"final_top_k": 200
}
}
},
{
"stage_name": "moment_group",
"stage_type": "reduce",
"config": {
"stage_id": "moment_group",
"parameters": {
"parent_field": "source_object_id",
"time_field": "start_time",
"merge_tolerance_ms": 1500,
"max_moments_per_parent": 3,
"score_strategy": "max",
"min_score": 0.6,
"output_mode": "annotated"
}
}
}
]
```
### Moment Extraction with Reranking
Search, rerank for precision, then group into moments:
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 500
}
],
"final_top_k": 500
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"top_k": 100
}
}
},
{
"stage_name": "moment_group",
"stage_type": "reduce",
"config": {
"stage_id": "moment_group",
"parameters": {
"parent_field": "source_object_id",
"time_field": "start_time",
"merge_tolerance_ms": 2000,
"max_moments_per_parent": 5,
"score_strategy": "max",
"output_mode": "annotated"
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ---------------------------------- | -------------------------- |
| Missing `parent_field` on document | Document skipped |
| Missing temporal metadata | Document skipped |
| No intervals pass `min_score` | Parent omitted from output |
| Empty input | Empty results returned |
## Related
* [Temporal](/docs/retrieval/stages/temporal) - Time-window aggregations with drift detection
* [Group By](/docs/retrieval/stages/group-by) - Group documents by any field value
* [Aggregate](/docs/retrieval/stages/aggregate) - Statistical aggregations
* [Deduplicate](/docs/retrieval/stages/deduplicate) - Remove duplicate documents
# Retriever Stages
Source: https://docs.mixpeek.com/docs/retrieval/stages/overview
The core of the warehouse query layer: composable retriever stages for multi-stage search pipelines
Retriever stages are the building blocks of search pipelines in Mixpeek. Each stage performs a specific operation on the document set, allowing you to compose complex retrieval workflows from simple, reusable components. Where a traditional database returns rows, the warehouse query layer returns ranked, enriched, and fused multimodal results assembled stage by stage.
<Card title="Compose stages over your own data" icon="layer-group" href="https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=docs&utm_medium=docs_stages_overview&utm_campaign=managed_onramp">
Create a managed namespace, index your files, then chain these stages into a retriever you can query.
</Card>
## Stage Categories
Stages are organized into six categories based on how they transform the document set:
<CardGroup>
<Card title="FILTER" icon="filter">
Reduce the document set by matching criteria. Outputs a subset of input documents.
**Stages**: feature\_search, attribute\_filter, llm\_filter, agent\_search, query\_expand
</Card>
<Card title="SORT" icon="arrow-up-arrow-down">
Reorder documents by relevance or field values. Same documents, different order.
**Stages**: sort\_relevance, sort\_attribute, mmr, rerank, score\_normalize
</Card>
<Card title="REDUCE" icon="compress">
Collapse results into aggregated values. Produces a single value or smaller set from the input.
**Stages**: aggregate, temporal, sample, summarize, limit, deduplicate, moment\_group, score\_threshold
</Card>
<Card title="GROUP" icon="layer-group">
Reshape results by bucketing documents into logical groups or clusters.
**Stages**: group\_by, cluster
</Card>
<Card title="APPLY" icon="wand-magic-sparkles">
Transform or restructure documents. May reshape fields, create new documents, or call external services.
**Stages**: json\_transform, rag\_prepare, external\_web\_search, api\_call, sql\_lookup, cross\_compare, web\_scrape, unwind, code\_execution
</Card>
<Card title="ENRICH" icon="sparkles">
Add knowledge to documents using AI models, taxonomies, or cross-collection joins.
**Stages**: llm\_enrich, taxonomy\_enrich, document\_enrich, agentic\_enrich
</Card>
</CardGroup>
## All Stages
### Filter Stages
| Stage | Description |
| ------------------------------------------------------ | ------------------------------------------------------------------------------- |
| [Feature Search](/docs/retrieval/stages/feature-search) | Search by vector similarity using multimodal embeddings |
| [Attribute Filter](/docs/retrieval/stages/attribute-filter) | Filter by metadata fields with boolean logic (AND/OR/NOT) |
| [LLM Filter](/docs/retrieval/stages/llm-filter) | Semantic filtering using LLM-based evaluation |
| [Agent Search](/docs/retrieval/stages/agent-search) | LLM-driven multi-step retrieval with iterative reasoning and tool orchestration |
| [Query Expand](/docs/retrieval/stages/query-expand) | LLM-powered query expansion with RRF result fusion |
### Sort Stages
| Stage | Description |
| ---------------------------------------------------- | ---------------------------------------------------------- |
| [Sort Relevance](/docs/retrieval/stages/sort-relevance) | Reorder by relevance scores |
| [Sort Attribute](/docs/retrieval/stages/sort-attribute) | Order by any metadata field (dates, price, etc.) |
| [MMR](/docs/retrieval/stages/mmr) | Diversify results with Maximal Marginal Relevance |
| [Rerank](/docs/retrieval/stages/rerank) | Re-score with cross-encoder models (e.g., BGE reranker) |
| [Score Normalize](/docs/retrieval/stages/score-normalize) | Rescale scores to a common range for consistent comparison |
### Reduce Stages
| Stage | Description |
| ---------------------------------------------------- | ------------------------------------------------------------------------------ |
| [Aggregate](/docs/retrieval/stages/aggregate) | Compute COUNT, SUM, AVG, percentile, stddev, frequency, correlation on results |
| [Temporal](/docs/retrieval/stages/temporal) | Group by time windows (hour/day/week/month/quarter/year) with drift detection |
| [Sample](/docs/retrieval/stages/sample) | Random or stratified sampling of results |
| [Summarize](/docs/retrieval/stages/summarize) | Condense documents into an LLM-generated summary |
| [Limit](/docs/retrieval/stages/limit) | Truncate results to a maximum count with optional offset |
| [Deduplicate](/docs/retrieval/stages/deduplicate) | Remove duplicate documents by field or content similarity |
| [Moment Group](/docs/retrieval/stages/moment-group) | Merge contiguous temporal intervals into consolidated video moments |
| [Score Threshold](/docs/retrieval/stages/score-threshold) | Drop results below an absolute score; return none when nothing qualifies |
### Group Stages
| Stage | Description |
| -------------------------------------- | ---------------------------------------------------- |
| [Group By](/docs/retrieval/stages/group-by) | Group documents by field value (decompose/recompose) |
| [Cluster](/docs/retrieval/stages/cluster) | Discover themes via embedding-based clustering |
### Apply Stages
| Stage | Description |
| ------------------------------------------------------------ | ---------------------------------------------------------- |
| [JSON Transform](/docs/retrieval/stages/json-transform) | Reshape documents using Jinja2 templates |
| [RAG Prepare](/docs/retrieval/stages/rag-prepare) | Format for LLM context with token management and citations |
| [External Web Search](/docs/retrieval/stages/external-web-search) | Augment with Exa AI-native web search |
| [API Call](/docs/retrieval/stages/api-call) | Enrich with external REST API responses |
| [SQL Lookup](/docs/retrieval/stages/sql-lookup) | Join with PostgreSQL/Snowflake data |
| [Cross Compare](/docs/retrieval/stages/cross-compare) | Multi-tier cross-collection matching with classification |
| [Web Scrape](/docs/retrieval/stages/web-scrape) | Extract full page content from URLs |
| [Unwind](/docs/retrieval/stages/unwind) | Decompose array fields into separate documents |
| [Code Execution](/docs/retrieval/stages/code-execution) | Execute Python/TypeScript/JavaScript in sandboxes |
### Enrich Stages
| Stage | Description |
| ---------------------------------------------------- | ------------------------------------------------------------ |
| [LLM Enrich](/docs/retrieval/stages/llm-enrich) | Generate new fields with LLM prompts |
| [Taxonomy Enrich](/docs/retrieval/stages/taxonomy-enrich) | Classify documents against taxonomy nodes |
| [Document Enrich](/docs/retrieval/stages/document-enrich) | Cross-collection joins (LEFT JOIN) |
| [Agentic Enrich](/docs/retrieval/stages/agentic-enrich) | Multi-turn agent with tool access for complex classification |
## Pipeline Patterns
### Basic RAG Pipeline
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 50}],
"final_top_k": 50
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "content",
"top_k": 10
}
}
},
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 8000,
"output_mode": "single_context"
}
}
}
]
```
### E-Commerce Search
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 100}],
"final_top_k": 100
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"AND": [
{"field": "metadata.in_stock", "operator": "eq", "value": true},
{"field": "metadata.price", "operator": "lte", "value": "{{INPUT.max_price}}"}
]
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.{{INPUT.sort_by}}",
"direction": "{{INPUT.sort_order}}"
}
}
}
]
```
### Research Assistant
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 100}],
"final_top_k": 100
}
}
},
{
"stage_name": "external_web_search",
"stage_type": "apply",
"config": {
"stage_id": "external_web_search",
"parameters": {
"query": "{{INPUT.query}}",
"num_results": 10,
"category": "research_paper"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"top_k": 15
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Synthesize findings on: {{INPUT.query}}"
}
}
}
]
```
### Enriched Document Retrieval
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 20}],
"final_top_k": 20
}
}
},
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_users",
"source_field": "metadata.author_id",
"target_field": "user_id",
"output_field": "author"
}
}
},
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Extract key topics and entities from: {{DOC.content}}",
"output_field": "analysis"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"top_k": 10
}
}
}
]
```
## Stage Selection Guide
| Goal | Recommended Stage |
| ----------------------------------------- | --------------------- |
| Find semantically similar documents | feature\_search |
| Filter by metadata fields | attribute\_filter |
| Filter by content meaning | llm\_filter |
| Improve recall with query variations | query\_expand |
| Get best relevance ranking | rerank |
| Order by price/date/rating | sort\_attribute |
| Re-sort by relevance scores | sort\_relevance |
| Diversify results | mmr |
| Normalize scores across sources | score\_normalize |
| Suppress weak results / "no good results" | score\_threshold |
| Truncate to top-N results | limit |
| Remove duplicate results | deduplicate |
| Expand array fields to documents | unwind |
| Answer questions from docs | summarize |
| Compute statistics on results | aggregate |
| Find themes in results | cluster |
| Locate moments/scenes in video | moment\_group |
| Group by category/author | group\_by |
| Random/stratified sampling | sample |
| Add external API data | api\_call |
| Add database data | sql\_lookup |
| Join Mixpeek collections | document\_enrich |
| Classify documents | taxonomy\_enrich |
| Complex multi-step classification | agentic\_enrich |
| Generate new fields with LLM | llm\_enrich |
| Transform document structure | json\_transform |
| Prepare for LLM context | rag\_prepare |
| Custom code transformations | code\_execution |
| Add web search results | external\_web\_search |
| Extract URL content | web\_scrape |
## Performance Considerations
| Stage | Typical Latency | Cost |
| --------------------- | --------------- | -------------------- |
| feature\_search | 5-50ms | Index storage |
| attribute\_filter | \< 5ms | Free |
| llm\_filter | 200-500ms | LLM API |
| query\_expand | 300-800ms | LLM API |
| rerank | 50-100ms | Inference |
| sort\_attribute | \< 5ms | Free |
| sort\_relevance | \< 5ms | Free |
| mmr | 10-50ms | Free |
| score\_normalize | \< 1ms | Free |
| score\_threshold | \< 1ms | Free |
| limit | \< 1ms | Free |
| deduplicate | 5-50ms | Free |
| unwind | \< 5ms | Free |
| summarize | 500-2000ms | LLM API |
| aggregate | 5-50ms | Free |
| cluster | 50-200ms | Inference |
| moment\_group | 5-50ms | Free |
| group\_by | 5-20ms | Free |
| sample | \< 5ms | Free |
| llm\_enrich | 300-800ms | LLM API |
| agentic\_enrich | 2-30s | LLM API (multi-turn) |
| api\_call | 50-500ms | External API |
| sql\_lookup | 10-100ms | Database |
| code\_execution | 5-50ms | Free |
| rag\_prepare | \< 10ms | Free |
| json\_transform | \< 5ms | Free |
| external\_web\_search | 100-500ms | Exa API |
| taxonomy\_enrich | 20-100ms | Inference |
| document\_enrich | 10-50ms | Database |
| web\_scrape | 500-5000ms | External |
<Tip>
Order stages efficiently: cheap operations (filters, sorts) before expensive ones (rerank, LLM calls). This reduces the document count before costly processing.
</Tip>
## Template Variables
All stages support template variables for dynamic configuration:
| Variable | Description |
| --------------- | ---------------------------------------------------- |
| `{{INPUT.*}}` | Input parameters from retriever call |
| `{{DOC.*}}` | Document fields (in APPLY, ENRICH, and GROUP stages) |
| `{{CONTEXT.*}}` | Pipeline context (index, citations) |
```json theme={null}
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.tenant_id",
"operator": "eq",
"value": "{{INPUT.tenant_id}}"
}
}
}
```
# Query Expand
Source: https://docs.mixpeek.com/docs/retrieval/stages/query-expand
Generate query variations using LLMs and fuse results for improved recall
<Frame>
<img alt="Query Expand stage showing LLM-powered query variations and result fusion" />
</Frame>
The Query Expand stage uses language models to generate multiple query variations from the original query, executes searches for each variation, and fuses the results. This improves recall by capturing different phrasings and aspects of the user's intent.
<Frame>
<img alt="Query expansion explained: an ambiguous two-word query (jaguar speed) is expanded by an LLM into multiple reformulations covering each interpretation, each searches independently, and results fuse via Reciprocal Rank Fusion into one broader ranked list; HyDE instead generates a hypothetical answer document and embeds that paragraph for more signal. Multi-query improves recall for ambiguous queries, HyDE improves precision for short single-intent queries, and both can be combined." />
</Frame>
<Note>
**Stage Category**: FILTER (Generates and fuses search results)
**Transformation**: 1 query → N query variations → fused results
</Note>
## When to Use
| Use Case | Description |
| ----------------------- | -------------------------------------------------- |
| **Improved recall** | Capture documents that match alternative phrasings |
| **Ambiguous queries** | Handle queries with multiple interpretations |
| **Synonym expansion** | Find documents using different terminology |
| **Multi-aspect search** | Break complex queries into sub-queries |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------------- | ------------------------- |
| Simple keyword search | `feature_search` directly |
| Low latency requirements | Pre-compute expansions |
| Precise single-intent queries | Standard search |
| Cost-sensitive applications | Use simpler search |
## Parameters
| Parameter | Type | Default | Description |
| ------------------ | ------- | ----------------- | -------------------------------------- |
| `model` | string | *Required* | LLM model for query generation |
| `query` | string | `{{INPUT.query}}` | Original query to expand |
| `num_variations` | integer | `3` | Number of query variations to generate |
| `vector_index` | string | *Required* | Vector index for searches |
| `top_k` | integer | `20` | Results per query variation |
| `fusion_strategy` | string | `rrf` | Result fusion: `rrf` or `linear` |
| `expansion_prompt` | string | *auto* | Custom prompt for query generation |
## Fusion Methods
| Method | Description | Best For |
| -------- | -------------------------- | -------------------------- |
| `rrf` | Reciprocal Rank Fusion | General purpose, balanced |
| `linear` | Weighted score combination | When scores are comparable |
| `max` | Take maximum score | When any match is good |
## Configuration Examples
<CodeGroup>
```json Basic Query Expansion theme={null}
{
"stage_name": "query_expand",
"stage_type": "filter",
"config": {
"stage_id": "query_expand",
"parameters": {
"model": "gpt-4o-mini",
"query": "{{INPUT.query}}",
"vector_index": "text_extractor_v1_embedding",
"num_variations": 3,
"top_k": 30
}
}
}
```
```json Custom Expansion Prompt theme={null}
{
"stage_name": "query_expand",
"stage_type": "filter",
"config": {
"stage_id": "query_expand",
"parameters": {
"model": "gpt-4o-mini",
"query": "{{INPUT.query}}",
"vector_index": "text_extractor_v1_embedding",
"num_variations": 5,
"top_k": 20,
"expansion_prompt": "Generate {{num_variations}} alternative phrasings of this query, focusing on different synonyms and related concepts: {{query}}"
}
}
}
```
```json High-Recall Configuration theme={null}
{
"stage_name": "query_expand",
"stage_type": "filter",
"config": {
"stage_id": "query_expand",
"parameters": {
"model": "gpt-4o",
"query": "{{INPUT.query}}",
"vector_index": "text_extractor_v1_embedding",
"num_variations": 5,
"top_k": 50,
"fusion_strategy": "rrf"
}
}
}
```
```json Domain-Specific Expansion theme={null}
{
"stage_name": "query_expand",
"stage_type": "filter",
"config": {
"stage_id": "query_expand",
"parameters": {
"model": "gpt-4o-mini",
"query": "{{INPUT.query}}",
"vector_index": "medical_embedding",
"num_variations": 4,
"expansion_prompt": "Generate {{num_variations}} medical search queries that capture different aspects of: {{query}}. Include medical terminology and layman's terms."
}
}
}
```
</CodeGroup>
## How Query Expansion Works
1. **Original Query**: "how to fix memory leaks"
2. **LLM Generates Variations**:
* "memory leak detection and resolution"
* "debugging memory issues in applications"
* "preventing memory leaks in code"
3. **Execute Searches**: Run vector search for each variation
4. **Fuse Results**: Combine using RRF or other fusion method
5. **Return**: Deduplicated, ranked result set
## Reciprocal Rank Fusion (RRF)
RRF combines results from multiple queries using the formula:
```
score(doc) = Σ 1 / (k + rank_i)
```
Where `k` is typically 60, and `rank_i` is the document's rank in query i's results.
| Advantage | Description |
| -------------- | ----------------------------------- |
| Score-agnostic | Works with different scoring scales |
| Rank-based | Focuses on relative ordering |
| Self-balancing | No manual weight tuning |
## Output Schema
Each document includes fusion metadata:
```json theme={null}
{
"document_id": "doc_123",
"content": "Document content...",
"score": 0.87,
"query_expand": {
"matched_variations": ["memory leak detection", "debugging memory issues"],
"fusion_score": 0.87,
"individual_ranks": [2, 5, null]
}
}
```
## Performance
| Metric | Value |
| ---------------- | -------------------------- |
| **Latency** | 300-800ms (LLM + searches) |
| **LLM calls** | 1 per execution |
| **Search calls** | N (num\_variations) |
| **Token usage** | \~50-100 tokens |
<Warning>
Query expansion adds latency due to LLM generation and multiple searches. Use judiciously for queries where recall improvement justifies the cost.
</Warning>
## Common Pipeline Patterns
### Expanded Search + Rerank
```json theme={null}
[
{
"stage_name": "query_expand",
"stage_type": "filter",
"config": {
"stage_id": "query_expand",
"parameters": {
"model": "gpt-4o-mini",
"query": "{{INPUT.query}}",
"vector_index": "text_extractor_v1_embedding",
"num_variations": 3,
"top_k": 50
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
}
]
```
### Expansion + Filter + Summarize
```json theme={null}
[
{
"stage_name": "query_expand",
"stage_type": "filter",
"config": {
"stage_id": "query_expand",
"parameters": {
"model": "gpt-4o-mini",
"query": "{{INPUT.query}}",
"vector_index": "text_extractor_v1_embedding",
"num_variations": 4,
"top_k": 30
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.verified",
"operator": "eq",
"value": true
}
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Answer based on the documents: {{INPUT.query}}"
}
}
}
]
```
## Cost Optimization
| Strategy | Impact |
| -------------------------- | ------------------------- |
| Reduce num\_variations | Fewer searches |
| Use cheaper LLM | `gpt-4o-mini` vs `gpt-4o` |
| Lower top\_k per variation | Less fusion overhead |
| Cache common expansions | Reduce LLM calls |
## Error Handling
| Error | Behavior |
| ---------------- | --------------------------- |
| LLM failure | Fall back to original query |
| Search failure | Skip that variation |
| Empty expansions | Use original query only |
| Timeout | Return partial results |
## Related
* [Feature Search](/docs/retrieval/stages/feature-search) - Single query search
* [Feature Search](/docs/retrieval/stages/feature-search) - Vector + text search
* [Rerank](/docs/retrieval/stages/rerank) - Re-score fused results
# RAG Prepare
Source: https://docs.mixpeek.com/docs/retrieval/stages/rag-prepare
Prepare documents for LLM context windows with token management and source citations — evidence/provenance per result so every claim traces back to a source clip
<Frame>
<img alt="RAG Prepare stage showing document formatting for LLM context" />
</Frame>
The RAG Prepare stage formats search results for LLM consumption by managing token budgets, formatting documents, and adding citations. This is a **preparation stage that does NOT call an LLM** - it prepares content for downstream LLM stages or external LLM calls.
<Note>
**Stage Category**: APPLY
**Transformation**:
* `single_context` mode: N documents → 1 combined context document
* `formatted_list` mode: N documents → N formatted documents
</Note>
## When to Use
| Use Case | Description |
| --------------------------- | ----------------------------------------- |
| **Before LLM generation** | Prepare context for summarization or Q\&A |
| **Token budget management** | Fit multiple docs into context window |
| **Citation tracking** | Enable source attribution in responses |
| **Consistent formatting** | Standardize document format for LLM input |
## When NOT to Use
| Scenario | Recommended Alternative |
| ---------------------------- | ----------------------------- |
| Want LLM to generate summary | `summarize` stage (calls LLM) |
| Don't need token management | Pass documents directly |
| Simple pass-through | Skip this stage |
## Parameters
| Parameter | Type | Default | Description |
| --------------------- | ------- | ----------------------------------------- | ----------------------------------- |
| `max_tokens` | integer | `8000` | Maximum tokens for combined output |
| `tokenizer` | string | `cl100k_base` | Tokenizer to use (GPT-4 compatible) |
| `truncation_strategy` | string | `priority_truncate` | How to handle token overflow |
| `output_mode` | string | `single_context` | Output format |
| `document_template` | string | `[{{CONTEXT.INDEX}}] {{DOC.content}}\n\n` | Template for each document |
| `content_field` | string | `content` | Field to extract content from |
| `separator` | string | `\n` | Separator between documents |
| `citation` | object | `{style: "numbered"}` | Citation configuration |
### Truncation Strategies
| Strategy | Behavior |
| ------------------- | ------------------------------------------------- |
| `priority_truncate` | Include docs in score order, truncate last to fit |
| `proportional` | Give each doc proportional token budget |
| `drop_last` | Include complete docs until limit, drop remaining |
### Output Modes
| Mode | Output | Use Case |
| ---------------- | ------------------------------------------ | ----------------- |
| `single_context` | 1 document with combined `context` string | Direct LLM input |
| `formatted_list` | N documents with `formatted_content` field | Custom processing |
## Configuration Examples
<CodeGroup>
```json Basic RAG Context theme={null}
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 8000,
"output_mode": "single_context"
}
}
}
```
```json With Numbered Citations theme={null}
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 4000,
"document_template": "[{{CONTEXT.INDEX}}] {{DOC.metadata.title}}\n{{DOC.content}}\n\n",
"truncation_strategy": "priority_truncate",
"citation": {
"style": "numbered",
"include_title": true
}
}
}
}
```
```json Large Context Window (GPT-4 Turbo) theme={null}
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 32000,
"tokenizer": "cl100k_base",
"truncation_strategy": "proportional"
}
}
}
```
```json Formatted List Mode theme={null}
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"output_mode": "formatted_list",
"document_template": "Source: {{DOC.metadata.source}}\n{{DOC.content}}"
}
}
}
```
```json Custom Template with URL theme={null}
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 8000,
"document_template": "Document {{CONTEXT.INDEX}}:\nTitle: {{DOC.metadata.title}}\nURL: {{DOC.metadata.url}}\n\n{{DOC.content}}\n\n---\n",
"citation": {
"style": "bracketed",
"include_url": true
}
}
}
}
```
</CodeGroup>
## Template Placeholders
| Placeholder | Description |
| ---------------------- | ---------------------------------------------------------------------- |
| `{{CONTEXT.INDEX}}` | 1-based position in result set (1, 2, 3...) |
| `{{CONTEXT.CITATION}}` | Citation marker based on citation.style |
| `{{DOC.*}}` | Any document field (e.g., `{{DOC.content}}`, `{{DOC.metadata.title}}`) |
## Citation Styles
| Style | Output | Example |
| ----------- | ------------------- | ---------------------- |
| `numbered` | `[1]`, `[2]`, `[3]` | Default, clean |
| `bracketed` | `[doc_id]` | Document ID references |
| `footnote` | Superscript numbers | Academic style |
| `none` | No citations | When not needed |
## Output Schema
### single\_context Mode
```json theme={null}
{
"rag_context": "[1] First document content...\n\n[2] Second document content...",
"citations": [
{"index": 1, "title": "Document Title", "document_id": "doc_123"},
{"index": 2, "title": "Another Title", "document_id": "doc_456"}
]
}
```
### formatted\_list Mode
Each document gets:
```json theme={null}
{
"document_id": "doc_123",
"formatted_content": "[1] Title\nContent here...",
"original_content": "Content here...",
"metadata": {...}
}
```
## Citing Timestamped Source Clips (Video / Audio)
The default `citation` object is `{index, title, document_id}` — enough to name the
source, but it does **not** inline the timestamp or a link back to the source clip.
For investigative / RAG use cases where every claim must resolve to a *timestamped,
clickable source* (e.g. "watch this at 03:12"), surface the anchor yourself — the
data is already on every document, so this needs **no new stage or endpoint**:
* Video/audio segment documents carry `start_time_s` / `end_time_s`
(see [Universal Extractor → Output](/docs/processing/extractors/universal)).
* Every document carries `_internal.lineage.root_object_id` pointing back to the
original source object (see [Lineage Traversal](/docs/retrieval/lineage-traversal)).
Reference those fields directly in `document_template` so the timestamp travels into
the LLM context (and into any answer the LLM cites):
```json Timestamped clip citations theme={null}
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 8000,
"output_mode": "single_context",
"document_template": "[{{CONTEXT.INDEX}}] {{DOC.metadata.title}} @ {{DOC.start_time_s}}s–{{DOC.end_time_s}}s (source: {{DOC._internal.lineage.root_object_id}})\n{{DOC.text}}\n\n",
"citation": { "style": "numbered", "include_title": true }
}
}
}
```
To render a **clickable** clip link, hydrate the `document_id` returned in the
`citations[]` array (or the `root_object_id` above) into a playable URL with
`GET /v1/documents/{id}?expand=root_object` — see
[Lineage Traversal](/docs/retrieval/lineage-traversal). Do this in the app layer after
`execute`; the timestamp and source object are preserved end-to-end in the document,
they are simply not inlined into the structured `citations[]` array by default.
<Note>
Citations attach at the **document** level. `rag_prepare` (and `summarize`'s
`source_document_ids`) cite the set of source documents behind the context, not
individual sentences — per-claim/per-sentence attribution is up to the downstream
LLM prompt. Ask the LLM to emit the `[n]` marker inline with each claim so the
numbered citation resolves back to the timestamped clip above.
</Note>
## Performance
| Metric | Value |
| ------------------ | ------------------------ |
| **Latency** | \< 10ms |
| **Token counting** | Uses tiktoken (accurate) |
| **No LLM calls** | Pure formatting |
<Tip>
This stage does NOT call an LLM. It only formats content for LLM consumption. Use the `summarize` stage if you want LLM-generated summaries.
</Tip>
## Common Pipeline Patterns
### Search + Prepare + External LLM
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
},
{
"stage_name": "rag_prepare",
"stage_type": "apply",
"config": {
"stage_id": "rag_prepare",
"parameters": {
"max_tokens": 8000,
"output_mode": "single_context",
"citation": {"style": "numbered"}
}
}
}
]
```
The output `rag_context` can then be passed to an external LLM call.
### vs Summarize Stage
| Feature | rag\_prepare | summarize |
| --------- | ------------------------ | ----------------- |
| Calls LLM | No | Yes |
| Output | Formatted context | Generated summary |
| Latency | \< 10ms | 500-2000ms |
| Cost | Free | LLM API costs |
| Use case | Prepare for external LLM | End-to-end RAG |
## Related
* [Summarize](/docs/retrieval/stages/summarize) - LLM-powered summarization
* [JSON Transform](/docs/retrieval/stages/json-transform) - Custom document formatting
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - Extract structured data with LLM
# Rerank
Source: https://docs.mixpeek.com/docs/retrieval/stages/rerank
Re-score and reorder search results using cross-encoder models for higher precision
<Frame>
<img alt="Rerank stage showing cross-encoder model re-scoring search results" />
</Frame>
The Rerank stage uses cross-encoder models to re-score and reorder search results. Unlike bi-encoder models (used in semantic search), cross-encoders process the query and document together, enabling more accurate relevance scoring at the cost of higher latency.
<Note>
**Stage Category**: SORT (Reorders documents)
**Transformation**: N documents → top\_k documents (re-ranked by relevance)
</Note>
## When to Use
| Use Case | Description |
| ------------------------------- | ----------------------------------------------- |
| **Two-stage retrieval** | Fast recall (search) + precise ranking (rerank) |
| **High-precision requirements** | When ranking quality is critical |
| **Top-N optimization** | Improve quality of final displayed results |
| **RAG applications** | Better context selection for LLM generation |
## When NOT to Use
| Scenario | Recommended Alternative |
| -------------------------------- | ----------------------------- |
| Large result sets (1000+) | Too slow; use `sort_by_field` |
| Real-time requirements (\< 20ms) | Use search scores directly |
| Simple attribute sorting | `sort_by_field` |
## Parameters
| Parameter | Type | Default | Description |
| ---------------- | ------- | -------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `inference_name` | string | `BAAI__bge_reranker_v2_m3` | Reranking inference service. List options with `GET /engine/inference`. Ignored when `feature_uri` is set. |
| `feature_uri` | string | `null` | Custom reranker plugin (`mixpeek://...`). Overrides `inference_name`. |
| `top_k` | integer | `null` | Number of results to keep after reranking (omit to keep all, reordered). |
| `query` | string | `{{INPUT.query}}` | Query for relevance scoring |
| `document_field` | string | `content` | Document field to rerank against |
## Available Models
The built-in reranker is `BAAI__bge_reranker_v2_m3` (multilingual cross-encoder). For other models, deploy your own via a [custom extractor](/docs/processing/custom-extractors) and reference it with `feature_uri`.
## Configuration Examples
<CodeGroup>
```json Basic Reranking theme={null}
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
}
```
```json High-Quality Reranking theme={null}
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 5
}
}
}
```
```json Custom Query theme={null}
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.question}}",
"top_k": 20
}
}
}
```
```json Large Candidate Pool theme={null}
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 50
}
}
}
```
</CodeGroup>
## How Cross-Encoders Work
| Bi-Encoder (Search) | Cross-Encoder (Rerank) |
| -------------------------------- | ------------------------------ |
| Query and doc encoded separately | Query + doc encoded together |
| Pre-compute doc embeddings | Must process each pair |
| Fast (\< 10ms for millions) | Slower (50-100ms for 100 docs) |
| Good approximate ranking | Precise relevance scoring |
Cross-encoders see the full context of both query and document together, enabling better understanding of semantic relationships.
## Two-Stage Retrieval Pattern
The recommended pattern is fast recall followed by precise reranking:
```json theme={null}
[
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
}
],
"final_top_k": 100
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
}
]
```
**Why this works:**
1. **Search stage**: Fast, retrieves 100 candidates (\< 20ms)
2. **Rerank stage**: Slower but precise, picks best 10 (50-100ms)
3. **Total**: High-quality results in 70-120ms
## Performance
| Metric | Value |
| ---------------------- | ------------------------------------- |
| **Latency** | 50-100ms (depends on candidate count) |
| **Optimal input size** | 50-200 documents |
| **Maximum practical** | \~500 documents |
| **Batching** | Automatic |
<Warning>
Reranking 1000+ documents is not recommended. Use `top_k` limits in the search stage to control candidate pool size.
</Warning>
## Output
Each returned document includes:
| Field | Type | Description |
| ----------------- | ------- | -------------------------- |
| `document_id` | string | Unique document identifier |
| `score` | float | Reranker relevance score |
| `original_score` | float | Score from previous stage |
| `rerank_position` | integer | Position after reranking |
## Common Pipeline Patterns
### Search + Rerank + Limit
```json theme={null}
[
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
}
],
"final_top_k": 100
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 20
}
}
},
{
"stage_name": "limit",
"stage_type": "reduce",
"config": {
"stage_id": "limit",
"parameters": {
"limit": 5
}
}
}
]
```
### Search + Filter + Rerank
```json theme={null}
[
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 200
}
],
"final_top_k": 200
}
}
},
{
"stage_name": "category_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.category",
"operator": "eq",
"value": "{{INPUT.category}}"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
}
]
```
## Custom Reranker (BYO Model)
Use your own reranker model deployed as a [custom extractor](/docs/processing/custom-extractors) instead of the built-in models. Set `feature_uri` to route reranking through your extractor's inference endpoint.
### Parameters
| Parameter | Type | Default | Description |
| ------------- | ------ | ------- | ----------------------------------------------------------------------------- |
| `feature_uri` | string | `null` | Feature URI of a custom reranker plugin. Overrides `inference_name` when set. |
Your plugin must accept `{pairs: [[query, doc], ...]}` and return `{scores: [float, ...]}`.
### Configuration Example
```json theme={null}
{
"stage_name": "my_rerank",
"config": {
"stage_id": "rerank",
"parameters": {
"feature_uri": "mixpeek://my_reranker@1.0.0/rerank",
"top_k": 10
}
}
}
```
<Tip>
Set `inference_type: "rerank"` in your plugin's manifest to declare compatibility with the rerank stage.
</Tip>
## Trade-offs
| Aspect | Impact |
| -------------------- | -------------------------- |
| **Higher precision** | Better relevance scoring |
| **Higher latency** | 50-100ms per batch |
| **Limited scale** | Best for \< 500 candidates |
| **API costs** | Per-document scoring |
## Related
* [Feature Search](/docs/retrieval/stages/feature-search) - Initial candidate retrieval
* [Feature Search](/docs/retrieval/stages/feature-search) - Combined vector + text search
* [Sort Attribute](/docs/retrieval/stages/sort-attribute) - Simple attribute sorting
# Sample
Source: https://docs.mixpeek.com/docs/retrieval/stages/sample
Select a random or stratified sample of documents from results
<Frame>
<img alt="Sample stage showing random and stratified sampling of results" />
</Frame>
The Sample stage selects a subset of documents from your results using random or stratified sampling. This is useful for creating representative samples, reducing result sets, or ensuring diversity across categories.
<Note>
**Stage Category**: REDUCE (Samples documents)
**Transformation**: N documents → S sampled documents (where S ≤ N)
</Note>
## When to Use
| Use Case | Description |
| -------------------------- | ---------------------------------- |
| **Representative samples** | Get a sample of large result sets |
| **A/B testing** | Random document selection |
| **Stratified selection** | Equal representation per category |
| **Cost reduction** | Sample before expensive operations |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ----------------------- |
| Top-N by relevance | `limit` or `rerank` |
| Diversity by similarity | `mmr` |
| Remove duplicates | `deduplicate` |
| All results needed | Skip sampling |
## Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------- | -------- | ------------------------------------------------------ |
| `strategy` | string | `random` | Sampling strategy: `random`, `stratified`, `reservoir` |
| `count` | integer | `10` | Number of documents to sample |
| `seed` | integer | *random* | Random seed for reproducibility |
| `stratify_by` | string | *none* | Field for stratified sampling |
| `min_per_stratum` | integer | `1` | Minimum samples per stratum |
| `preserve_top_k` | integer | `0` | Always keep top K by score, sample the rest |
## Sampling Strategies
| Strategy | Description | Best For |
| ------------ | ------------------------------ | ---------------------- |
| `random` | Uniform random selection | General sampling |
| `stratified` | Proportional samples per group | Category balance |
| `reservoir` | Memory-efficient sampling | Very large result sets |
## Configuration Examples
<CodeGroup>
```json Random Sample theme={null}
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "random",
"count": 20
}
}
}
```
```json Reproducible Random Sample theme={null}
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "random",
"count": 10,
"seed": 42
}
}
}
```
```json Stratified by Category theme={null}
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "stratified",
"stratify_by": "metadata.category",
"count": 30,
"min_per_stratum": 3
}
}
}
```
```json Reservoir Sampling theme={null}
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "reservoir",
"count": 25
}
}
}
```
```json Stratified with Total Limit theme={null}
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "stratified",
"stratify_by": "metadata.source",
"count": 30
}
}
}
```
</CodeGroup>
## How Sampling Works
### Random Sampling
Selects documents with uniform probability:
```
Input: [A, B, C, D, E, F, G, H, I, J] (10 docs)
Sample(count=3): [D, G, B] (random selection)
```
### Stratified Sampling
Ensures representation from each group:
```
Input:
- Category A: [A1, A2, A3, A4, A5]
- Category B: [B1, B2, B3]
- Category C: [C1, C2]
Stratified(min_per_stratum=2):
[A1, A3, B2, B1, C1, C2]
```
### Reservoir Sampling
Memory-efficient uniform sampling for very large or streaming result sets:
```
Input: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] (processed one at a time)
Reservoir(count=3): [2, 6, 9] (uniform random, single pass)
```
## Output Schema
```json theme={null}
{
"documents": [
{
"document_id": "doc_123",
"content": "Sampled document content...",
"score": 0.85,
"sample": {
"method": "stratified",
"stratum": "electronics",
"sample_index": 0
}
}
],
"metadata": {
"method": "stratified",
"total_input": 100,
"sample_size": 15,
"strata": {
"electronics": {"input": 45, "sampled": 5},
"clothing": {"input": 35, "sampled": 5},
"books": {"input": 20, "sampled": 5}
}
}
}
```
## Performance
| Metric | Value |
| -------------- | ---------------------------------- |
| **Latency** | \< 5ms |
| **Memory** | O(N) |
| **Cost** | Free |
| **Complexity** | O(N) random, O(N log N) stratified |
## Common Pipeline Patterns
### Search + Sample for Testing
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 1000 }
],
"final_top_k": 1000
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "random",
"count": 50,
"seed": 42
}
}
}
]
```
### Balanced Category Sample
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 500 }
],
"final_top_k": 500
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "stratified",
"stratify_by": "metadata.category",
"count": 25,
"min_per_stratum": 5
}
}
}
]
```
### Sample Before LLM Processing
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 }
],
"final_top_k": 200
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 50
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "random",
"count": 10
}
}
},
{
"stage_name": "llm_enrichment",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"model": "gpt-4o",
"prompt": "Extract key insights",
"output_field": "insights"
}
}
}
]
```
### Cluster + Sample Representatives
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 }
],
"final_top_k": 200
}
}
},
{
"stage_name": "cluster",
"stage_type": "group",
"config": {
"stage_id": "cluster",
"parameters": {
"n_clusters": 10
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "stratified",
"stratify_by": "cluster.cluster_id",
"count": 20,
"min_per_stratum": 2
}
}
}
]
```
### Multi-Source Balanced Sample
```json theme={null}
[
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.date",
"operator": "gte",
"value": "2024-01-01"
}
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"strategy": "stratified",
"stratify_by": "metadata.source",
"count": 30,
"min_per_stratum": 10
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Compare perspectives from different sources on this topic\n\n{{DOCUMENTS}}"
}
}
}
]
```
## Stratified Sampling Details
### Minimum Per Stratum
```json theme={null}
{
"strategy": "stratified",
"stratify_by": "metadata.category",
"count": 30,
"min_per_stratum": 5
}
```
Each group is guaranteed at least 5 samples (if available), with the remainder allocated proportionally up to `count`.
### Proportional Allocation
```json theme={null}
{
"strategy": "stratified",
"stratify_by": "metadata.category",
"count": 30
}
```
Stratified sampling allocates samples proportional to group size by default.
## Reproducibility
Use `seed` for reproducible results:
```json theme={null}
{
"strategy": "random",
"count": 20,
"seed": 12345
}
```
Same seed + same input = same output.
## Error Handling
| Error | Behavior |
| -------------------- | -------------------- |
| count > input | Return all documents |
| Empty stratum | Skip that stratum |
| Invalid stratify\_by | Fall back to random |
| count = 0 | Return empty |
## Sample vs Other Reduction Stages
| Stage | Selection Basis | Deterministic |
| ------------- | --------------------- | ------------- |
| `sample` | Random/Stratified | With seed |
| `limit` | Position | Yes |
| `mmr` | Diversity + relevance | Yes |
| `deduplicate` | Uniqueness | Yes |
## Related
* [Aggregate](/docs/retrieval/stages/aggregate) - Statistical analysis
* [Group By](/docs/retrieval/stages/group-by) - Group before sampling
* [Cluster](/docs/retrieval/stages/cluster) - Semantic grouping
* [MMR](/docs/retrieval/stages/mmr) - Diversity-based selection
# Score Normalize
Source: https://docs.mixpeek.com/docs/retrieval/stages/score-normalize
Rescale document scores to a common range for consistent comparison
<Frame>
<img alt="Score normalize stage showing score rescaling from raw to normalized range" />
</Frame>
The Score Normalize stage rescales document scores using statistical normalization methods, enabling meaningful comparison across different scoring sources and consistent downstream thresholding.
<Note>
**Stage Category**: SORT (Rescales scores)
**Transformation**: N documents → N documents (same order, normalized scores)
</Note>
## When to Use
| Use Case | Description |
| -------------------------- | -------------------------------------------------- |
| **Hybrid search fusion** | Normalize text and vector scores before combining |
| **Score thresholding** | Set consistent cutoffs across different retrievers |
| **Cross-model comparison** | Make scores from different models comparable |
| **Probability ranking** | Convert scores to probability distribution |
| **Multi-stage pipelines** | Normalize between reranking stages |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------------- | --------------------------------- |
| Reordering by relevance | `sort_relevance` |
| Reranking with cross-encoders | `rerank` |
| Filtering by score threshold | `attribute_filter` on score field |
| Single scoring source | Scores are already comparable |
## Parameters
| Parameter | Type | Default | Description |
| -------------- | ------ | --------- | ----------------------------------------------------------- |
| `method` | string | `min_max` | Normalization method: `min_max`, `z_score`, `softmax`, `l2` |
| `score_field` | string | `score` | Field containing the score to normalize |
| `output_field` | string | `null` | Write normalized score to this field (preserves original) |
| `min_value` | float | `null` | Custom minimum for min\_max (uses actual min if null) |
| `max_value` | float | `null` | Custom maximum for min\_max (uses actual max if null) |
## Normalization Methods
| Method | Formula | Output Range | Best For |
| --------- | ----------------------- | ------------- | ------------------------ |
| `min_max` | (x - min) / (max - min) | \[0, 1] | Bounded comparison |
| `z_score` | (x - mean) / std | (-∞, +∞) | Statistical thresholding |
| `softmax` | exp(x) / Σexp | (0, 1), sum=1 | Probability distribution |
| `l2` | x / ‖x‖₂ | \[-1, 1] | Geometric comparison |
## Configuration Examples
<CodeGroup>
```json Min-Max to [0,1] theme={null}
{
"stage_name": "score_normalize",
"stage_type": "sort",
"config": {
"stage_id": "score_normalize",
"parameters": {
"method": "min_max",
"score_field": "score"
}
}
}
```
```json Z-Score with Separate Output theme={null}
{
"stage_name": "score_normalize",
"stage_type": "sort",
"config": {
"stage_id": "score_normalize",
"parameters": {
"method": "z_score",
"score_field": "score",
"output_field": "z_score"
}
}
}
```
```json Softmax Probability theme={null}
{
"stage_name": "score_normalize",
"stage_type": "sort",
"config": {
"stage_id": "score_normalize",
"parameters": {
"method": "softmax",
"score_field": "score",
"output_field": "probability"
}
}
}
```
```json Custom Range Bounds theme={null}
{
"stage_name": "score_normalize",
"stage_type": "sort",
"config": {
"stage_id": "score_normalize",
"parameters": {
"method": "min_max",
"min_value": 0.0,
"max_value": 1.0
}
}
}
```
</CodeGroup>
<Tip>
Use `output_field` to preserve the original score alongside the normalized value. This is useful for debugging or when you need both raw and normalized scores downstream.
</Tip>
## Performance
| Metric | Value |
| -------------- | ------------------------------------ |
| **Latency** | \< 1ms |
| **Memory** | O(N) for score array |
| **Cost** | Free |
| **Complexity** | O(N) (two passes: stats + normalize) |
## Common Pipeline Patterns
### Hybrid Search Fusion
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 50}],
"final_top_k": 50
}
}
},
{
"stage_name": "score_normalize",
"stage_type": "sort",
"config": {
"stage_id": "score_normalize",
"parameters": {
"method": "min_max",
"score_field": "score"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "content"
}
}
}
]
```
### Score Thresholding After Normalization
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 100}],
"final_top_k": 100
}
}
},
{
"stage_name": "score_normalize",
"stage_type": "sort",
"config": {
"stage_id": "score_normalize",
"parameters": {
"method": "min_max"
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"AND": [
{"field": "score", "operator": "gte", "value": 0.5}
]
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ------------------- | ---------------------------------------------------------- |
| Single document | min\_max returns 1.0; z\_score returns 0.0 |
| All same scores | min\_max returns 1.0 for all; z\_score returns 0.0 for all |
| Score field missing | Treated as 0.0 |
| Non-numeric score | Treated as 0.0 |
## Related
* [Sort Relevance](/docs/retrieval/stages/sort-relevance) - Reorder by relevance scores
* [Rerank](/docs/retrieval/stages/rerank) - Re-score with cross-encoder models
* [Sort Attribute](/docs/retrieval/stages/sort-attribute) - Sort by any metadata field
# Score Threshold
Source: https://docs.mixpeek.com/docs/retrieval/stages/score-threshold
Drop results below an absolute score and return no results when nothing qualifies
The Score Threshold stage applies an **absolute** quality gate: it drops every document whose score on a chosen field fails a minimum bar. When nothing clears the bar it returns an **empty** result set and sets `all_below_threshold` — the signal your UI uses to show a "no good results" state instead of presenting weak matches.
<Note>
**Stage Category**: REDUCE (N documents → ≤ N documents)
**Transformation**: keeps only documents meeting `min_score`; the set may become empty.
</Note>
## Why not just normalize and filter?
`score_normalize` with `min_max` always rescales the **top** result to `1.0` — so a threshold on the normalized score can *never* reject an all-bad result set (the best match is always 1.0). Score Threshold gates on the **raw or calibrated** score, so "everything is below the bar" is expressible. Threshold on a calibrated score — the [`rerank`](/docs/retrieval/stages/rerank) cross-encoder score is ideal (`score_field: "scores.rerank"`), since it is far more absolute and comparable than a raw cosine similarity.
## When to Use
| Use Case | Description |
| ----------------------------- | ------------------------------------------------------------------- |
| **Suppress weak matches** | Don't show results that aren't good enough to be useful |
| **"No good results" UX** | Branch to a no-results / suggestion state via `all_below_threshold` |
| **Quality gate after rerank** | Hard-gate on the calibrated cross-encoder score |
| **Confidence cutoffs** | Only surface high-confidence matches to end users |
## When NOT to Use
| Scenario | Recommended Alternative |
| -------------------------------- | -------------------------------------------------------- |
| Rescale scores for comparison | [`score_normalize`](/docs/retrieval/stages/score-normalize) |
| Keep top-N regardless of quality | [`limit`](/docs/retrieval/stages/limit) |
| Filter by metadata fields | [`attribute_filter`](/docs/retrieval/stages/attribute-filter) |
| Reorder by score | [`sort_relevance`](/docs/retrieval/stages/sort-relevance) |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------ | ------------ | -------------------------------------------------------------------------------------- |
| `min_score` | float | *(required)* | Absolute minimum score a document must meet to be kept |
| `score_field` | string | `score` | Score field to gate on. Dot-paths supported (e.g. `scores.rerank`, `metadata.quality`) |
| `comparison` | string | `gte` | Keep docs whose score is `gte` (≥) or `gt` (strictly >) `min_score` |
| `missing_score` | string | `drop` | What to do with documents lacking `score_field`: `drop` or `keep` |
## Response Metadata
| Field | Description |
| ------------------------------ | ---------------------------------------------------------- |
| `all_below_threshold` | `true` when **no** document met the bar (empty result set) |
| `input_count` / `output_count` | Documents in / kept |
| `dropped` | Documents removed |
| `missing_score_docs` | Documents that lacked `score_field` |
## Configuration Examples
<CodeGroup>
```json Gate on the rerank score theme={null}
{
"stage_name": "score_threshold",
"stage_type": "reduce",
"config": {
"stage_id": "score_threshold",
"parameters": {
"min_score": 0.5,
"score_field": "scores.rerank"
}
}
}
```
```json Strict cutoff on raw score theme={null}
{
"stage_name": "score_threshold",
"stage_type": "reduce",
"config": {
"stage_id": "score_threshold",
"parameters": {
"min_score": 0.7,
"comparison": "gt"
}
}
}
```
```json Keep docs missing the score field theme={null}
{
"stage_name": "score_threshold",
"stage_type": "reduce",
"config": {
"stage_id": "score_threshold",
"parameters": {
"min_score": 0.5,
"score_field": "scores.rerank",
"missing_score": "keep"
}
}
}
```
</CodeGroup>
## "No Good Results → Suggestions" Pattern
Gate on the rerank score; when nothing qualifies, the empty result + `all_below_threshold` tells your application to fall back to [`query_expand`](/docs/retrieval/stages/query-expand) for adjacent suggestions instead of showing weak matches.
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}}
],
"final_top_k": 50
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "content"
}
}
},
{
"stage_name": "score_threshold",
"stage_type": "reduce",
"config": {
"stage_id": "score_threshold",
"parameters": {
"min_score": 0.5,
"score_field": "scores.rerank"
}
}
}
]
```
<Tip>
Calibrate `min_score` empirically: run representative queries (including ones that *should* return nothing) and pick the value that separates good from bad on the rerank score. The right cutoff is query-domain specific.
</Tip>
## Performance
| Metric | Value |
| -------------- | ------------------ |
| **Latency** | \< 1ms |
| **Memory** | O(N) |
| **Cost** | Free |
| **Complexity** | O(N) (single pass) |
## Related
* [Score Normalize](/docs/retrieval/stages/score-normalize) - Rescale scores to a common range
* [Rerank](/docs/retrieval/stages/rerank) - Calibrated cross-encoder scores to gate on
* [Query Expand](/docs/retrieval/stages/query-expand) - Adjacent suggestions when nothing qualifies
* [Limit](/docs/retrieval/stages/limit) - Truncate to top-N regardless of score
# Sort Attribute
Source: https://docs.mixpeek.com/docs/retrieval/stages/sort-attribute
Reorder documents by any metadata field value
<Frame>
<img alt="Sort Attribute stage showing attribute-based document ordering" />
</Frame>
The Sort Attribute stage reorders documents based on metadata field values. It supports ascending/descending order, nested fields, and multiple sort criteria.
<Note>
**Stage Category**: SORT (Reorders documents)
**Transformation**: N documents → N documents (reordered by field value)
</Note>
## When to Use
| Use Case | Description |
| ------------------- | ------------------------- |
| **Price sorting** | Order products by price |
| **Recency sorting** | Most recent first |
| **Rating sorting** | Highest rated first |
| **Alphabetical** | Sort by name or title |
| **Custom ranking** | Sort by any numeric field |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------ | ---------------------------- |
| Relevance ranking | `rerank` (neural re-scoring) |
| Complex scoring logic | `api_call` to custom service |
| Already sorted by search | Skip this stage |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------ | --------------------- | ------------------------------------- |
| `field` | string | `metadata.created_at` | Field path to sort by |
| `direction` | string | `desc` | Sort direction: `asc` or `desc` |
| `null_handling` | string | `last` | Where to place nulls: `first`, `last` |
## Configuration Examples
<CodeGroup>
```json Price Low to High theme={null}
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.price",
"direction": "asc"
}
}
}
```
```json Newest First theme={null}
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.created_at",
"direction": "desc"
}
}
}
```
```json Highest Rated theme={null}
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.rating",
"direction": "desc",
"null_handling": "last"
}
}
}
```
```json Alphabetical theme={null}
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.title",
"direction": "asc"
}
}
}
```
```json Nested Field theme={null}
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.reviews.average_rating",
"direction": "desc"
}
}
}
```
```json By Relevance Score theme={null}
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "score",
"direction": "desc"
}
}
}
```
</CodeGroup>
## Field Types
The stage handles various field types:
| Type | Sort Behavior |
| ------------ | ---------------------------- |
| Number | Numeric comparison |
| String | Lexicographic (alphabetical) |
| Date/ISO8601 | Chronological |
| Boolean | false \< true |
## Null Handling
| Setting | Behavior |
| ------- | ----------------------------------------- |
| `first` | Null/missing values appear first |
| `last` | Null/missing values appear last (default) |
```json theme={null}
{
"field": "metadata.optional_field",
"direction": "desc",
"null_handling": "last"
}
```
## Performance
| Metric | Value |
| ----------------- | --------------------------- |
| **Latency** | \< 5ms |
| **Complexity** | O(n log n) |
| **Memory** | In-place sort |
| **Index support** | Uses indexes when available |
<Tip>
Sorting is very fast. Unlike `rerank`, it doesn't require model inference, making it ideal for simple ordering by attributes.
</Tip>
## Common Pipeline Patterns
### Search + Filter + Sort
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 50
}
],
"final_top_k": 50
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "metadata.in_stock",
"operator": "eq",
"value": true
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.price",
"direction": "asc"
}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"count": 10
}
}
}
]
```
### Rerank + Sort (Tie-Breaking)
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"final_top_k": 100
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 20
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.created_at",
"direction": "desc"
}
}
}
]
```
### Dynamic Sort Direction
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"final_top_k": 100
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "{{INPUT.sort_by}}",
"direction": "{{INPUT.sort_order}}"
}
}
}
]
```
## Comparison: sort\_attribute vs rerank
| Feature | sort\_attribute | rerank |
| -------------- | ------------------ | ----------------- |
| Based on | Field values | Query relevance |
| Speed | \< 5ms | 50-100ms |
| Model required | No | Yes |
| Use case | Attribute ordering | Relevance scoring |
| Cost | Free | API calls |
## Multiple Sort Criteria
For multi-field sorting, chain multiple sort stages (last sort is primary):
```json theme={null}
[
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.created_at",
"direction": "desc"
}
}
},
{
"stage_name": "sort_attribute",
"stage_type": "sort",
"config": {
"stage_id": "sort_attribute",
"parameters": {
"field": "metadata.featured",
"direction": "desc"
}
}
}
]
```
This sorts by `featured` first, then by `created_at` for ties.
## Error Handling
| Error | Behavior |
| --------------- | -------------------------- |
| Field not found | Treated as null |
| Type mismatch | String comparison fallback |
| Invalid order | Defaults to `desc` |
## Related
* [Rerank](/docs/retrieval/stages/rerank) - Neural relevance ranking
* [Sample](/docs/retrieval/stages/sample) - Reduce result count
* [Attribute Filter](/docs/retrieval/stages/attribute-filter) - Filter before sorting
# Sort Relevance
Source: https://docs.mixpeek.com/docs/retrieval/stages/sort-relevance
Reorder documents by their relevance scores from previous stages
<Frame>
<img alt="Sort Relevance stage showing document reordering by relevance scores" />
</Frame>
The Sort Relevance stage reorders documents based on their relevance scores from previous stages. This is useful when documents have been modified or filtered and need to be re-sorted by their original search scores.
<Note>
**Stage Category**: SORT (Reorders documents)
**Transformation**: N documents → N documents (reordered by relevance)
</Note>
## When to Use
| Use Case | Description |
| ----------------------- | ----------------------------------------------------- |
| **Post-filter sorting** | Re-sort after attribute\_filter removes documents |
| **Score normalization** | Apply consistent sorting after multi-stage processing |
| **Restoring order** | Return to relevance order after other sort operations |
| **Combining scores** | Sort by combined scores from multiple sources |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------- | ----------------------- |
| Sorting by metadata field | `sort_by_field` |
| Neural re-scoring | `rerank` |
| Results already sorted | Skip this stage |
| Diversifying results | `mmr` |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------ | -------- | ------------------------------------------------------------- |
| `score_field` | string | `score` | Field containing relevance score |
| `direction` | string | `desc` | Sort direction: `desc` (highest first) or `asc` |
| `missing_score` | string | `bottom` | Where docs lacking a score go: `bottom`, `top`, or `preserve` |
## Configuration Examples
<CodeGroup>
```json Basic Relevance Sort theme={null}
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": {
"direction": "desc"
}
}
}
```
```json Custom Score Field theme={null}
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": {
"score_field": "search_score",
"direction": "desc"
}
}
}
```
```json Missing Score Handling theme={null}
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": {
"score_field": "score",
"direction": "desc",
"missing_score": "bottom"
}
}
}
```
```json Ascending Order theme={null}
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": {
"score_field": "distance",
"direction": "asc"
}
}
}
```
</CodeGroup>
## How It Works
1. **Extract Scores**: Read the score field from each document
2. **Sort**: Order documents by score (descending by default)
3. **Optionally Normalize**: Scale scores to 0-1 range
4. **Return**: Documents in new order
## Output Schema
```json theme={null}
{
"document_id": "doc_123",
"content": "Document content...",
"score": 0.95,
"sort_relevance": {
"original_position": 3,
"new_position": 1,
"normalized_score": 1.0
}
}
```
## Performance
| Metric | Value |
| -------------- | ---------- |
| **Latency** | \< 5ms |
| **Memory** | O(N) |
| **Cost** | Free |
| **Complexity** | O(N log N) |
## Common Pipeline Patterns
### Search + Filter + Re-sort
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.status",
"operator": "eq",
"value": "published"
}
}
}
},
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": {
"direction": "desc"
}
}
}
]
```
### Multi-Stage with Score Normalization
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "llm_enrichment",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"model": "gpt-4o-mini",
"prompt": "Extract key topics",
"output_field": "topics"
}
}
},
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": {
"direction": "desc"
}
}
}
]
```
## Comparison with Other Sort Stages
| Stage | Purpose | Score Source |
| ---------------- | ----------------- | --------------------- |
| `sort_relevance` | Relevance scores | Search/fusion scores |
| `sort_by_field` | Metadata values | Any document field |
| `rerank` | Neural re-scoring | Cross-encoder model |
| `mmr` | Diversity | Relevance + diversity |
## Error Handling
| Error | Behavior |
| ------------------- | ----------------------- |
| Missing score field | Use 0 as default |
| Non-numeric score | Move to end |
| Empty input | Return empty |
| Equal scores | Maintain original order |
## Related
* [Rerank](/docs/retrieval/stages/rerank) - Neural re-scoring
* [Sort Attribute](/docs/retrieval/stages/sort-attribute) - Metadata sorting
* [MMR](/docs/retrieval/stages/mmr) - Diversity-aware sorting
# SQL Lookup
Source: https://docs.mixpeek.com/docs/retrieval/stages/sql-lookup
Enrich documents with data from SQL databases using parameterized queries
<Frame>
<img alt="SQL Lookup stage showing database enrichment with parameterized queries" />
</Frame>
The SQL Lookup stage enriches documents by querying external SQL databases. It executes parameterized queries using document fields as inputs, joining external structured data with your search results.
<Note>
**Stage Category**: APPLY (Enriches documents)
**Transformation**: N documents → N documents (with SQL data added)
</Note>
## When to Use
| Use Case | Description |
| ---------------------------- | --------------------------------------------- |
| **Customer data enrichment** | Join customer details from CRM database |
| **Inventory lookups** | Add real-time stock levels to product results |
| **Pricing data** | Enrich with current pricing from ERP |
| **Cross-system joins** | Combine vector search with relational data |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------- | ----------------------------------- |
| Simple key-value lookups | `document_enrich` (collection join) |
| Read-heavy, cacheable data | Consider pre-indexing in Mixpeek |
| Real-time transactional queries | Direct database access |
## Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------- | ------------ | ----------------------------------------------------------------------------------- |
| `connection_id` | string | *Required* | Database connection identifier |
| `query` | string | *Required* | SQL query with parameter placeholders |
| `parameters` | object | `{}` | Mapping of placeholder names to document fields |
| `output_field` | string | `sql_result` | Dot-path where query results are stored |
| `result_handling` | string | `first` | How to handle multiple rows (`first`, `all`, `error_if_empty`, `error_if_multiple`) |
| `on_no_results` | string | `null` | Behavior when no rows returned (`skip`, `null`, `error`) |
| `timeout` | integer | `30` | Query timeout in seconds (1-300) |
| `on_error` | string | `skip` | Error handling strategy (`skip`, `remove`, `raise`) |
## Supported Databases
| Database | Connection Type | Notes |
| ---------- | --------------- | ------------ |
| PostgreSQL | `postgres` | Full support |
| MySQL | `mysql` | Full support |
| SQL Server | `mssql` | Full support |
| SQLite | `sqlite` | Read-only |
## Configuration Examples
<CodeGroup>
```json Basic Customer Lookup theme={null}
{
"stage_name": "sql_lookup",
"stage_type": "apply",
"config": {
"stage_id": "sql_lookup",
"parameters": {
"connection_id": "crm_postgres",
"query": "SELECT name, email, tier FROM customers WHERE id = :customer_id",
"parameters": {
"customer_id": "{{DOC.metadata.customer_id}}"
},
"output_field": "customer_data"
}
}
}
```
```json Inventory Enrichment theme={null}
{
"stage_name": "sql_lookup",
"stage_type": "apply",
"config": {
"stage_id": "sql_lookup",
"parameters": {
"connection_id": "inventory_db",
"query": "SELECT stock_level, warehouse_location, last_updated FROM inventory WHERE sku = :sku",
"parameters": {
"sku": "{{DOC.metadata.sku}}"
},
"output_field": "inventory",
"timeout": 5
}
}
}
```
```json Multiple Row Results theme={null}
{
"stage_name": "sql_lookup",
"stage_type": "apply",
"config": {
"stage_id": "sql_lookup",
"parameters": {
"connection_id": "orders_db",
"query": "SELECT order_id, status, total FROM orders WHERE customer_id = :cid ORDER BY created_at DESC LIMIT 5",
"parameters": {
"cid": "{{DOC.metadata.customer_id}}"
},
"output_field": "recent_orders",
"result_handling": "all"
}
}
}
```
```json Join with Multiple Fields theme={null}
{
"stage_name": "sql_lookup",
"stage_type": "apply",
"config": {
"stage_id": "sql_lookup",
"parameters": {
"connection_id": "product_db",
"query": "SELECT price, currency, discount_pct FROM pricing WHERE product_id = :pid AND region = :region",
"parameters": {
"pid": "{{DOC.document_id}}",
"region": "{{INPUT.user_region}}"
},
"output_field": "pricing"
}
}
}
```
</CodeGroup>
## Query Syntax
### Parameter Placeholders
Use `:name` syntax for parameter placeholders:
```sql theme={null}
SELECT * FROM users WHERE id = :user_id AND status = :status
```
Parameters are properly escaped to prevent SQL injection.
### Template Variables
| Variable | Description |
| --------------- | ------------------------------------ |
| `{{DOC.*}}` | Any document field |
| `{{INPUT.*}}` | Input parameters from retriever call |
| `{{CONTEXT.*}}` | Pipeline context variables |
## Output Schema
### Single Row (default)
```json theme={null}
{
"document_id": "doc_123",
"content": "...",
"sql_result": {
"name": "John Doe",
"email": "john@example.com",
"tier": "premium"
}
}
```
### Multiple Rows
```json theme={null}
{
"document_id": "doc_123",
"content": "...",
"recent_orders": [
{"order_id": "ord_1", "status": "delivered", "total": 99.99},
{"order_id": "ord_2", "status": "shipped", "total": 149.99}
]
}
```
### No Results
```json theme={null}
{
"document_id": "doc_123",
"content": "...",
"sql_result": null
}
```
## Security
<Warning>
SQL queries are parameterized to prevent injection attacks. Never concatenate user input directly into query strings.
</Warning>
| Security Feature | Description |
| ------------------------- | ------------------------------------------ |
| **Parameterized queries** | All parameters are escaped |
| **Connection isolation** | Each connection uses dedicated credentials |
| **Read-only option** | Configure connections as read-only |
| **Query timeout** | Prevent long-running queries |
## Performance
| Metric | Value |
| ---------------------- | -------------------------------------- |
| **Latency** | 10-100ms (depends on query complexity) |
| **Connection pooling** | Automatic |
| **Parallel execution** | Up to 10 concurrent queries |
| **Timeout handling** | Graceful with null result |
<Tip>
For high-volume lookups, ensure your database has appropriate indexes on the queried columns. Consider caching frequently accessed data.
</Tip>
## Common Pipeline Patterns
### Search + SQL Enrichment
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 20 }
],
"final_top_k": 20
}
}
},
{
"stage_name": "sql_lookup",
"stage_type": "apply",
"config": {
"stage_id": "sql_lookup",
"parameters": {
"connection_id": "product_db",
"query": "SELECT price, stock FROM products WHERE id = :id",
"parameters": {
"id": "{{DOC.metadata.product_id}}"
},
"output_field": "product_data"
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "product_data.stock",
"operator": "gt",
"value": 0
}
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ------------------ | ------------------------------------------------------------------- |
| Query timeout | Handled per `on_error` (default: document passes through unchanged) |
| Connection failure | Handled per `on_error` (default: `skip`) |
| No rows returned | `output_field` set to `null` (default `on_no_results`) |
| Invalid SQL | Stage fails with error message |
## Related
* [Document Enrich](/docs/retrieval/stages/document-enrich) - Collection-based joins
* [API Call](/docs/retrieval/stages/api-call) - REST API enrichment
* [JSON Transform](/docs/retrieval/stages/json-transform) - Transform enriched data
# Summarize
Source: https://docs.mixpeek.com/docs/retrieval/stages/summarize
Generate LLM-powered summaries from document sets
<Frame>
<img alt="Summarize stage showing LLM-powered document summarization" />
</Frame>
The Summarize stage uses language models to generate summaries from document sets. It can create single summaries from multiple documents, per-document summaries, or answer questions based on the retrieved content.
<Note>
**Stage Category**: REDUCE (Aggregates documents)
**Transformation**: N documents → 1 summary document (or N documents with summaries)
</Note>
## When to Use
| Use Case | Description |
| ------------------------- | ----------------------------------------- |
| **RAG summarization** | Generate answers from search results |
| **Document synthesis** | Combine multiple sources into one summary |
| **Key points extraction** | Distill long documents to essentials |
| **Question answering** | Answer user questions from retrieved docs |
## When NOT to Use
| Scenario | Recommended Alternative |
| -------------------------- | --------------------------- |
| Just formatting for LLM | `rag_prepare` (no LLM call) |
| Extracting structured data | `llm_enrich` |
| Real-time low-latency | Pre-compute summaries |
## Parameters
| Parameter | Type | Default | Description |
| ------------------ | ------- | ------------------ | -------------------------------------------------------------------- |
| `prompt` | string | *Required* | Summarization instructions (must include `{{DOCUMENTS}}`) |
| `provider` | string | `google` | LLM provider: `openai`, `google`, `anthropic` |
| `model_name` | string | *provider default* | Specific LLM model to use |
| `content_field` | string | `content` | Field containing text to summarize |
| `group_by` | string | *none* | Field to group by (one summary per group); omit for a single summary |
| `max_input_tokens` | integer | `8000` | Max tokens to send to LLM |
| `include_sources` | boolean | `true` | Add source document IDs to output |
| `output_field` | string | `summary` | Field for summary output |
## Available Models
Set `provider` and `model_name` together. If `provider` is omitted, it is inferred from `model_name` (defaults to `google` / `gemini-2.5-flash-lite`).
| Provider | `model_name` examples | Speed | Quality |
| ----------- | --------------------------- | ------ | --------- |
| `google` | `gemini-2.5-flash-lite` | Fast | Good |
| `openai` | `gpt-4o-mini` | Fast | Good |
| `openai` | `gpt-4o` | Medium | Excellent |
| `anthropic` | `claude-haiku-4-5-20251001` | Fast | Good |
## Configuration Examples
<CodeGroup>
```json Basic RAG Summary theme={null}
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Based on the provided documents, answer the user's question: {{INPUT.query}}\n\n{{DOCUMENTS}}",
"include_sources": true
}
}
}
```
```json Detailed Summary theme={null}
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Synthesize the following documents into a comprehensive summary. Include key findings, important details, and any contradictions between sources.\n\n{{DOCUMENTS}}",
"max_input_tokens": 16000,
"include_sources": true
}
}
}
```
```json Per-Category Summaries theme={null}
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Summarize the documents for '{{GROUP_VALUE}}' in 2-3 sentences, focusing on the main points.\n\n{{DOCUMENTS}}",
"group_by": "metadata.category",
"output_field": "category_summary"
}
}
}
```
```json Executive Brief theme={null}
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "anthropic",
"model_name": "claude-haiku-4-5-20251001",
"prompt": "Create an executive summary of these documents. Include: key takeaways (3-5 bullet points), recommendations, and any risks or concerns mentioned.\n\n{{DOCUMENTS}}",
"include_sources": true
}
}
}
```
```json Q&A with Sources theme={null}
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Answer the following question based only on the provided documents. If the answer cannot be found in the documents, say so. Question: {{INPUT.question}}\n\n{{DOCUMENTS}}",
"include_sources": true,
"max_input_tokens": 12000
}
}
}
```
</CodeGroup>
## Grouping
### Single Summary (default)
With no `group_by`, all documents are combined into one summary (N→1):
```
[Doc1, Doc2, Doc3] → "Combined summary of all documents..."
```
### Per-Group Summaries
Set `group_by` to a field path to produce one summary per unique group value (N→M).
Use `{{GROUP_VALUE}}` in the prompt to reference the current group:
```
group_by: "metadata.category"
[Doc1(A), Doc2(A), Doc3(B)] → ["A" summary, "B" summary]
```
## Output Schema
The summary is written to `output_field` (default `summary`). When `include_sources` is
true, `source_document_ids` is added; when `include_metadata` is true, `document_count`
and `tokens_used` are added.
### Single Summary (no `group_by`)
```json theme={null}
{
"summary": "Based on the documents, the answer is...",
"source_document_ids": ["doc_123", "doc_456"],
"document_count": 2,
"tokens_used": 1250
}
```
### Per-Group (with `group_by`)
One summary document per unique group value:
```json theme={null}
[
{
"summary": "Summary for the electronics category...",
"source_document_ids": ["doc_123", "doc_456"],
"document_count": 2
},
{
"summary": "Summary for the clothing category...",
"source_document_ids": ["doc_789"],
"document_count": 1
}
]
```
## Performance
| Metric | Value |
| --------------- | --------------------- |
| **Latency** | 500-2000ms |
| **Token usage** | Depends on input size |
| **Max input** | Model context window |
| **Streaming** | Supported |
<Warning>
Summarization calls the LLM and incurs API costs. Use `rag_prepare` if you only need to format content for external LLM calls.
</Warning>
## Common Pipeline Patterns
### Full RAG Pipeline
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 10
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o",
"prompt": "Answer the user's question based on the provided documents: {{INPUT.query}}\n\n{{DOCUMENTS}}",
"include_sources": true
}
}
}
]
```
### Multi-Document Synthesis
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.topic}}" }, "top_k": 20 }
],
"final_top_k": 20
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.type",
"operator": "eq",
"value": "research_paper"
}
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "anthropic",
"model_name": "claude-haiku-4-5-20251001",
"prompt": "Synthesize the research findings from these papers on {{INPUT.topic}}. Identify common themes, contradictions, and gaps in the research.\n\n{{DOCUMENTS}}",
"max_input_tokens": 32000
}
}
}
]
```
### Preview Summaries
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 10 }
],
"final_top_k": 10
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"prompt": "Create a one-sentence summary for the '{{GROUP_VALUE}}' group:\n\n{{DOCUMENTS}}",
"group_by": "metadata.source",
"output_field": "preview"
}
}
}
]
```
## Comparison: summarize vs rag\_prepare
| Feature | summarize | rag\_prepare |
| --------- | ----------------- | ------------------------ |
| Calls LLM | Yes | No |
| Output | Generated summary | Formatted context |
| Latency | 500-2000ms | \< 10ms |
| Cost | LLM API costs | Free |
| Use case | End-to-end RAG | Prepare for external LLM |
## Error Handling
| Error | Behavior |
| -------------------- | -------------------------- |
| Token limit exceeded | Truncates input, continues |
| LLM timeout | Retry once, then fail |
| Rate limit | Automatic backoff |
| Empty input | Returns empty summary |
## Related
* [RAG Prepare](/docs/retrieval/stages/rag-prepare) - Format for LLM without calling
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - Structured extraction
* [Rerank](/docs/retrieval/stages/rerank) - Improve input quality first
# Taxonomy Enrich
Source: https://docs.mixpeek.com/docs/retrieval/stages/taxonomy-enrich
Classify and tag documents using predefined taxonomies
<Frame>
<img alt="Taxonomy Enrich stage showing document classification with hierarchical taxonomies" />
</Frame>
The Taxonomy Enrich stage classifies documents against predefined taxonomies, adding structured category labels and hierarchical classifications to your search results.
<Note>
**Stage Category**: ENRICH (Enriches documents with classifications)
**Transformation**: N documents → N documents (with taxonomy labels added)
</Note>
## When to Use
| Use Case | Description |
| -------------------------- | ----------------------------------- |
| **Content categorization** | Auto-classify documents into topics |
| **Faceted search** | Add filterable category facets |
| **Compliance tagging** | Apply regulatory classifications |
| **Product taxonomy** | Classify into product hierarchies |
## When NOT to Use
| Scenario | Recommended Alternative |
| --------------------------- | ---------------------------- |
| Free-form tagging | `llm_enrich` |
| Pre-classified content | Skip this stage |
| Custom classification logic | `api_call` to custom service |
## Parameters
| Parameter | Type | Default | Description |
| ------------------- | ------- | ---------- | -------------------------------- |
| `taxonomy_id` | string | *Required* | ID of the taxonomy to use |
| `content_field` | string | `content` | Field to classify |
| `result_field` | string | `taxonomy` | Field for classification results |
| `max_depth` | integer | `null` | Maximum hierarchy depth |
| `top_k` | integer | `3` | Number of top classifications |
| `min_score` | float | `0.5` | Minimum confidence threshold |
| `include_ancestors` | boolean | `true` | Include parent categories |
## Configuration Examples
<CodeGroup>
```json Basic Classification theme={null}
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "product_categories"
}
}
}
```
```json High-Confidence Only theme={null}
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "topics",
"top_k": 1,
"min_score": 0.8
}
}
}
```
```json Limit Matches Per Document theme={null}
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "industry_taxonomy",
"top_k": 3
}
}
}
```
```json Multiple Classifications theme={null}
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "content_types",
"top_k": 5,
"min_score": 0.3
}
}
}
```
</CodeGroup>
## Output Schema
### Basic Classification
```json theme={null}
{
"document_id": "doc_123",
"content": "Latest smartphone with 5G connectivity...",
"categories": {
"primary": {
"id": "electronics.mobile.smartphones",
"name": "Smartphones",
"confidence": 0.95
},
"all": [
{"id": "electronics.mobile.smartphones", "name": "Smartphones", "confidence": 0.95},
{"id": "electronics.mobile", "name": "Mobile Devices", "confidence": 0.82},
{"id": "electronics", "name": "Electronics", "confidence": 0.78}
]
}
}
```
### With Ancestors
```json theme={null}
{
"document_id": "doc_456",
"content": "Investment banking services...",
"industry": {
"primary": {
"id": "finance.banking.investment",
"name": "Investment Banking",
"confidence": 0.91
},
"ancestors": [
{"id": "finance.banking", "name": "Banking", "level": 2},
{"id": "finance", "name": "Finance", "level": 1}
],
"path": "Finance > Banking > Investment Banking"
}
}
```
### Low Confidence (No Match)
```json theme={null}
{
"document_id": "doc_789",
"content": "Random unrelated content...",
"categories": {
"primary": null,
"all": [],
"message": "No classifications above confidence threshold"
}
}
```
## Taxonomy Structure
Taxonomies are hierarchical classification systems:
```
Electronics
├── Mobile Devices
│ ├── Smartphones
│ ├── Tablets
│ └── Wearables
├── Computers
│ ├── Laptops
│ ├── Desktops
│ └── Components
└── Audio
├── Headphones
└── Speakers
```
Each node has:
* **ID**: Dot-notation path (e.g., `electronics.mobile.smartphones`)
* **Name**: Human-readable label
* **Level**: Depth in hierarchy (1 = root)
## Performance
| Metric | Value |
| ---------------------- | ------------------------------ |
| **Latency** | 10-50ms per document |
| **Batch processing** | Automatic |
| **Model type** | Embedding-based classification |
| **Parallel execution** | Up to 20 concurrent |
<Tip>
Pre-compute taxonomy embeddings for faster classification. Use `top_k: 1` and higher `min_score` when you only need the best match.
</Tip>
## Common Pipeline Patterns
### Search + Classify + Filter
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "product_categories",
"top_k": 1,
"min_score": 0.7
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "category.primary.id",
"operator": "starts_with",
"value": "{{INPUT.category_filter}}"
}
}
}
}
]
```
### Multi-Taxonomy Classification
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "topics"
}
}
},
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "sentiment"
}
}
}
]
```
### Faceted Search Results
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"final_top_k": 100
}
}
},
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "categories",
"top_k": 3
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ---------------------- | ------------------------------- |
| Unknown taxonomy\_id | Stage fails |
| No match found | Empty classification, continues |
| Invalid content\_field | Stage fails |
| Low confidence | Filtered by `min_score` |
## Related
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - Free-form extraction
* [Document Enrich](/docs/retrieval/stages/document-enrich) - Collection joins
* [Attribute Filter](/docs/retrieval/stages/attribute-filter) - Filter by categories
# Temporal
Source: https://docs.mixpeek.com/docs/retrieval/stages/temporal
Group documents by time windows and compute per-window aggregations with drift detection
<Frame>
<img alt="Temporal stage showing time-window grouping with drift detection" />
</Frame>
The Temporal stage groups documents into time windows (hour, day, week, month, quarter, year) and computes aggregations per window. It can also detect drift between consecutive windows, flagging spikes or drops that exceed a threshold.
<Note>
**Stage Category**: REDUCE (Groups results by time)
**Transformation**: N documents → M time-window results (with optional drift detection)
</Note>
## When to Use
| Use Case | Description |
| ------------------------- | -------------------------------------------------------- |
| **Trend analysis** | Track how metrics change over time |
| **Spike detection** | Flag windows where a metric jumps or drops significantly |
| **Content velocity** | Count new documents per day/week/month |
| **Temporal distribution** | Understand when content was created or modified |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------ | ----------------------- |
| Simple counting | `aggregate` |
| Full time-series from database | Aggregation API |
| Grouping by non-time fields | `group_by` |
| LLM-based trend analysis | `summarize` |
## Parameters
| Parameter | Type | Default | Description |
| ------------------- | ------- | ---------- | --------------------------------------------------------------------- |
| `time_field` | string | *Required* | Document field containing the timestamp |
| `window` | string | *Required* | Window granularity: `hour`, `day`, `week`, `month`, `quarter`, `year` |
| `aggregations` | array | *Required* | List of aggregation operations per window |
| `drift` | object | `null` | Drift detection configuration |
| `sort_order` | string | `"asc"` | Sort windows `asc` (oldest first) or `desc` (newest first) |
| `limit` | integer | `null` | Max number of windows to return |
| `include_documents` | boolean | `false` | Include original documents in output |
## Aggregation Types
| Type | Field Required | Description |
| ------------------ | -------------- | ------------------------------ |
| `count` | No | Number of documents in window |
| `sum` | Yes | Sum of field values |
| `avg` | Yes | Average value |
| `min` | Yes | Minimum value |
| `max` | Yes | Maximum value |
| `count_distinct` | Yes | Unique values count |
| `collect_distinct` | Yes | Gather unique values into list |
## Drift Detection
Drift detection compares a metric between consecutive windows and computes the percent change.
| Parameter | Type | Default | Description |
| ----------------- | ------- | --------------------- | --------------------------------------------------------- |
| `drift.enabled` | boolean | `false` | Enable drift detection |
| `drift.metric` | string | *Required if enabled* | Which aggregation alias to track |
| `drift.threshold` | float | `null` | Percent change to flag (e.g., `50.0` flags changes > 50%) |
## Timestamp Formats
The stage parses timestamps automatically:
| Format | Example |
| --------------------- | ------------------------ |
| ISO 8601 | `"2026-04-01T10:30:00Z"` |
| ISO date | `"2026-04-01"` |
| Epoch seconds (int) | `1743465600` |
| Epoch seconds (float) | `1743465600.123` |
Documents with missing or unparseable timestamps are skipped (counted in `num_documents_skipped`).
## Configuration Examples
<CodeGroup>
```json Daily Trend Analysis theme={null}
{
"stage_name": "temporal",
"stage_type": "reduce",
"config": {
"stage_id": "temporal",
"parameters": {
"time_field": "created_at",
"window": "day",
"aggregations": [
{"function": "count", "alias": "posts_per_day"},
{"function": "avg", "field": "score", "alias": "avg_relevance"}
]
}
}
}
```
```json Spike Detection with Drift theme={null}
{
"stage_name": "temporal",
"stage_type": "reduce",
"config": {
"stage_id": "temporal",
"parameters": {
"time_field": "published_at",
"window": "day",
"aggregations": [
{"function": "count", "alias": "count"}
],
"drift": {
"enabled": true,
"metric": "count",
"threshold": 50.0
}
}
}
}
```
```json Monthly Content Velocity theme={null}
{
"stage_name": "temporal",
"stage_type": "reduce",
"config": {
"stage_id": "temporal",
"parameters": {
"time_field": "created_at",
"window": "month",
"aggregations": [
{"function": "count", "alias": "total"},
{"function": "count_distinct", "field": "author", "alias": "unique_authors"},
{"function": "collect_distinct", "field": "category", "alias": "categories"}
],
"sort_order": "desc",
"limit": 12
}
}
}
```
```json Quarterly Revenue Trends theme={null}
{
"stage_name": "temporal",
"stage_type": "reduce",
"config": {
"stage_id": "temporal",
"parameters": {
"time_field": "transaction_date",
"window": "quarter",
"aggregations": [
{"function": "sum", "field": "amount", "alias": "revenue"},
{"function": "count", "alias": "transactions"},
{"function": "avg", "field": "amount", "alias": "avg_transaction"}
],
"drift": {
"enabled": true,
"metric": "revenue",
"threshold": 20.0
}
}
}
}
```
```json Hourly Activity Monitoring theme={null}
{
"stage_name": "temporal",
"stage_type": "reduce",
"config": {
"stage_id": "temporal",
"parameters": {
"time_field": "timestamp",
"window": "hour",
"aggregations": [
{"function": "count", "alias": "events"},
{"function": "max", "field": "latency_ms", "alias": "peak_latency"}
],
"drift": {
"enabled": true,
"metric": "events"
},
"sort_order": "desc",
"limit": 24
}
}
}
```
</CodeGroup>
## Output Schema
### Window Results
```json theme={null}
{
"metadata": {
"windows": [
{
"window": "2026-04-01",
"metrics": {
"count": 3,
"avg_score": 0.85
}
},
{
"window": "2026-04-02",
"metrics": {
"count": 1,
"avg_score": 0.72
},
"drift": {
"absolute_change": -2,
"percent_change": -66.67,
"flagged": true
}
}
],
"num_windows": 2,
"num_documents_in": 4,
"num_documents_skipped": 0,
"window_granularity": "day"
}
}
```
### Window Key Formats
| Window | Format | Example |
| --------- | --------------------- | --------------------- |
| `hour` | `YYYY-MM-DDTHH:00:00` | `2026-04-01T15:00:00` |
| `day` | `YYYY-MM-DD` | `2026-04-01` |
| `week` | `YYYY-WNN` | `2026-W14` |
| `month` | `YYYY-MM` | `2026-04` |
| `quarter` | `YYYY-QN` | `2026-Q2` |
| `year` | `YYYY` | `2026` |
## Performance
| Metric | Value |
| --------------- | ------------------------------- |
| **Latency** | 5-50ms |
| **Memory** | O(windows x aggregations) |
| **Cost** | Free |
| **Scalability** | Efficient for large result sets |
## Common Pipeline Patterns
### Search + Temporal Analysis
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 500
}
],
"final_top_k": 500
}
}
},
{
"stage_name": "temporal",
"stage_type": "reduce",
"config": {
"stage_id": "temporal",
"parameters": {
"time_field": "created_at",
"window": "day",
"aggregations": [
{"function": "count", "alias": "matches_per_day"},
{"function": "avg", "field": "score", "alias": "avg_relevance"}
],
"drift": {
"enabled": true,
"metric": "matches_per_day",
"threshold": 100.0
}
}
}
}
]
```
### Brand Monitoring with Spike Detection
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.brand_name}}",
"top_k": 1000
}
],
"final_top_k": 1000
}
}
},
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Classify sentiment as positive, neutral, or negative: {{DOC.content}}",
"output_field": "sentiment"
}
}
},
{
"stage_name": "temporal",
"stage_type": "reduce",
"config": {
"stage_id": "temporal",
"parameters": {
"time_field": "published_at",
"window": "week",
"aggregations": [
{"function": "count", "alias": "mentions"},
{"function": "count_distinct", "field": "sentiment", "alias": "sentiment_spread"}
],
"drift": {
"enabled": true,
"metric": "mentions",
"threshold": 50.0
}
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ----------------------------- | ------------------------------------- |
| Missing timestamp field | Document skipped |
| Unparseable timestamp | Document skipped |
| Non-numeric field for sum/avg | Document skipped for that aggregation |
| Empty results | 0 windows returned |
| Unknown aggregation | Returns null |
## Related
* [Aggregate](/docs/retrieval/stages/aggregate) - Statistical aggregations without time grouping
* [Group By](/docs/retrieval/stages/group-by) - Group documents by any field
* [Sample](/docs/retrieval/stages/sample) - Statistical sampling
# Traverse Edge
Source: https://docs.mixpeek.com/docs/retrieval/stages/traverse-edge
Follow typed relationships (edges) from documents to their linked documents
The Traverse Edge stage follows **[object edges](/docs/platform/data-model)** — typed, directed, customer-owned relationships stored on a document — to fetch the documents they point to. Starting from the documents in the pipeline, it reads each document's root-level `edges`, filters by edge type and direction, and returns the linked documents (carrying the edge's attributes with each result).
<Note>
**Stage Category**: APPLY (relationship traversal)
**Transformation**: N documents → M linked documents (optionally including the originating documents)
</Note>
Edges are saved onto documents at ingestion — for example, an assembled ad linked to every piece of footage it uses (see [Iconik project-file linkage](/docs/integrations/object-storage/iconik#project-file-linkage)). Traverse Edge is how you follow those saved relationships at query time instead of recomputing them.
## When to Use
| Use Case | Description |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **Footage ↔ ad linkage** | From an ad, fetch every footage clip it uses (with clip order + start-ticks); from footage, find the ads that use it |
| **Bill-of-materials / composition** | Follow "part-of" relationships from an assembly to its components |
| **Reference expansion** | Pull the documents a result explicitly links to, without a similarity search |
| **Graph hops** | Walk saved relationships between entities you've modeled as edges |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------------ | ----------------------- |
| Similarity / semantic matching | `feature_search` |
| Joining by a shared field value | `document_enrich` |
| Comparing content across collections | `cross_compare` |
| Filtering on an attribute | `attribute_filter` |
## How it works
1. For each document entering the stage, the stage reads its root-level `edges` list.
2. Edges are kept when their `type` matches `edge_type` and their `direction` matches the configured `direction`.
3. Each kept edge's `target_object_id` is resolved to the linked document(s); with `target_collection_id` set, only documents from that collection are returned.
4. Matched documents are returned with the originating edge's `attributes` attached (e.g. `clip_order`, `start_ticks_in`/`start_ticks_out`), so downstream stages can use them.
Because edges are stored **reciprocally** (an edge and its inverse can be written on both endpoints), you can traverse the same relationship from either side by choosing the matching `edge_type` and `direction`.
## Creating edges on objects
Edges are **customer-owned data** written at the **root** of an object when you
create it (never under `_internal`). They flow automatically from the object,
through your collections, into each document's search payload — so a stage can
follow them at query time without you re-computing anything.
Pass an `edges` array to `POST /v1/buckets/{bucket_id}/objects`:
```json theme={null}
{
"key_prefix": "ads/ad1.txt",
"blobs": [{"property": "content", "type": "text", "data": "final produced ad one"}],
"edges": [
{
"type": "uses_footage",
"target_object_id": "obj_1a2b3c...",
"direction": "out",
"attributes": {"clip_order": 3, "start_ticks_in": 123456789, "start_ticks_out": 987654321}
}
]
}
```
| Edge field | Description |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | The edge type you'll match on in `traverse_edge` (e.g. `uses_footage`). |
| `target_object_id` | The object this edge points to. It resolves to that object's document(s) at traversal time. |
| `direction` | `out` or `in` — which way the edge points. Follow it with the stage's matching `direction`. |
| `attributes` | Arbitrary key/values carried on the edge (e.g. `clip_order`, `start_ticks_in`/`start_ticks_out`). They ride into results via `traversed_via`. |
<Tip>
Write **reciprocal** edges to walk a relationship from either end: put
`uses_footage`/`out` on the ad and `used_in_ad`/`out` on the footage, each
carrying the same attributes. Connectors like [Iconik](/docs/integrations/object-storage/iconik#project-file-linkage)
populate this shape automatically from editorial project files.
</Tip>
## Parameters
| Parameter | Type | Default | Description |
| ---------------------- | ------------------- | ---------- | --------------------------------------------------------------------------------- |
| `edge_type` | string \| string\[] | *Required* | Edge type(s) to follow, e.g. `"uses_footage"` or `["uses_footage", "used_in_ad"]` |
| `direction` | string | `any` | Which edge directions to follow: `out`, `in`, or `any` |
| `target_collection_id` | string | `null` | Only return traversed documents from this collection (else all collections) |
| `include_source` | boolean | `false` | Also keep the originating documents in the output |
| `max_per_source` | integer | `50` | Max edges to follow per source document (guards fan-out) |
| `limit` | integer | `200` | Max total traversed documents to fetch across all sources |
## Example
Starting from an ad document, fetch every piece of footage it uses:
```json theme={null}
{
"stage_name": "footage_used",
"config": {
"stage_id": "traverse_edge",
"parameters": {
"edge_type": "uses_footage",
"direction": "out"
}
}
}
```
`edge_type` also accepts a **list** to follow several relationship types in a single
stage — e.g. `"edge_type": ["uses_footage", "used_in_ad"]` returns documents reached
by either edge, each still labelled with its own `traversed_via`.
Each returned footage document carries the edge attributes from the ad that referenced it — `clip_order`, `start_ticks_in`, and `start_ticks_out` — so you know exactly where in the footage each clip was taken.
## The `traversed_via` response
Every document produced by traversal carries a `traversed_via` field describing
the edge (or edges) that reached it:
| Field | Description |
| -------------------- | ------------------------------------------------------------------------------------------- |
| `edge_type` | The edge type that was followed. |
| `direction` | The direction it was followed in (`out` / `in`). |
| `attributes` | The attributes stored on that edge (e.g. `clip_order`, `start_ticks_in`/`start_ticks_out`). |
| `source_document_id` | The document the traversal started from — the other end of the edge. |
<Warning>
`traversed_via` is an **object** when a single source reached the document, but
a **list** when several sources traversed to the *same* document (a many-to-one
fan-in — the document is returned once, with one entry per incoming edge). Handle
both shapes: check whether `traversed_via` is a list before indexing it.
</Warning>
Single source (footage → ad, `used_in_ad`) — `traversed_via` is an object:
```json theme={null}
{
"document_id": "doc_9bd3572ecf180e72b7042b96",
"traversed_via": {
"edge_type": "used_in_ad",
"direction": "out",
"attributes": {"clip_order": 3, "start_ticks_in": 123456789, "start_ticks_out": 987654321},
"source_document_id": "doc_e32de934c79775b870047c62"
}
}
```
Multiple sources fan in to one document (ad → footage, `uses_footage`, where two
ads reference the same footage) — `traversed_via` is a list:
```json theme={null}
{
"document_id": "doc_0b84044df6cbd7e4f10bbcbc",
"traversed_via": [
{
"edge_type": "uses_footage",
"direction": "out",
"attributes": {"clip_order": 2, "start_ticks_in": 2000000, "start_ticks_out": 4000000},
"source_document_id": "doc_a56a844765839e4abfcdbca2"
},
{
"edge_type": "uses_footage",
"direction": "out",
"attributes": {"clip_order": 4, "start_ticks_in": 4000000, "start_ticks_out": 8000000},
"source_document_id": "doc_7547528e54cc390a0fb59b1e"
}
]
}
```
Because reciprocal edges are stored on both endpoints, the same pair is walkable
from either side — traverse `uses_footage`/`out` from an ad to reach its footage,
or `used_in_ad`/`out` from that footage to reach the ads that use it — and the
`attributes` (clip order, ticks) are identical in both directions.
## Related
* [Objects](/docs/platform/data-model) — how edges are stored on objects and flow to documents
* [Iconik integration](/docs/integrations/object-storage/iconik#project-file-linkage) — how footage↔ad edges are captured at ingestion
* [Cross Compare](/docs/retrieval/stages/cross-compare) — match content across collections by similarity instead of saved edges
# Unwind
Source: https://docs.mixpeek.com/docs/retrieval/stages/unwind
Decompose array fields into separate documents for per-element processing
<Frame>
<img alt="Unwind stage showing array decomposition into separate documents" />
</Frame>
The Unwind stage decomposes array fields into separate documents, producing one output document per array element. This is the retriever pipeline equivalent of MongoDB's `$unwind`, Snowflake's `LATERAL FLATTEN`, and Spark's `explode()`.
<Note>
**Stage Category**: APPLY (Expands documents)
**Transformation**: N documents → M documents (where M ≥ N, one per array element)
</Note>
## When to Use
| Use Case | Description |
| ------------------------- | ----------------------------------------------------- |
| **Tag expansion** | Decompose multi-tag documents for per-tag analysis |
| **Segment decomposition** | Flatten video/audio segments into individual results |
| **Author attribution** | Expand author lists for per-author scoring |
| **Chunk flattening** | Convert grouped chunks back into individual documents |
| **Category expansion** | Expand multi-category items for faceted search |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------- | ------------------------------------ |
| Filtering documents | `attribute_filter` or `llm_filter` |
| Restructuring without expansion | `json_transform` |
| Sorting documents | `sort_attribute` or `sort_relevance` |
| Grouping documents | `group_by` (inverse operation) |
## Parameters
| Parameter | Type | Default | Description |
| ------------------------- | ------- | ---------- | -------------------------------------------------------- |
| `field` | string | *required* | Dot-notation path to the array field to unwind |
| `preserve_null_and_empty` | boolean | `false` | Keep documents where array is null/missing/empty |
| `include_array_index` | string | `null` | Field name to store the element's array index |
| `output_field` | string | `null` | Place unwound element in this field instead of replacing |
## Configuration Examples
<CodeGroup>
```json Basic Tag Unwind theme={null}
{
"stage_name": "unwind",
"stage_type": "apply",
"config": {
"stage_id": "unwind",
"parameters": {
"field": "metadata.tags"
}
}
}
```
```json With Array Index Tracking theme={null}
{
"stage_name": "unwind",
"stage_type": "apply",
"config": {
"stage_id": "unwind",
"parameters": {
"field": "content.segments",
"include_array_index": "segment_index",
"preserve_null_and_empty": true
}
}
}
```
```json Output to Separate Field theme={null}
{
"stage_name": "unwind",
"stage_type": "apply",
"config": {
"stage_id": "unwind",
"parameters": {
"field": "metadata.authors",
"output_field": "current_author"
}
}
}
```
</CodeGroup>
## How It Works
1. For each input document, extracts the array value at the specified `field` path
2. If the value is an array with K elements, produces K output documents
3. Each output document preserves all original fields, with the array field replaced by a single element
4. Documents with null/empty arrays are either dropped or preserved based on `preserve_null_and_empty`
5. Non-array values are passed through unchanged
<Tip>
Use `include_array_index` when you need to reconstruct the original order later, such as when reassembling video segments after per-segment scoring.
</Tip>
## Performance
| Metric | Value |
| -------------- | ---------------------------- |
| **Latency** | \< 5ms |
| **Memory** | Proportional to output count |
| **Cost** | Free |
| **Complexity** | O(total array elements) |
## Common Pipeline Patterns
### Per-Tag Scoring
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 50}],
"final_top_k": 50
}
}
},
{
"stage_name": "unwind",
"stage_type": "apply",
"config": {
"stage_id": "unwind",
"parameters": {
"field": "metadata.tags",
"include_array_index": "tag_index"
}
}
},
{
"stage_name": "group_by",
"stage_type": "group",
"config": {
"stage_id": "group_by",
"parameters": {
"group_by_field": "metadata.tags"
}
}
}
]
```
### Segment-Level Retrieval
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 20}],
"final_top_k": 20
}
}
},
{
"stage_name": "unwind",
"stage_type": "apply",
"config": {
"stage_id": "unwind",
"parameters": {
"field": "content.segments",
"output_field": "current_segment"
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "current_segment"
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ------------------------ | --------------------------------------------------------------------------- |
| Field path doesn't exist | Document dropped (or preserved if `preserve_null_and_empty=true`) |
| Field is not an array | Document passed through unchanged |
| Empty array | Document dropped (or preserved with null if `preserve_null_and_empty=true`) |
| Null field value | Same as empty array behavior |
## Related
* [JSON Transform](/docs/retrieval/stages/json-transform) - Restructure document fields without expansion
* [Group By](/docs/retrieval/stages/group-by) - Inverse operation: group documents by field value
* [Deduplicate](/docs/retrieval/stages/deduplicate) - Remove duplicates after expansion
# Web Scrape
Source: https://docs.mixpeek.com/docs/retrieval/stages/web-scrape
Extract and parse web content from URLs using Firecrawl
<Frame>
<img alt="Web Scrape stage showing Firecrawl content extraction from URLs" />
</Frame>
The Web Scrape stage extracts content from web URLs using Firecrawl. It handles JavaScript rendering, content extraction, and structured parsing to add web content to your retrieval pipeline.
<Note>
**Stage Category**: APPLY (Enriches documents with scraped content)
**Transformation**: N documents → N documents (with web content added)
</Note>
## When to Use
| Use Case | Description |
| ----------------------- | -------------------------------------- |
| **URL enrichment** | Extract content from URLs in documents |
| **Reference expansion** | Scrape linked references for context |
| **Content aggregation** | Pull in external content sources |
| **Real-time content** | Access current webpage content |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------------ | ---------------------------- |
| Searching the web | `external_web_search` (Exa) |
| Static content already indexed | Use indexed content |
| High-volume scraping | Pre-index content in Mixpeek |
## Parameters
| Parameter | Type | Default | Description |
| ------------------ | ------- | ----------------- | --------------------------------------- |
| `url_field` | string | *Required* | Document field containing URL to scrape |
| `result_field` | string | `scraped_content` | Field for extracted content |
| `include_markdown` | boolean | `true` | Return content as markdown |
| `include_html` | boolean | `false` | Return raw HTML |
| `include_links` | boolean | `false` | Extract all page links |
| `include_images` | boolean | `false` | Extract image URLs |
| `wait_for` | integer | `0` | Wait ms for JS rendering |
| `timeout_ms` | integer | `30000` | Request timeout |
## Configuration Examples
<CodeGroup>
```json Basic URL Scraping theme={null}
{
"stage_name": "web_scrape",
"stage_type": "apply",
"config": {
"stage_id": "web_scrape",
"parameters": {
"url_field": "metadata.source_url",
"result_field": "page_content"
}
}
}
```
```json With JavaScript Rendering theme={null}
{
"stage_name": "web_scrape",
"stage_type": "apply",
"config": {
"stage_id": "web_scrape",
"parameters": {
"url_field": "metadata.url",
"result_field": "content",
"wait_for": 3000,
"include_markdown": true
}
}
}
```
```json Full Content Extraction theme={null}
{
"stage_name": "web_scrape",
"stage_type": "apply",
"config": {
"stage_id": "web_scrape",
"parameters": {
"url_field": "metadata.reference_url",
"result_field": "reference",
"include_markdown": true,
"include_links": true,
"include_images": true
}
}
}
```
```json HTML Extraction theme={null}
{
"stage_name": "web_scrape",
"stage_type": "apply",
"config": {
"stage_id": "web_scrape",
"parameters": {
"url_field": "url",
"result_field": "html_content",
"include_html": true,
"include_markdown": false,
"timeout_ms": 15000
}
}
}
```
</CodeGroup>
## Output Schema
### Markdown Output (default)
```json theme={null}
{
"document_id": "doc_123",
"metadata": {
"source_url": "https://example.com/article"
},
"scraped_content": {
"markdown": "# Article Title\n\nArticle content here...",
"title": "Article Title",
"description": "Meta description",
"language": "en",
"status": "success"
}
}
```
### Full Extraction
```json theme={null}
{
"document_id": "doc_123",
"scraped_content": {
"markdown": "# Article Title\n\n...",
"html": "<html>...</html>",
"links": [
{"text": "Link 1", "href": "https://example.com/link1"},
{"text": "Link 2", "href": "https://example.com/link2"}
],
"images": [
{"alt": "Image 1", "src": "https://example.com/img1.jpg"},
{"alt": "Image 2", "src": "https://example.com/img2.png"}
],
"title": "Article Title",
"status": "success"
}
}
```
### Error Case
```json theme={null}
{
"document_id": "doc_123",
"scraped_content": {
"status": "error",
"error": "Timeout exceeded",
"markdown": null
}
}
```
## Firecrawl Features
| Feature | Description |
| ------------------------ | ---------------------------------- |
| **JavaScript rendering** | Full browser rendering for SPAs |
| **Content extraction** | Intelligent main content detection |
| **Markdown conversion** | Clean, structured output |
| **Anti-bot handling** | Bypasses common protections |
<Tip>
Use `wait_for` when scraping JavaScript-heavy sites. Start with 2000-3000ms and adjust based on page complexity.
</Tip>
## Performance
| Metric | Value |
| ----------------------- | ---------------------------------- |
| **Latency** | 1-10s (depends on page complexity) |
| **Concurrent requests** | Up to 5 per pipeline |
| **Timeout default** | 30 seconds |
| **Retry behavior** | 2 retries on failure |
<Warning>
Web scraping adds significant latency. Use sparingly and consider pre-indexing frequently accessed content.
</Warning>
## Common Pipeline Patterns
### Enrich Documents with Referenced Content
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 10 }
],
"final_top_k": 10
}
}
},
{
"stage_name": "structured_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"field": "metadata.has_url",
"operator": "eq",
"value": true
}
}
}
},
{
"stage_name": "web_scrape",
"stage_type": "apply",
"config": {
"stage_id": "web_scrape",
"parameters": {
"url_field": "metadata.source_url",
"result_field": "source_content",
"wait_for": 2000
}
}
}
]
```
### Scrape and Summarize
```json theme={null}
[
{
"stage_name": "web_scrape",
"stage_type": "apply",
"config": {
"stage_id": "web_scrape",
"parameters": {
"url_field": "metadata.url",
"result_field": "page_content"
}
}
},
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": {
"content": "{{ DOC.page_content.markdown }}",
"source": "{{ DOC.metadata.url }}"
}
}
}
},
{
"stage_name": "summarize",
"stage_type": "reduce",
"config": {
"stage_id": "summarize",
"parameters": {
"provider": "google",
"model_name": "gemini-2.5-flash-lite",
"prompt": "Summarize the key points from this webpage"
}
}
}
]
```
## Error Handling
| Error | Behavior |
| ------------ | --------------------------------------- |
| Invalid URL | `status: "error"`, continues pipeline |
| Timeout | `status: "error"`, null content |
| 404/403 | `status: "error"`, HTTP status in error |
| Rate limited | Retry with backoff |
## Rate Limits and Best Practices
1. **Batch wisely**: Limit to 5-10 URLs per pipeline run
2. **Cache results**: Consider storing scraped content
3. **Respect robots.txt**: Firecrawl handles this automatically
4. **Use timeouts**: Set appropriate `timeout_ms` for your use case
## Related
* [External Web Search](/docs/retrieval/stages/external-web-search) - Search the web (Exa)
* [API Call](/docs/retrieval/stages/api-call) - General HTTP enrichment
* [Document Enrich](/docs/retrieval/stages/document-enrich) - Collection joins
# Buckets
Source: https://docs.mixpeek.com/docs/studio/buckets
<Frame>
<img alt="Studio Buckets page listing buckets with schema fields, object counts, and storage size" />
</Frame>
<iframe title="Buckets" />
Buckets are the warehouse's ingestion layer, where raw files enter before being decomposed into features.
### Manage buckets and objects
* **Create bucket**: Define a JSON-like schema to validate objects. Required for object registration. API parity: [Create Bucket](/docs/api-reference/buckets/create-bucket).
* **View bucket**: Inspect schema, object counts, and metadata. API parity: [Get Bucket](/docs/api-reference/buckets/get-bucket).
* **Update bucket**: Rename or adjust description and metadata. API parity: [Update Bucket](/docs/api-reference/buckets/update-bucket).
* **Delete bucket**: Enqueues a task to remove bucket and objects. Track under Tasks. API parity: [Delete Bucket](/docs/api-reference/buckets/delete-bucket).
* **Add objects**: Register raw inputs and blobs without processing. Use single or batch create. API: [Create Object](/docs/api-reference/bucket-objects/create-object), [Create Objects in Batch](/docs/api-reference/bucket-objects/create-objects-in-batch).
* **Batch flows**: Group object IDs, then submit for processing. Track task status. API: [Create Batch](/docs/api-reference/bucket-batches/create-batch), [Add Objects](/docs/api-reference/bucket-batches/add-objects-to-batch), [Submit Batch](/docs/api-reference/bucket-batches/submit-batch-for-processing).
### Tips
* Object creation does not process files; processing happens when Engine runs extraction based on collections pointing to this bucket.
* Use batch submit for large ingestions to parallelize Engine work and monitor via Tasks in Studio.
<Tabs>
<Tab title="Bucket lifecycle">
<Steps>
<Step title="Create">Define schema and metadata.</Step>
<Step title="Update">Adjust name, description, metadata.</Step>
<Step title="Delete">Enqueue task; track in Tasks.</Step>
</Steps>
</Tab>
<Tab title="Objects & batches">
<Steps>
<Step title="Register objects">Single or batch registration without processing.</Step>
<Step title="Create batch">Group object IDs for processing.</Step>
<Step title="Submit batch">Trigger Engine processing and monitor progress.</Step>
</Steps>
</Tab>
</Tabs>
<Callout type="info">
Processing runs in the Engine and writes to your active namespace; ensure it’s set correctly in the top bar.
</Callout>
# Clusters
Source: https://docs.mixpeek.com/docs/studio/clusters
<Frame>
<img alt="Studio Clusters page listing clusters with algorithm, results, and completion status" />
</Frame>
<iframe title="Clusters" />
Clusters provide warehouse-native grouping, the multimodal equivalent of SQL GROUP BY.
### Create and run clustering jobs
* **Create**: Click New Cluster, select collections, pick vector or attribute clustering, and configure algorithm params. API: [Create Cluster](/docs/api-reference/clusters/create-cluster).
* **Execute**: Run real-time clustering on the Engine or submit as a job for async processing. API: [Execute Clustering](/docs/api-reference/clusters/execute-clustering) and [Submit Job](/docs/api-reference/clusters/submit-clustering-job).
* **Inspect**: Review centroids, metrics, and members if saved. Download artifacts like parquet paths under Artifacts. API: [Get Artifacts](/docs/api-reference/clusters/get-cluster-artifacts).
* **List/Get/Delete**: Manage clustering configurations and results. API: [List](/docs/api-reference/clusters/list-clusters), [Get](/docs/api-reference/clusters/get-cluster), [Delete](/docs/api-reference/clusters/delete-cluster).
* **Stream data**: Browse cluster centroids and members directly. API: [Stream Data](/docs/api-reference/clusters/stream-cluster-data).
* **Apply enrichment**: Attach cluster labels back to a source or target collection at scale. API: [Apply Enrichment](/docs/api-reference/clusters/apply-cluster-enrichment).
<Callout type="warning">
**Choosing an algorithm at scale.** For collections over \~100K documents, pick a **linear** algorithm — **K-Means**, **Gaussian Mixture**, or **Leiden**. The density/graph algorithms that build a pairwise distance matrix — HDBSCAN, DBSCAN, Agglomerative, Spectral, OPTICS, Mean Shift — do not scale past 100K and will error on larger datasets (an N×N distance matrix at 1M rows would need \~7,000 GB of RAM). To run one of those on a large collection, set a `sample_size` to cluster a representative subset instead. The create-cluster wizard surfaces this guidance inline when you choose the algorithm.
</Callout>
### Visualization
The cluster scatter plot maps reduced coordinates to position and size:
* **x, y** → point position on the chart
* **z** (when `dimension_reduction.components` is `3`) → **dot size**, where larger dots represent higher z-values
This depth-cue approach surfaces the third dimension without requiring a full 3D renderer, making it easy to spot structure that would be lost in a flat 2D projection.
### Tips
* Start with a sample size to validate parameters before full runs.
* Use LLM labeling for human-friendly labels when vectors are dense and unlabeled.
* Set `dimension_reduction.components` to `3` to see depth-based sizing in the scatter plot.
<Steps>
<Step title="Create a cluster job">
Choose collections and configure algorithm parameters; optionally set dimensionality reduction.
</Step>
<Step title="Execute or submit">
Run in real-time or submit as an asynchronous job and track via Tasks.
</Step>
<Step title="Inspect and enrich">
Review centroids and metrics, then apply enrichment back to collections if desired.
</Step>
</Steps>
<Callout type="info">
Artifacts such as parquet paths allow downstream analytics and reproducible exploration.
</Callout>
# Collections
Source: https://docs.mixpeek.com/docs/studio/collections
<Frame>
<img alt="Studio Collections page listing collections with document counts and enabled status" />
</Frame>
<iframe title="Collections" />
Collections define how the warehouse decomposes raw files into queryable features.
### Build collections
* **Create**: Click New Collection, name it, provide a source (bucket or collection), and pick the [features](/docs/processing/features) you want (per your content's modality). API: [Create Collection](/docs/api-reference/collections/create-collection).
* **Features**: Studio shows available feature addresses and vector indexes. Use this to configure retrievers. API: [Describe Features](/docs/api-reference/collections/describe-collection-features).
* **Update**: Modify description, enablement, and taxonomy applications. API: [Update Collection](/docs/api-reference/collections/update-collection).
* **List/Get/Delete**: Browse, inspect, or remove collections. API: [List](/docs/api-reference/collections/list-collections), [Get](/docs/api-reference/collections/get-collection), [Delete](/docs/api-reference/collections/delete-collection).
### Documents
* View documents produced by extraction with metadata and vectors.
* Create or update documents directly for advanced workflows. API: [Create](/docs/api-reference/collection-documents/create-a-document), [Update](/docs/api-reference/collection-documents/update-document), [Get](/docs/api-reference/collection-documents/get-a-document-by-id), [List](/docs/api-reference/collection-documents/list-documents), [Delete](/docs/api-reference/collection-documents/delete-a-document-by-id).
### Tips
* Collections are the searchable view of your data; choose features that match your modalities.
* Ensure `internal_id` and `X-Namespace` context is correct—Studio handles this automatically.
<Steps>
<Step title="Create a collection">
Select a source, pick features, and save. Studio bootstraps vector indexes as needed.
</Step>
<Step title="Inspect features">
Use Describe Features to confirm vector names, dimensions, and index metadata for retriever setup.
</Step>
<Step title="Attach taxonomies (optional)">
Configure on-demand or materialized applications from the collection page.
</Step>
<Step title="Attach retriever enrichments (optional)">
Configure retriever enrichments to run retriever pipelines on documents during post-processing. Each enrichment maps document fields to retriever inputs, executes the retriever, and writes selected result fields back to the document. Use for LLM classification, cross-collection joins, or multi-stage enrichment at ingestion time.
</Step>
</Steps>
<Callout type="info">
Documents are stored as MVS records with payload and vectors; the record ID is the document\_id in results.
</Callout>
# Explorer
Source: https://docs.mixpeek.com/docs/studio/explorer
<iframe title="Explorer" />
### Explore and search your documents
* **Search**: Enter queries that map to your retriever inputs or perform fielded searches using filters. For retriever-backed search, use the `Research` area; Explorer provides ad-hoc discovery across collections.
* **Filters**: Apply AND/OR/NOT filters, ranges, and keyword matches on metadata. API filter model mirrors \[LogicalOperator] fields used across list and execute endpoints.
* **Sort and paginate**: Order by score or any document field and control page size. Mirrors retriever execution sort behaviour.
* **Select fields**: Toggle columns to tailor the view. Useful to inspect vector presence, enrichments, and metadata.
* **Preview assets**: Enable presigned URLs to open media directly from S3-compatible storage.
### Tips
* Switch namespaces to change the collection set you are exploring.
* For production search UX, prefer saved retrievers; Explorer is ideal for QA and data auditing.
<Callout type="info">
Explorer is optimized for ad-hoc discovery. For structured pipelines, build a retriever and test in Research.
</Callout>
<Details>
`eq`, `ne`, `gt`, `lt`, `gte`, `lte`, `in`, `nin`, `contains`, `starts_with`, `ends_with`, `regex`, `exists`, `is_null`, `text`.
</Details>
# Namespaces
Source: https://docs.mixpeek.com/docs/studio/namespaces
<Frame>
<img alt="Studio Namespaces page listing namespaces with document and object counts" />
</Frame>
<iframe title="Namespaces" />
Namespaces are isolated warehouse environments, each with its own storage, collections, and retrieval pipelines.
### Manage namespaces in Studio
* **Create**: Click New Namespace, provide a name and optional description. You can attach feature extractors and payload indexes up-front. API parity: [Create Namespace](/docs/api-reference/namespaces/create-namespace).
* **Switch active namespace**: Use the top bar to scope every page. This sets the `X-Namespace` in API calls made by Studio.
* **Update**: Edit payload indexes or rename. API parity: [Update Namespace](/docs/api-reference/namespaces/update-namespace).
* **List and search**: Quickly find namespaces you have access to. API parity: [List Namespaces](/docs/api-reference/namespaces/list-namespaces).
* **Get details**: Inspect configured feature extractors and indexes. API parity: [Get Namespace](/docs/api-reference/namespaces/get-namespace).
* **Delete**: Remove a namespace that you no longer need. This deletes the associated MVS namespace. API parity: [Delete Namespace](/docs/api-reference/namespaces/delete-namespace).
### Tips
* Namespaces map 1 to 1 with [MVS](https://mixpeek.com/mvs) namespaces. Choose clear names per environment like `acme_dev`, `acme_prod`.
* Keep one or more feature extractors attached; Engine ensures the collection is bootstrapped on first ingestion.
<Steps>
<Step title="Create a namespace">
Click New Namespace, add details, and optionally attach feature extractors and payload indexes.
</Step>
<Step title="Switch active namespace">
Use the top bar switcher; this sets the `X-Namespace` applied by Studio.
</Step>
<Step title="Manage and clean up">
Update indexes as needs evolve; delete unused namespaces to keep your workspace tidy.
</Step>
</Steps>
<Callout type="warning">
Deleting a namespace removes its backing MVS namespace for the org scope. Ensure backups as needed.
</Callout>
# Quickstart
Source: https://docs.mixpeek.com/docs/studio/quickstart
<Frame>
<img alt="Studio home showing the what-to-make-searchable picker, file drop zone, and sample-data card" />
</Frame>
<iframe title="Quickstart overview" />
Studio is the visual interface for the multimodal data warehouse, where you manage namespaces, collections, and retrievers.
### Use the Studio to manage your organization
* **Set active namespace**: Use the top bar namespace switcher. All actions scope to this selection. Namespaces mirror API behavior for `X-Namespace`. Create one in Studio or via [Create Namespace](/docs/api-reference/namespaces/create-namespace).
* **Create buckets**: Define schemas to validate your objects. Buckets are the entry point for raw data. See [Create Bucket](/docs/api-reference/buckets/create-bucket) and [Create Object](/docs/api-reference/bucket-objects/create-object).
* **Create collections**: Point to a source and pick [features](/docs/processing/features). Collections produce searchable documents and vectors. See [Create Collection](/docs/api-reference/collections/create-collection) and [Describe Features](/docs/api-reference/collections/describe-collection-features).
* **Ingest data**: Upload objects to buckets, optionally in batches. Processing runs in the Engine and writes documents to your namespace.
* **Explore results**: Use `Explorer` to search, filter, and preview documents with presigned URLs.
* **Build retrievers**: Configure stages, test inputs, and save pipelines to power your apps. See [Create Retriever](/docs/api-reference/retrievers/create-retriever) and [Execute Retriever](/docs/api-reference/retrievers/execute-retriever).
* **Enrich with taxonomies**: Attach flat or hierarchical taxonomies on-demand or materialized. See [Create Taxonomy](/docs/api-reference/taxonomies/create-taxonomy) and [Test Taxonomy](/docs/api-reference/taxonomies/test-taxonomy-configuration-validation-only).
* **Discover with clusters**: Run clustering jobs and apply enrichment back to collections. See [Create Cluster](/docs/api-reference/clusters/create-cluster) and [Execute Clustering](/docs/api-reference/clusters/execute-clustering).
### Tips
* **Auth is automatic in Studio**: Your API key and `X-Namespace` header are handled for you. When calling APIs directly, include both.
* **Rate limits and health**: Check `Operations` for service health and limits. API reference: [Healthcheck](/docs/api-reference/health/healthcheck) and [Limits](/docs/troubleshoot/limits).
<Steps>
<Step title="Set active namespace">
Use the top bar switcher. This scopes all actions and mirrors the `X-Namespace` header.
</Step>
<Step title="Create buckets">
Define schemas, then register objects. Buckets are your raw input boundary.
</Step>
<Step title="Create collections">
Pick features and index configs; this produces searchable documents and vectors.
</Step>
<Step title="Ingest and process">
Upload objects or submit batches; the Engine writes vectors and payloads to [MVS](https://mixpeek.com/mvs).
</Step>
<Step title="Explore and retrieve">
Use Explorer for ad-hoc discovery and Retrievers for production search.
</Step>
<Step title="Enrich with taxonomies and clusters">
Attach taxonomies on-demand or materialize; run clustering for discovery and analytics.
</Step>
</Steps>
<Callout type="info">
Studio follows Mintlify component patterns for clarity and scannability; see the Mintlify starter for examples.
</Callout>
# Research
Source: https://docs.mixpeek.com/docs/studio/research
<iframe title="Research" />
### Research and debug
* **Ad-hoc retrieval**: Quickly prototype queries against selected retrievers. Tweak inputs, filters, sorts, and grouping to validate pipeline behavior without code.
* **Inspect stage outputs**: Compare per-stage results, scores, and timings to tune parameters.
* **Debug inference**: Call inference backends directly to inspect embeddings and model outputs. API: [Raw Inference](/docs/api-reference/inference/execute-raw-inference).
* **Iterate**: Promote successful experiments to saved retrievers and wire them into your apps.
### Tips
* Use small limits and targeted filters while tuning to keep responses fast.
* Validate feature availability in your collections via `Describe Features` before testing embedding stages.
<Steps>
<Step title="Pick a retriever">
Select the pipeline to test and provide sample inputs.
</Step>
<Step title="Run and inspect">
Compare per-stage results, scores, and timings to tune configs.
</Step>
<Step title="Debug inference (optional)">
Call inference backends to inspect embeddings and raw outputs.
</Step>
</Steps>
<Callout type="success">
Promote stable experiments to saved retrievers to get governance and reuse across environments.
</Callout>
# Retrievers
Source: https://docs.mixpeek.com/docs/studio/retrievers
<Frame>
<img alt="Studio Retrievers page listing retrievers with their stage badges and collections" />
</Frame>
<iframe title="Retrievers" />
Retrievers are the warehouse's Reassemble layer, where you compose multi-stage pipelines to query across all your features.
### Build and execute retrievers
* **Create**: Click New Retriever, name it, select collections, and add stages from the catalog. API: [Create Retriever](/docs/api-reference/retrievers/create-retriever) and [List Stages](/docs/api-reference/retriever-stages/list-available-retriever-stages).
* **Stages**: Configure parameters per stage, including pre/post filters. Common stages include KNN search, hybrid fusion, reranking, and grouping.
* **Test**: Use the Run panel to execute with sample inputs, filters, sorts, and pagination. API parity: [Execute Retriever](/docs/api-reference/retrievers/execute-retriever).
* **Inspect**: View per-stage results, timing, and final results. Toggle URL presigning when you need asset links.
* **Manage**: List, view, update, or delete retrievers as they evolve. API: [Get](/docs/api-reference/retrievers/get-retriever), [List](/docs/api-reference/retrievers/list-retrievers), [Delete](/docs/api-reference/retrievers/delete-retriever), [Raw Inference](/docs/api-reference/inference/execute-raw-inference).
### Tips
* Ensure your collections expose the vectors you reference; confirm via `Describe Features` on the collection.
* Use `group_by` to collapse results into logical groups, then control member sorting via stage or final sort.
<Steps>
<Step title="Create a retriever">
Name it, select collections, and add stages from the catalog.
</Step>
<Step title="Configure stages">
Set parameters and optional pre/post filters; chain multiple stages for hybrid or rerank flows.
</Step>
<Step title="Test and iterate">
Run with sample inputs and inspect per-stage timing and results.
</Step>
</Steps>
<Callout type="success">
Use Debug Inference to inspect raw model outputs and embeddings when tuning parameters.
</Callout>
# Taxonomies
Source: https://docs.mixpeek.com/docs/studio/taxonomies
<Frame>
<img alt="Studio Taxonomies page listing a hierarchical taxonomy with its version and status" />
</Frame>
<iframe title="Taxonomies" />
Taxonomies provide warehouse-native enrichment, the multimodal equivalent of a SQL JOIN.
### Create and apply taxonomies
* **Create**: Click New Taxonomy, choose flat or hierarchical, and define config. API parity: [Create Taxonomy](/docs/api-reference/taxonomies/create-taxonomy).
* **Flat**: Join a source collection to a taxonomy collection via a retriever. Configure input mappings and enrichment fields.
* **Hierarchical**: Define nodes explicitly or infer via schema, clusters, or LLM. Per-node retrievers and input mappings are supported.
* **Execute on-demand**: Validate configs against sample documents. API: [Test Taxonomy](/docs/api-reference/taxonomies/test-taxonomy-configuration-validation-only). Note this is for testing only; on-demand enrichment typically happens inside retrievers.
* **Materialize**: Attach to collections with `execution_mode` set to materialize so Engine enriches after ingestion. Manage under Collections → Taxonomy Applications. API: [Update Collection](/docs/api-reference/collections/update-collection).
* **Versions**: Snapshot configurations and browse history. API: [Create Version](/docs/api-reference/taxonomies/create-taxonomy-version), [List Versions](/docs/api-reference/taxonomies/list-taxonomy-versions).
* **Manage**: Get, list, or delete taxonomies. API: [Get](/docs/api-reference/taxonomies/get-taxonomy), [List](/docs/api-reference/taxonomies/list-taxonomies), [Delete](/docs/api-reference/taxonomies/delete-taxonomy).
### Tips
* Use on-demand for interactive retrieval flows and materialize for post-ingestion enrichment.
* Keep enrichment fields minimal to avoid payload bloat; prefer app-side joins when possible.
<Tabs>
<Tab title="Flat">
<Steps>
<Step title="Create">Select retriever, input mappings, and source collection.</Step>
<Step title="Validate">Use Execute for on-demand checks with sample docs.</Step>
<Step title="Apply">Attach to a collection as on-demand or materialize.</Step>
</Steps>
</Tab>
<Tab title="Hierarchical">
<Steps>
<Step title="Define or infer">Add nodes or choose schema/cluster/LLM inference.</Step>
<Step title="Per-node options">Override retriever and input mappings as needed.</Step>
<Step title="Version">Snapshot configs and track changes over time.</Step>
</Steps>
</Tab>
</Tabs>
<Callout type="info">
Materialized enrichments are executed by the Engine post-ingestion—no separate API call is needed.
</Callout>
# LangChain
Source: https://docs.mixpeek.com/docs/agent-integrations/langchain
Give your AI agents eyes, ears, and memory with Mixpeek's LangChain integration
The `langchain-mixpeek` package gives LangChain agents the ability to see video, hear audio, search images, and act on unstructured content — all through Mixpeek's multimodal infrastructure.
<Card title="Connect LangChain to your own content" icon="rocket" href="https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=docs&utm_medium=docs_langchain&utm_campaign=managed_onramp">
Create a managed namespace to get the `api_key`, `namespace`, and `retriever_id` that every example on this page needs.
</Card>
## Installation
```bash theme={null}
pip install langchain-mixpeek
```
## Quick Start
### 1. Search (Retriever)
```python theme={null}
from langchain_mixpeek import MixpeekRetriever
retriever = MixpeekRetriever(
api_key="mxp_...",
retriever_id="ret_abc123",
namespace="my-namespace",
)
docs = retriever.invoke("find the red cup")
```
Each result is a LangChain `Document` with `page_content` and metadata (`document_id`, `score`, `namespace`).
### 2. Agent Tool
```python theme={null}
from langchain_mixpeek import MixpeekRetriever
retriever = MixpeekRetriever(
api_key="mxp_...",
retriever_id="ret_abc123",
namespace="my-namespace",
)
# One line — retriever becomes an agent tool
tool = retriever.as_tool()
```
### 3. Full Toolkit (search + ingest + classify + cluster + alert)
```python theme={null}
from langchain_mixpeek import MixpeekToolkit
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
toolkit = MixpeekToolkit(
api_key="mxp_...",
namespace="my-namespace",
bucket_id="bkt_abc123",
collection_id="col_def456",
retriever_id="ret_ghi789",
)
agent = create_react_agent(
ChatAnthropic(model="claude-sonnet-4-20250514"),
toolkit.get_tools(),
)
result = agent.invoke({
"messages": [("user", "Scan these product URLs and alert me about counterfeits")]
})
```
The toolkit gives your agent 6 capabilities:
| Tool | What it does |
| ------------------ | -------------------------------------------------------------------------- |
| `mixpeek_search` | Search video, images, audio, documents by natural language |
| `mixpeek_ingest` | Upload text, images, video, audio, PDFs, spreadsheets |
| `mixpeek_process` | Trigger feature extraction (embedding, OCR, transcription, face detection) |
| `mixpeek_classify` | Run taxonomy classification on documents |
| `mixpeek_cluster` | Group similar documents (kmeans, dbscan, hdbscan, etc.) |
| `mixpeek_alert` | Set up monitoring with webhook, Slack, or email notifications |
### 4. VectorStore (full pipeline)
```python theme={null}
from langchain_mixpeek import MixpeekVectorStore
store = MixpeekVectorStore(
api_key="mxp_...",
namespace="my-namespace",
bucket_id="bkt_abc123",
collection_id="col_def456",
retriever_id="ret_ghi789",
)
# Ingest any content type
store.add_texts(["product description..."])
store.add_images(["https://example.com/photo.jpg"])
store.add_videos(["https://example.com/clip.mp4"])
store.add_audio(["https://example.com/recording.mp3"])
store.add_pdfs(["https://example.com/doc.pdf"])
store.add_excel(["https://example.com/data.xlsx"])
# Trigger processing (embedding, OCR, face detection, etc.)
store.trigger_processing()
# Search
docs = store.similarity_search("red cup on the table")
# Convert to agent tools anytime
tool = store.as_tool()
toolkit = store.as_toolkit()
retriever = store.as_retriever()
```
### 5. Search-Only (minimal config)
If you only need search, skip the bucket/collection config:
```python theme={null}
store = MixpeekVectorStore.from_retriever(
api_key="mxp_...",
namespace="my-namespace",
retriever_id="ret_abc123",
)
docs = store.similarity_search("red cup")
```
## Configuration
| Parameter | Type | Default | Description |
| --------------- | ---- | ---------- | ------------------------------------- |
| `api_key` | str | required | Mixpeek API key (`mxp_...`) |
| `retriever_id` | str | required | Retriever ID for search (`ret_...`) |
| `namespace` | str | required | Namespace to operate in |
| `bucket_id` | str | required\* | Bucket for uploads (`bkt_...`) |
| `collection_id` | str | required\* | Collection for processing (`col_...`) |
| `top_k` | int | `10` / `5` | Max results (retriever / tool) |
| `content_field` | str | `"text"` | Field to use as `page_content` |
| `filters` | dict | `None` | Attribute filters (retriever only) |
\*Required for ingest/processing. Not needed for search-only via `from_retriever()`.
<Tip>
The `content_field` can reference any field in your retriever results — including enrichment fields like `trend_insight` or `brand_alignment`. If the field contains a dict with a `text` key, the text is automatically extracted.
</Tip>
## Examples
### Brand Protection Agent
An agent that scans marketplace listings and alerts on counterfeits:
```python theme={null}
from langchain_mixpeek import MixpeekToolkit
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
toolkit = MixpeekToolkit(
api_key="mxp_...",
namespace="brand-protection",
bucket_id="bkt_...",
collection_id="col_...",
retriever_id="ret_...",
)
# Only give the agent the tools it needs
agent = create_react_agent(
ChatAnthropic(model="claude-sonnet-4-20250514"),
toolkit.get_tools(actions=["search", "ingest", "process", "alert"]),
prompt="You are a brand protection agent. Scan products and flag counterfeits.",
)
result = agent.invoke({
"messages": [("user", "Check if these 5 Amazon listings are selling counterfeit Stanley cups")]
})
```
### RAG Chain
Standard retrieval-augmented generation:
```python theme={null}
from langchain_core.prompts import ChatPromptTemplate
from langchain_anthropic import ChatAnthropic
from langchain_mixpeek import MixpeekRetriever
retriever = MixpeekRetriever(
api_key="mxp_...",
retriever_id="ret_...",
namespace="my-namespace",
)
llm = ChatAnthropic(model="claude-sonnet-4-20250514")
prompt = ChatPromptTemplate.from_template(
"Answer using this context:\n{context}\n\nQuestion: {question}"
)
chain = {"context": retriever, "question": lambda x: x} | prompt | llm
response = chain.invoke("what happens at 2 minutes?")
```
### Multi-Retriever Agent
Different retrievers for different content types:
```python theme={null}
from langchain_mixpeek import MixpeekTool
from langgraph.prebuilt import create_react_agent
video_search = MixpeekTool(
api_key="mxp_...",
retriever_id="ret_video_archive",
namespace="archive",
name="search_video_archive",
description="Search video archive for specific scenes, faces, or moments.",
)
image_search = MixpeekTool(
api_key="mxp_...",
retriever_id="ret_product_images",
namespace="catalog",
name="search_product_images",
description="Search product image catalog by visual similarity.",
)
agent = create_react_agent(llm, [video_search, image_search])
```
## Platform Features
The VectorStore exposes the full Mixpeek platform:
### Taxonomies (document classification)
```python theme={null}
# Create a taxonomy
store.create_taxonomy(
name="product-categories",
config={
"taxonomy_type": "flat",
"retriever_id": "ret_...",
"collection_id": "col_...",
"input_mappings": [...],
"enrichment_fields": [...],
},
)
# List and execute
taxonomies = store.list_taxonomies()
results = store.execute_taxonomy("tax_abc123")
```
### Clusters (unsupervised grouping)
```python theme={null}
# Create and run clustering
cluster = store.create_cluster(
cluster_type="vector",
vector_config={
"algorithm": "kmeans", # or dbscan, hdbscan, spectral, etc.
"algorithm_params": {"n_clusters": 10},
},
)
store.execute_cluster(cluster["cluster_id"])
groups = store.get_cluster_groups(cluster["cluster_id"])
```
### Alerts (match notifications)
```python theme={null}
# Create an alert with webhook + Slack
store.create_alert(
name="counterfeit-detection",
notification_config={
"channels": [
{"channel_type": "webhook", "config": {"url": "https://..."}},
{"channel_type": "slack", "channel_id": "#alerts"},
],
"include_matches": True,
"include_scores": True,
},
)
# Check results
results = store.get_alert_results("alt_abc123")
```
### Custom Plugins
```python theme={null}
# List deployed plugins
plugins = store.list_plugins()
# Check deployment status
status = store.get_plugin_status("plg_abc123")
# Test a realtime plugin
result = store.test_plugin("plg_abc123", inputs={"text": "hello"})
```
## Tips
### Selecting Toolkit Actions
Don't give agents tools they don't need. Use `actions` to scope:
```python theme={null}
# Search-only agent
toolkit.get_tools(actions=["search"])
# Ingest + search agent
toolkit.get_tools(actions=["search", "ingest", "process"])
# Full platform agent
toolkit.get_tools() # all 6 tools
```
### Error Handling
All toolkit tools catch exceptions and return error strings instead of crashing the agent. The retriever raises exceptions normally.
### Token Efficiency
Set `top_k` to limit results. Large result sets waste tokens without improving quality. Start with `top_k=5`.
## Source Code
* **PyPI**: [langchain-mixpeek](https://pypi.org/project/langchain-mixpeek/) (Python)
* **npm**: [@mixpeek/langchain](https://www.npmjs.com/package/@mixpeek/langchain) (JavaScript)
* **GitHub**: [mixpeek/langchain-mixpeek](https://github.com/mixpeek/langchain-mixpeek)
* **LangChain Docs**: [Tools](https://docs.langchain.com/oss/python/integrations/tools/mixpeek) · [Vector Store](https://docs.langchain.com/oss/python/integrations/vectorstores/mixpeek)
* **Connector Page**: [mixpeek.com/connectors/langchain](https://mixpeek.com/connectors/langchain)
## Next Steps
<CardGroup>
<Card title="MCP Server" icon="plug" href="/agent-integrations/mcp">
Connect Claude directly via the Model Context Protocol
</Card>
<Card title="OpenAI Function Calling" icon="robot" href="/agent-integrations/openai-function-calling">
Wire Mixpeek into OpenAI assistants
</Card>
<Card title="Feature Extractors" icon="wand-magic-sparkles" href="/processing/feature-extractors">
15+ extractors: text, image, video, audio, face, PDF, web scraper
</Card>
<Card title="Python SDK" icon="python" href="/integrations/developer-tools/python-sdk">
Full SDK reference
</Card>
</CardGroup>
# MCP Server
Source: https://docs.mixpeek.com/docs/agent-integrations/mcp
Give AI agents access to video, image, and audio search through the Model Context Protocol
Connect Claude (or any MCP-compatible client) to Mixpeek so it can search video, image, and audio content.
<Frame>
<img alt="Mixpeek MCP server architecture: an MCP client connects over HTTPS with a Bearer API key to the scope-gated Mixpeek MCP server (48 tools), which exposes retrievers as tools — agent_search for autonomous multimodal retrieval, retriever_execute for multi-stage filter/join/rerank pipelines, features_get for faces, scenes, OCR, transcripts and embeddings, and agentic_enrich (ingestion scope, off by default) — operating on a Mixpeek namespace of video, images, audio and documents stored as features plus dense, sparse and BM25 vectors on object storage." />
</Frame>
## Hosted Server URLs
Mixpeek runs four hosted MCP servers. Each exposes a different subset of tools so you only load what your agent needs.
| Scope | URL | Tools |
| ------------- | --------------------------------------- | -------------------------------------- |
| **Full** | `https://mcp.mixpeek.com/mcp` | 48 -- everything |
| **Ingestion** | `https://mcp.mixpeek.com/ingestion/mcp` | 20 -- buckets, collections, documents |
| **Retrieval** | `https://mcp.mixpeek.com/retrieval/mcp` | 11 -- retrievers, agents, search |
| **Admin** | `https://mcp.mixpeek.com/admin/mcp` | 17 -- namespaces, taxonomies, clusters |
<Tip>
For a focused search agent, use the [Per-Retriever Server](#per-retriever-mcp-server) instead. It exposes a single typed `search` tool generated from your retriever's input schema.
</Tip>
***
## Setup
Replace `YOUR_API_KEY` with your key from the [Mixpeek dashboard](https://mixpeek.com/dashboard).
<Tabs>
<Tab title="Claude Desktop">
Add this to your Claude Desktop config file (`claude_desktop_config.json`):
<CodeGroup>
```json Full theme={null}
{
"mcpServers": {
"mixpeek": {
"url": "https://mcp.mixpeek.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```json Ingestion theme={null}
{
"mcpServers": {
"mixpeek-ingestion": {
"url": "https://mcp.mixpeek.com/ingestion/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```json Retrieval theme={null}
{
"mcpServers": {
"mixpeek-retrieval": {
"url": "https://mcp.mixpeek.com/retrieval/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```json Admin theme={null}
{
"mcpServers": {
"mixpeek-admin": {
"url": "https://mcp.mixpeek.com/admin/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
</CodeGroup>
</Tab>
<Tab title="Claude Code">
Run this in your terminal:
<CodeGroup>
```bash Full theme={null}
claude mcp add mixpeek \
--transport streamable-http \
--url https://mcp.mixpeek.com/mcp \
--header "Authorization: Bearer YOUR_API_KEY"
```
```bash Ingestion theme={null}
claude mcp add mixpeek-ingestion \
--transport streamable-http \
--url https://mcp.mixpeek.com/ingestion/mcp \
--header "Authorization: Bearer YOUR_API_KEY"
```
```bash Retrieval theme={null}
claude mcp add mixpeek-retrieval \
--transport streamable-http \
--url https://mcp.mixpeek.com/retrieval/mcp \
--header "Authorization: Bearer YOUR_API_KEY"
```
```bash Admin theme={null}
claude mcp add mixpeek-admin \
--transport streamable-http \
--url https://mcp.mixpeek.com/admin/mcp \
--header "Authorization: Bearer YOUR_API_KEY"
```
</CodeGroup>
</Tab>
</Tabs>
***
## Per-Retriever MCP Server
The retriever server is a lightweight MCP server scoped to a **single retriever**. It reads your retriever's `input_schema` at startup and generates a typed `search` tool whose parameters match exactly.
<Steps>
<Step title="Install">
```bash theme={null}
pip install mixpeek-mcp-retriever
```
</Step>
<Step title="Run">
```bash theme={null}
mixpeek-mcp-retriever \
--retriever-id ret_xxx \
--namespace-id ns_xxx \
--api-key YOUR_API_KEY
```
</Step>
<Step title="Connect">
Add to your Claude Desktop config:
```json theme={null}
{
"mcpServers": {
"my-search": {
"command": "mixpeek-mcp-retriever",
"args": [
"--retriever-id", "ret_xxx",
"--namespace-id", "ns_xxx",
"--api-key", "YOUR_API_KEY"
]
}
}
}
```
</Step>
</Steps>
The retriever server exposes three tools:
| Tool | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `search` | Execute the retriever. Parameters are generated from the retriever's `input_schema` -- field names, types, required flags, enums, and descriptions. Pagination (`page`, `page_size`) is added automatically. |
| `describe` | Returns structured metadata: retriever ID, name, collections, input fields, and stage configuration. |
| `explain` | Returns a human-readable explanation of the pipeline: what each stage does, in order. |
<Note>
If your retriever's `input_schema` has a field named `page` or `page_size`, the pagination parameters are automatically renamed to `_pagination_page` and `_pagination_page_size` to avoid conflicts.
</Note>
You can also run the retriever server over HTTP for deployed agents:
```bash theme={null}
mixpeek-mcp-retriever \
--retriever-id ret_xxx \
--namespace-id ns_xxx \
--api-key YOUR_API_KEY \
--transport http \
--port 8081
```
Or configure everything via environment variables (prefixed with `RETRIEVER_MCP_`):
```bash theme={null}
export RETRIEVER_MCP_RETRIEVER_ID=ret_xxx
export RETRIEVER_MCP_NAMESPACE_ID=ns_xxx
export RETRIEVER_MCP_API_KEY=YOUR_API_KEY
export RETRIEVER_MCP_TRANSPORT=stdio
mixpeek-mcp-retriever
```
***
## Available Tools by Scope
### Ingestion -- 18 tools
| Tool | Description |
| -------------------- | -------------------------------------------- |
| `create_bucket` | Create a new bucket for file storage |
| `list_buckets` | List all buckets in a namespace |
| `get_bucket` | Get bucket details |
| `update_bucket` | Update bucket configuration |
| `delete_bucket` | Delete a bucket and its objects |
| `upload_object` | Upload a file to a bucket from a URL |
| `create_collection` | Create a collection with a feature extractor |
| `list_collections` | List all collections in a namespace |
| `get_collection` | Get collection details |
| `update_collection` | Update collection configuration |
| `clone_collection` | Clone a collection with optional overrides |
| `trigger_collection` | Trigger processing on bucket objects |
| `delete_collection` | Delete a collection and its documents |
| `create_document` | Create a document in a collection |
| `list_documents` | List documents with filters |
| `get_document` | Get a document by ID |
| `update_document` | Update a document |
| `delete_document` | Delete a document |
### Retrieval -- 11 tools
| Tool | Description |
| ---------------------- | -------------------------------------------------- |
| `create_retriever` | Create a multi-stage search pipeline |
| `list_retrievers` | List all retrievers in a namespace |
| `get_retriever` | Get retriever configuration and stages |
| `update_retriever` | Update retriever metadata |
| `clone_retriever` | Clone a retriever with modifications |
| `execute_retriever` | Execute a retriever and get search results |
| `delete_retriever` | Delete a retriever |
| `create_agent_session` | Create a conversational agent session |
| `send_agent_message` | Send a message and get a retriever-backed response |
| `get_agent_history` | Get conversation history |
| `search_namespace` | Search across all resources in a namespace |
### Admin -- 14 tools
| Tool | Description |
| ------------------ | --------------------------------------------- |
| `create_namespace` | Create a workspace |
| `list_namespaces` | List all namespaces |
| `get_namespace` | Get namespace details |
| `update_namespace` | Update namespace configuration |
| `delete_namespace` | Delete a namespace and all its resources |
| `create_taxonomy` | Create a hierarchical classification taxonomy |
| `list_taxonomies` | List all taxonomies |
| `get_taxonomy` | Get taxonomy details |
| `execute_taxonomy` | Apply taxonomy classification to documents |
| `delete_taxonomy` | Delete a taxonomy |
| `create_cluster` | Create a document clustering configuration |
| `list_clusters` | List all clusters |
| `execute_cluster` | Run clustering on a collection |
| `delete_cluster` | Delete a cluster configuration |
***
## Authentication
All MCP servers authenticate with your Mixpeek API key. The key carries the same RBAC permissions as the REST API.
* **Hosted servers (HTTP):** Pass the key in the `Authorization: Bearer YOUR_API_KEY` header. The server injects it into every tool call.
* **Stdio servers:** Set the `MIXPEEK_API_KEY` environment variable, or pass `--api-key` as a CLI argument.
<Warning>
Never commit API keys to version control. For deployed agents, use environment variables or a secrets manager.
</Warning>
***
## Example Conversation
Here is what a session looks like with the Retrieval server connected:
```
You: "Find all video clips where someone mentions quarterly revenue"
Claude: [calls execute_retriever with query="quarterly revenue"]
--> Returns 8 matching video segments with timestamps and transcript excerpts.
You: "Summarize the top 3 results"
Claude: [calls execute_retriever with query="quarterly revenue", then processes results]
--> "1. Q3 earnings call (2:34) -- CFO reports 22% YoY growth.
2. Board presentation (14:12) -- Revenue breakdown by region.
3. Team standup (0:45) -- Quick mention of hitting quarterly target."
You: "Create a retriever that searches product images by description and filters by brand"
Claude: [calls create_retriever with feature_search + attribute_filter stages]
--> "Created retriever ret_abc123 with 2 stages: feature search on image
embeddings, then attribute filter on the brand field."
```
And with the per-retriever server:
```
You: "What does this retriever do?"
Claude: [calls describe]
--> "This is 'Product Search' -- searches your products collection
by text query with an optional category filter, using
feature search -> attribute filter -> reranking."
You: "Search for wireless headphones under electronics"
Claude: [calls search with query="wireless headphones", category="electronics"]
--> Returns top 10 matching products with scores and metadata.
You: "Explain the pipeline"
Claude: [calls explain]
--> "1. feature_search: embeds your query and finds the top 50 matches.
2. attribute_filter: filters by category.
3. rerank: reranks with Cohere down to the top 10."
```
***
## Architecture
```
+-----------------------------------------------------------+
| Claude / AI Agent |
+-----------------------------+-----------------------------+
| MCP Protocol (HTTP / Stdio)
v
+-----------------------------------------------------------+
| Mixpeek MCP Server |
| +--------+ +-----------+ +-----------+ +--------+ |
| | Full | | Ingestion | | Retrieval | | Admin | |
| | (43) | | (18) | | (11) | | (14) | |
| +---+----+ +-----+-----+ +-----+-----+ +---+----+ |
| +------------+-------------+-----------+ |
| Tool Handlers |
+-----------------------------+-----------------------------+
| +--------------------+
| | Retriever Server |
| | (per-retriever, 3) |
| +---------+----------+
+--------+--------+
| API calls
v
+----------------------+
| Mixpeek Platform |
+----------------------+
```
The scoped servers share the same codebase. Scoping controls which tools are registered, not how they execute. Each scope is mounted at its own path prefix (`/ingestion`, `/retrieval`, `/admin`) while the full server runs at `/full`.
***
## Troubleshooting
<AccordionGroup>
<Accordion title="Claude can't connect to the MCP server">
* Verify the URL is correct (e.g. `https://mcp.mixpeek.com/ingestion/mcp`)
* Check that the `Authorization` header format is `Bearer YOUR_API_KEY`
* Restart Claude Desktop or Claude Code after changing config
</Accordion>
<Accordion title="Tools return 'Unauthorized' or 'Invalid API key'">
* Verify your API key at [mixpeek.com/dashboard](https://mixpeek.com/dashboard)
* Check that the key has permissions for the namespace you're accessing
* Make sure there are no extra spaces in the key
</Accordion>
<Accordion title="Tool not found (404)">
* You may be calling a tool on the wrong scoped server (e.g. `execute_retriever` on `/ingestion`)
* Use the full server if you need all tools
</Accordion>
<Accordion title="Retriever Server fails to start">
* Ensure `--retriever-id` and `--namespace-id` are correct
* Verify the API key has access to that namespace
* Check that the retriever exists via the API
</Accordion>
<Accordion title="Search returns empty results">
* Confirm your collection has processed documents (not just uploaded files)
* Check that the retriever's `feature_uri` matches your collection's extractor
* Try a broader query or remove optional filters
</Accordion>
</AccordionGroup>
## Next Steps
<CardGroup>
<Card title="LangChain Integration" icon="link" href="/agent-integrations/langchain">
Use Mixpeek as a LangChain tool
</Card>
<Card title="Retriever Stages" icon="filter" href="/retrieval/stages/overview">
Configure multi-stage search pipelines
</Card>
<Card title="Feature Extractors" icon="microchip" href="/processing/feature-extractors">
Choose the right extractor for your data
</Card>
<Card title="Core Concepts" icon="book" href="/overview/concepts">
Understand namespaces, collections, and documents
</Card>
</CardGroup>
# OpenAI Function Calling
Source: https://docs.mixpeek.com/docs/agent-integrations/openai-function-calling
Wire Mixpeek retrievers into OpenAI's function calling API so GPT models can search multimodal content
Wire Mixpeek retrievers into OpenAI's function calling API so GPT models can search video, image, and audio content on demand.
<Note>
There is no `langchain-mixpeek` package. You use the standard `mixpeek` SDK and wrap it as an OpenAI function. This gives you full control over input parsing, error handling, and response formatting.
</Note>
## The Pattern
OpenAI function calling lets GPT models decide when to invoke external tools during a conversation. You define a function schema, register it as a tool, and handle the call in your completion loop.
<Steps>
<Step title="Define the function schema">
Describe `search_mixpeek` with a name, description, and parameters so the model knows when and how to call it.
</Step>
<Step title="Register it as a tool">
Pass the schema in the `tools` array when calling `chat.completions.create()`.
</Step>
<Step title="Handle tool calls in the completion loop">
When the model returns `tool_calls`, execute `client.retrievers.execute()` with the provided arguments and append the results as a tool message.
</Step>
<Step title="Get the final response">
Call `chat.completions.create()` again with the tool results. The model incorporates the search results into its answer.
</Step>
</Steps>
## Installation
<CodeGroup>
```bash Python theme={null}
pip install mixpeek openai
```
```bash JavaScript theme={null}
npm install mixpeek openai
```
</CodeGroup>
## Function Schema
Define a function that tells GPT what Mixpeek search does and what inputs it accepts:
```json theme={null}
{
"type": "function",
"function": {
"name": "search_mixpeek",
"description": "Search across video, image, and audio content indexed in Mixpeek. Use this when the user asks about visual content, media files, or multimedia information.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Natural language search query describing what to find"
},
"limit": {
"type": "integer",
"description": "Maximum number of results to return (default 5)"
}
},
"required": ["query"]
}
}
}
```
<Tip>
Write a specific `description` that tells the model *when* to call this tool. Mention the content types your retriever handles (video, images, audio). Generic descriptions like "search for things" cause the model to over- or under-use the function.
</Tip>
## Full Working Example: Chat Completions API
<CodeGroup>
```python Python theme={null}
import json
from openai import OpenAI
from mixpeek import Mixpeek
openai_client = OpenAI(api_key="YOUR_OPENAI_KEY")
mixpeek_client = Mixpeek(api_key="YOUR_MIXPEEK_KEY")
# Define the tool
tools = [
{
"type": "function",
"function": {
"name": "search_mixpeek",
"description": (
"Search across video, image, and audio content. "
"Use when the user asks about visual content, media files, "
"or multimedia information."
),
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Natural language search query",
},
"limit": {
"type": "integer",
"description": "Max results to return (default 5)",
},
},
"required": ["query"],
},
},
}
]
def execute_search(query: str, limit: int = 5) -> str:
"""Call the Mixpeek retriever and return results as JSON."""
results = mixpeek_client.retrievers.execute(
retriever_id="ret_abc123",
inputs={"query": query},
namespace="my-namespace",
)
# Trim to limit and keep only essential fields for token efficiency
trimmed = [
{
"document_id": doc["document_id"],
"score": doc["score"],
"metadata": doc.get("metadata", {}),
}
for doc in results[:limit]
]
return json.dumps(trimmed, indent=2)
def chat(user_message: str):
messages = [
{
"role": "system",
"content": "You help users find and analyze multimedia content.",
},
{"role": "user", "content": user_message},
]
# First call -- model may request a tool call
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
)
message = response.choices[0].message
# Handle tool calls
if message.tool_calls:
messages.append(message)
for tool_call in message.tool_calls:
args = json.loads(tool_call.function.arguments)
result = execute_search(
query=args["query"],
limit=args.get("limit", 5),
)
messages.append(
{
"role": "tool",
"tool_call_id": tool_call.id,
"content": result,
}
)
# Second call -- model generates final answer with results
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
)
message = response.choices[0].message
return message.content
print(chat("Find clips where someone mentions the product launch"))
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
import Mixpeek from "mixpeek";
const openai = new OpenAI({ apiKey: "YOUR_OPENAI_KEY" });
const mixpeek = new Mixpeek({ apiKey: "YOUR_MIXPEEK_KEY" });
const tools = [
{
type: "function",
function: {
name: "search_mixpeek",
description:
"Search across video, image, and audio content. " +
"Use when the user asks about visual content, media files, " +
"or multimedia information.",
parameters: {
type: "object",
properties: {
query: {
type: "string",
description: "Natural language search query",
},
limit: {
type: "integer",
description: "Max results to return (default 5)",
},
},
required: ["query"],
},
},
},
];
async function executeSearch(query, limit = 5) {
const results = await mixpeek.retrievers.execute({
retrieverId: "ret_abc123",
inputs: { query },
namespace: "my-namespace",
});
const trimmed = results.slice(0, limit).map((doc) => ({
document_id: doc.document_id,
score: doc.score,
metadata: doc.metadata || {},
}));
return JSON.stringify(trimmed, null, 2);
}
async function chat(userMessage) {
const messages = [
{ role: "system", content: "You help users find and analyze multimedia content." },
{ role: "user", content: userMessage },
];
let response = await openai.chat.completions.create({
model: "gpt-4o",
messages,
tools,
});
let message = response.choices[0].message;
if (message.tool_calls) {
messages.push(message);
for (const toolCall of message.tool_calls) {
const args = JSON.parse(toolCall.function.arguments);
const result = await executeSearch(args.query, args.limit || 5);
messages.push({
role: "tool",
tool_call_id: toolCall.id,
content: result,
});
}
response = await openai.chat.completions.create({
model: "gpt-4o",
messages,
tools,
});
message = response.choices[0].message;
}
return message.content;
}
console.log(await chat("Find clips where someone mentions the product launch"));
```
</CodeGroup>
## Assistants API
You can register Mixpeek search as a tool on an OpenAI Assistant. The Assistants API manages conversation state and persistent threads for you.
```python theme={null}
import json
import time
from openai import OpenAI
from mixpeek import Mixpeek
openai_client = OpenAI(api_key="YOUR_OPENAI_KEY")
mixpeek_client = Mixpeek(api_key="YOUR_MIXPEEK_KEY")
# Create an assistant with the Mixpeek tool
assistant = openai_client.beta.assistants.create(
name="Media Search Assistant",
instructions=(
"You help users search and analyze video, image, and audio content. "
"Use the search_mixpeek tool whenever the user asks about media files."
),
model="gpt-4o",
tools=[
{
"type": "function",
"function": {
"name": "search_mixpeek",
"description": (
"Search across video, image, and audio content indexed in Mixpeek."
),
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Natural language search query",
},
"limit": {
"type": "integer",
"description": "Max results to return (default 5)",
},
},
"required": ["query"],
},
},
}
],
)
# Create a thread and send a message
thread = openai_client.beta.threads.create()
openai_client.beta.threads.messages.create(
thread_id=thread.id,
role="user",
content="Find video frames showing the CEO on stage at the keynote",
)
# Start a run
run = openai_client.beta.threads.runs.create(
thread_id=thread.id,
assistant_id=assistant.id,
)
# Poll until complete, handling tool calls
while True:
run = openai_client.beta.threads.runs.retrieve(
thread_id=thread.id, run_id=run.id
)
if run.status == "requires_action":
tool_outputs = []
for tool_call in run.required_action.submit_tool_outputs.tool_calls:
args = json.loads(tool_call.function.arguments)
results = mixpeek_client.retrievers.execute(
retriever_id="ret_abc123",
inputs={"query": args["query"]},
namespace="my-namespace",
)
trimmed = [
{
"document_id": doc["document_id"],
"score": doc["score"],
"metadata": doc.get("metadata", {}),
}
for doc in results[: args.get("limit", 5)]
]
tool_outputs.append(
{
"tool_call_id": tool_call.id,
"output": json.dumps(trimmed),
}
)
run = openai_client.beta.threads.runs.submit_tool_outputs(
thread_id=thread.id,
run_id=run.id,
tool_outputs=tool_outputs,
)
elif run.status == "completed":
break
elif run.status in ("failed", "cancelled", "expired"):
print(f"Run ended with status: {run.status}")
break
else:
time.sleep(1)
# Get the assistant's response
messages = openai_client.beta.threads.messages.list(thread_id=thread.id)
print(messages.data[0].content[0].text.value)
```
## Tips
### Write Descriptive Function Schemas
The `description` field on your function and its parameters directly affects when and how the model calls your tool. Be specific about what content types your retriever handles.
```python theme={null}
# Good -- tells the model exactly when to use it
"Search video frames, audio transcripts, and image content in the media library. Returns timestamped results with relevance scores."
# Bad -- too vague, model won't know when to call it
"Search for stuff in the database."
```
### Limit Result Size for Token Efficiency
Every result you return gets added to the conversation context. Strip unnecessary fields and cap the number of results to avoid wasting tokens.
```python theme={null}
def execute_search(query: str, limit: int = 5) -> str:
results = mixpeek_client.retrievers.execute(
retriever_id="ret_abc123",
inputs={"query": query},
namespace="my-namespace",
)
# Return only what the model needs to formulate an answer
trimmed = [
{
"id": doc["document_id"],
"score": round(doc["score"], 3),
"title": doc.get("metadata", {}).get("title", ""),
"summary": doc.get("metadata", {}).get("description", "")[:200],
}
for doc in results[:limit]
]
return json.dumps(trimmed)
```
### Handle Errors Gracefully
Return error messages as tool output instead of raising exceptions. This lets the model recover or ask the user to rephrase.
```python theme={null}
def execute_search(query: str, limit: int = 5) -> str:
try:
results = mixpeek_client.retrievers.execute(
retriever_id="ret_abc123",
inputs={"query": query},
namespace="my-namespace",
)
trimmed = [
{"id": doc["document_id"], "score": doc["score"]}
for doc in results[:limit]
]
return json.dumps(trimmed)
except Exception as e:
return json.dumps({"error": str(e), "suggestion": "Try a different query."})
```
<Note>
If you use `parallel_tool_calls` (enabled by default in the Chat Completions API), the model may issue multiple `search_mixpeek` calls in a single response. The handler loop in the examples above already supports this -- each tool call is processed independently.
</Note>
## Next Steps
<CardGroup>
<Card title="LangChain" icon="link" href="/agent-integrations/langchain">
Use Mixpeek as a LangChain tool for agent workflows
</Card>
<Card title="MCP Server" icon="plug" href="/agent-integrations/mcp">
Connect Claude directly via the Model Context Protocol
</Card>
<Card title="Retriever Stages" icon="filter" href="/retrieval/stages/overview">
Configure multi-stage search pipelines
</Card>
<Card title="Python SDK" icon="python" href="/integrations/developer-tools/python-sdk">
Full SDK reference
</Card>
</CardGroup>
# Whoami
Source: https://docs.mixpeek.com/docs/api-reference/account/whoami
get /v1/me
Return the caller's identity: organization, tier, key metadata, limits.
Works for every valid key. `/account` is an undocumented alias so both
natural spellings resolve.
# Create Api Key
Source: https://docs.mixpeek.com/docs/api-reference/organization-api-keys/create-api-key
post /v1/organizations/users/{user_email}/api-keys
Create a new API key for a user.
# List Api Keys
Source: https://docs.mixpeek.com/docs/api-reference/organization-api-keys/list-api-keys
get /v1/organizations/users/{user_email}/api-keys
List API keys for a user.
# Update Api Key
Source: https://docs.mixpeek.com/docs/api-reference/organization-api-keys/update-api-key
patch /v1/organizations/users/{user_email}/api-keys/{key_name}
Update an API key's metadata or permissions.
🔒 The "admin-key" is protected and cannot be modified.
# Create User
Source: https://docs.mixpeek.com/docs/api-reference/organization-users/create-user
post /v1/organizations/users
Create a new organization user.
# Delete User
Source: https://docs.mixpeek.com/docs/api-reference/organization-users/delete-user
delete /v1/organizations/users/{user_email}
Delete a user and revoke their API keys.
# Get User
Source: https://docs.mixpeek.com/docs/api-reference/organization-users/get-user
get /v1/organizations/users/{user_email}
Return a user by email address.
# List Users
Source: https://docs.mixpeek.com/docs/api-reference/organization-users/list-users
get /v1/organizations/users
List organization users with pagination and optional filters.
# Update User
Source: https://docs.mixpeek.com/docs/api-reference/organization-users/update-user
patch /v1/organizations/users/{user_email}
Apply partial updates to an existing user.
# Add Credits
Source: https://docs.mixpeek.com/docs/api-reference/organizations/add-credits
post /v1/organizations/credits
Add credits to the organization.
When credits are added to a FREE-tier organization:
- If new balance >= 100,000: Auto-upgrade to PRO tier
- If new balance >= 1,000,000: Auto-upgrade to TEAM tier
PRO and TEAM tiers get enhanced rate limits automatically.
# Get Organization
Source: https://docs.mixpeek.com/docs/api-reference/organizations/get-organization
get /v1/organizations
Get current organization details.
Security: Infrastructure configuration is NOT exposed via this endpoint.
Infrastructure (Qdrant URLs, Ray clusters) is only accessible via private admin endpoints.
# Get Organization Targets
Source: https://docs.mixpeek.com/docs/api-reference/organizations/get-organization-targets
get /v1/organizations/targets
Get this organization's resource + SLO targets.
Tenancy (dedicated vs shared) is resolved SERVER-SIDE from the org —
never from a client-supplied header (BACKE-2564). ``targets_writable``
is the capability flag a client should read rather than infer from
``plane``: shared-plane orgs are always read-only by platform policy;
dedicated orgs may write once the write path ships (not this
endpoint).
# Update Organization
Source: https://docs.mixpeek.com/docs/api-reference/organizations/update-organization
patch /v1/organizations
Update organization settings (requires ADMIN permission).
Security: Infrastructure configuration cannot be modified via this endpoint.
Infrastructure updates require Mixpeek admin access via private endpoints.
# Apps
Source: https://docs.mixpeek.com/docs/canvas/apps
Deploy your own frontend code — React, vanilla JS, or any static site — connected to your Mixpeek retrievers. Served on your custom domain with auth built in.
<Note>
Canvas Apps require an **Enterprise** plan. [Contact us](https://mixpeek.com/contact) to upgrade.
</Note>
## Overview
Canvas is Mixpeek's **application hosting platform**. Build a web app using any frontend framework — React, Vue, Svelte, vanilla JS, or plain HTML — and deploy it as a zip bundle. Mixpeek handles hosting, CDN, auth, versioning, and the entire backend.
Your app is live at `{slug}.mxp.co` the moment you deploy. No servers to manage, no infrastructure to configure, no API keys in your frontend code.
### How it works
You write frontend code. Mixpeek provides everything else:
* **Hosting** — your built assets (HTML, JS, CSS, images) are stored on S3 and served via CDN with immutable caching
* **Backend proxy** — all `/api/*` requests are forwarded to `api.mixpeek.com` with your org's API key and namespace injected server-side. Your credentials never reach the browser.
* **Runtime injection** — the Canvas runtime injects `window.__MIXPEEK__` into your HTML at serve time with auth config, environment variables, and monitoring hooks
* **Auth** — optionally enable Clerk-powered sign-in (Google, GitHub, email) with per-app user management. The auth SDK is auto-injected — no libraries to install.
* **Environments** — deploy to staging (`staging-{slug}.mxp.co`) or production (`{slug}.mxp.co`) independently
* **Versioning** — every deploy creates an immutable version with content hash, commit message, and source files. Rollback instantly.
* **Monitoring** — error boundaries, Sentry, and PostHog are auto-injected for crash detection and usage analytics
You bring the UI, Mixpeek brings search, retrieval, ingestion, and multimodal AI.
<CardGroup>
<Card title="Deploy from code" icon="upload" href="/canvas/apps/deploy">
Upload any React, vanilla JS, or static site. Mixpeek serves it from S3 with CDN caching.
</Card>
<Card title="Custom domains" icon="globe" href="/canvas/apps/domains">
Default domain at `{slug}.mxp.co`. Add your own subdomain via CNAME with auto TLS.
</Card>
<Card title="Built-in auth" icon="lock" href="/canvas/apps/authentication">
Clerk-powered sign-in (Google, GitHub, email) — auto-injected into your app with zero config.
</Card>
<Card title="User management" icon="users" href="/canvas/apps/users">
Invite members, assign roles, and control per-app access — powered by Clerk organizations.
</Card>
<Card title="Git-like version control" icon="code-branch" href="/canvas/apps/versions">
Every publish and deploy creates an immutable version with a content hash, message, and diffable snapshot.
</Card>
</CardGroup>
***
## Architecture
When a user visits `{slug}.mxp.co`, here's the request lifecycle:
1. **DNS** — resolves to the Canvas runtime (Cloudflare-proxied, DDoS protected)
2. **Routing** — the Express server maps the hostname to your `app_id` (cached in Redis). Supports `{slug}.mxp.co`, `staging-{slug}.mxp.co`, and custom domains.
3. **HTML** — `index.html` is fetched from S3 and dynamically injected with `window.__MIXPEEK__` (runtime config, auth, monitoring)
4. **Assets** — JS, CSS, and images are served from S3 via CDN with immutable cache headers (content-hashed filenames → permanent caching)
5. **SPA routing** — non-asset paths fall through to `index.html` so client-side routing (React Router, etc.) works out of the box
6. **API proxy** — `/api/*` requests are forwarded to `api.mixpeek.com` with `Authorization` and `X-Namespace` headers injected server-side
<Note>
Your API key is **never** sent to the browser. The `/api` proxy runs server-side and injects credentials on every request. Your frontend code only calls relative paths like `/api/v1/retrievers/execute`.
</Note>
### Runtime config injection
The canvas runtime injects a `window.__MIXPEEK__` object into your app's HTML at serve time:
```javascript theme={null}
window.__MIXPEEK__ = {
apiUrl: "https://api.mixpeek.com", // Always present
// Your custom env vars from build_config.env_vars
MY_CUSTOM_VAR: "value",
// Auth config (if enabled)
auth: {
mode: "clerk",
publishableKey: "pk_live_...",
orgId: "org_xyz",
providers: ["google", "github", "email"],
},
}
```
Access these values in your app code:
```jsx theme={null}
const apiUrl = window.__MIXPEEK__?.apiUrl
const customVar = window.__MIXPEEK__?.MY_CUSTOM_VAR
```
<Warning>
Do not use `process.env` in your app — it will crash in the browser. Use `window.__MIXPEEK__` for runtime config or `import.meta.env` for Vite build-time variables.
</Warning>
***
## Quickstart
<Steps>
<Step title="Create an App in Studio">
Go to **Apps** → **Create App**. Give it a name and a globally-unique slug — this becomes your default URL at `https://{slug}.mxp.co`.
</Step>
<Step title="Build your frontend">
Write a React app (or any static site) that calls Mixpeek APIs via the canvas proxy. See the [example below](#canvas-sdk).
```bash theme={null}
# Bootstrap a React app
npm create vite@latest my-search-app -- --template react
cd my-search-app
npm install
npm run build
# Zip the dist/ folder
zip -r my-search-app.zip dist/
```
</Step>
<Step title="Deploy the bundle">
In the App details page, drag & drop your `.zip` onto the Deploy panel and click **Deploy**. Your build is queued immediately.
Or via API:
```bash theme={null}
# 1. Get a presigned upload URL
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy/upload-url \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"filename": "my-app.zip", "content_type": "application/zip"}'
# 2. PUT the zip to the returned upload_url
curl -X PUT "$UPLOAD_URL" \
-H "Content-Type: application/zip" \
--data-binary @my-app.zip
# 3. Trigger the build
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"source": "cli_upload", "bundle_s3_key": "$BUNDLE_KEY", "environment": "production", "message": "Initial deploy"}'
```
</Step>
<Step title="Visit your App">
Your app is live at `https://{slug}.mxp.co` within seconds of a successful build. Add a custom domain to use your own URL.
</Step>
</Steps>
***
## Canvas SDK
The canvas runtime injects credentials **server-side** — your API key never reaches the browser. Call any Mixpeek API through the `/api` proxy:
```jsx theme={null}
// src/App.jsx — a minimal React search app
import { useState } from 'react'
async function search(query) {
// No auth headers needed — the canvas proxy injects them server-side
const res = await fetch('/api/v1/retrievers/ret_abc123/execute', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
inputs: { query },
settings: { limit: 20 },
}),
})
return res.json()
}
export default function App() {
const [query, setQuery] = useState('')
const [results, setResults] = useState([])
const [loading, setLoading] = useState(false)
const handleSearch = async (e) => {
e.preventDefault()
setLoading(true)
const data = await search(query)
setResults(data.documents ?? [])
setLoading(false)
}
return (
<div style={{ maxWidth: 800, margin: '0 auto', padding: 24 }}>
<h1>Search</h1>
<form onSubmit={handleSearch}>
<input
value={query}
onChange={(e) => setQuery(e.target.value)}
placeholder="Search..."
style={{ width: '100%', padding: 8, fontSize: 16 }}
/>
<button type="submit" disabled={loading}>
{loading ? 'Searching...' : 'Search'}
</button>
</form>
<ul>
{results.map((r) => (
<li key={r.document_id}>
<strong>{r.fields?.title ?? r.document_id}</strong>
<p>{r.fields?.description}</p>
</li>
))}
</ul>
</div>
)
}
```
### What `/api` supports
The proxy forwards **all** Mixpeek API methods with your org's credentials injected:
| Category | Example paths |
| --------------- | ------------------------------------------------------------- |
| **Retrievers** | `/api/v1/retrievers/{id}/execute`, `/api/v1/retrievers/list` |
| **Collections** | `/api/v1/collections/list`, `/api/v1/collections/{id}` |
| **Documents** | `/api/v1/documents/{id}`, `/api/v1/documents/list` |
| **Namespaces** | `/api/v1/namespaces/list` |
| **Batches** | `/api/v1/batches/list`, `/api/v1/batches/{id}` |
| **Tasks** | `/api/v1/tasks/{id}`, `/api/v1/tasks/list` |
| **Marketplace** | `/api/v1/marketplace/catalog/{name}/execute` |
| **Taxonomies** | `/api/v1/taxonomies/list`, `/api/v1/taxonomies/{id}/classify` |
<Note>
Use `/api/v1/...` (relative path) instead of `https://api.mixpeek.com/v1/...` — the canvas proxy injects `Authorization` and `X-Namespace` headers automatically, avoiding CORS and keeping API keys out of your bundle.
</Note>
### Using the Mixpeek JS SDK
You can also use the `mixpeek` npm package pointed at `/api`:
```jsx theme={null}
import { Mixpeek } from 'mixpeek'
// apiKey is a placeholder — the proxy injects the real key server-side
const client = new Mixpeek({ apiKey: 'canvas', baseUrl: '/api' })
// Execute a retriever
const results = await client.request({
method: 'POST',
url: '/v1/retrievers/ret_abc123/execute',
data: { inputs: { query: 'red shoes' }, settings: { limit: 10 } },
})
// Execute a marketplace retriever
const marketplace = await client.request({
method: 'POST',
url: '/v1/marketplace/catalog/brand-safety/execute',
data: { inputs: { query: 'test content' }, settings: { limit: 5 } },
})
```
### Billing
All API calls through `/api` are billed to the organization that owns the canvas app. End-users don't need their own Mixpeek API keys or accounts — usage is attributed to your org automatically.
***
## Environments
Each app supports two independent environments:
| Environment | URL | Use case |
| -------------- | ----------------------- | ------------------------------ |
| **Staging** | `staging-{slug}.mxp.co` | Test changes before going live |
| **Production** | `{slug}.mxp.co` | Live, user-facing deployment |
Each environment has its own S3 asset prefix, so staging and production can serve different versions simultaneously. Deploy to either environment from Studio or via the API.
### Deploying to staging
Set `"environment": "staging"` in the deploy request. Your staging build is live at `https://staging-{slug}.mxp.co`.
### Promoting to production
Deploy the same bundle to production, or use the restore endpoint to point production at a staging version's assets:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/versions/$STAGING_VERSION/restore \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"environment": "production"}'
```
***
## Creating an App
### Via Studio
Go to **Apps** → **Create App**. The wizard collects:
1. **Name + Slug** — the name is display-only; the slug becomes your URL (`{slug}.mxp.co`) and must be globally unique
2. **Access Control** — choose how end-users authenticate (public, password, API key, JWT, or SSO)
### Via API
Only `slug` and `meta` are required. After creation, deploy your code via the deploy pipeline.
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
app = client.apps.create(
slug="product-search",
meta={"title": "Product Search"},
auth_config={"mode": "clerk"}, # optional — enable Clerk auth
)
print(f"App created: {app.app_id}")
print(f"URL: https://{app.slug}.mxp.co")
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/apps \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"slug": "product-search",
"meta": { "title": "Product Search" },
"auth_config": { "mode": "clerk" }
}'
```
</CodeGroup>
***
## Deploy lifecycle
Every deploy creates a new versioned build. The lifecycle is:
| Stage | Description |
| ---------- | --------------------------------------------------------------- |
| `queued` | Bundle uploaded, build queued |
| `building` | Mixpeek is packaging and deploying your bundle |
| `complete` | Build complete — app is live |
| `failed` | Build failed — previous version stays live, check error message |
### Operations
| Operation | What it does | Creates a version? |
| ------------------------------------------------------- | ------------------------------------------ | ------------------------------------- |
| **Deploy** (`POST /v1/apps/{id}/deploy`) | Upload and serve a new code bundle | Yes — with asset manifest and message |
| **Restore** (`POST /v1/apps/{id}/versions/{v}/restore`) | Roll back to any previous version's assets | No — instant redirect |
| **Promote** (`POST /v1/apps/{id}/promote`) | Promote staging deploy to production | No — swaps environments |
| **Update** (`PATCH /v1/apps/{id}`) | Update app config (auth, meta, etc.) | No |
Each deploy requires a commit message describing what changed. See [Version History](/docs/canvas/apps/versions) for the full version control system.
***
## Authentication modes
| Mode | Description |
| ---------- | ------------------------------------------------------------------------ |
| `public` | Anyone can access — no login required |
| `password` | Shared password gate |
| `api_key` | Requires `X-App-Key` header |
| `clerk` | Clerk-powered sign-in with [per-app user management](/docs/canvas/apps/users) |
| `jwt` | Customer-managed tokens |
| `sso_oidc` | Okta, Auth0, Azure AD |
| `sso_saml` | Enterprise SAML 2.0 IdP |
<Note>
Auth enforcement activates when the canvas runtime is deployed. Config is stored now and takes effect automatically.
</Note>
***
## Custom domains
Every app gets a default URL at `https://{slug}.mxp.co`. You can also add your own subdomain:
1. Enter your subdomain (e.g., `search.yourdomain.com`) in the **Domains** panel
2. Add a DNS `CNAME` record pointing to the target shown in the response
3. Click **Verify** — Mixpeek provisions a TLS certificate automatically via Cloudflare
See [Custom Domains](/docs/canvas/apps/domains) for the full setup guide.
***
## Example apps
<CardGroup>
<Card title="E-commerce product search" icon="bag-shopping">
React app with multimodal search — text queries, image upload, and faceted filters backed by a Mixpeek retriever.
</Card>
<Card title="Internal knowledge base" icon="book">
Password-protected semantic search over company documents, PDFs, and meeting transcripts.
</Card>
<Card title="Media library" icon="photo-film">
Video + image search portal with scene-level results, thumbnails, and timestamp previews.
</Card>
<Card title="Content moderation dashboard" icon="shield-check">
Real-time moderation queue pulling from alert webhooks and showing flagged content with similarity scores.
</Card>
</CardGroup>
***
## Related
* [Version History](/docs/canvas/apps/versions)
* [Deploy from Code](/docs/canvas/apps/deploy)
* [Authentication](/docs/canvas/apps/authentication)
* [User Management](/docs/canvas/apps/users)
* [Custom Domains](/docs/canvas/apps/domains)
* [Create App (API)](/docs/api-reference/apps/create-app)
* [Deploy App (API)](/docs/api-reference/apps/deploy-app)
# Authentication
Source: https://docs.mixpeek.com/docs/canvas/apps/authentication
Add Clerk-based sign-in to your Canvas app with zero configuration. Users authenticate via Google, GitHub, or email — the SDK is auto-injected.
## Overview
Canvas apps support built-in authentication powered by Clerk. When you enable auth, your users can sign up and sign in via Google, GitHub, or email/password — with no extra libraries or configuration on your part.
The auth SDK is **auto-injected** into your app at runtime. You get a `window.MixpeekAuth` object with methods to check sign-in state, show modals, and access user profiles.
***
## Enabling Auth
Enable Clerk authentication by updating your app's `auth_config`:
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
client.apps.update(
app_id="app_abc123",
auth_config={"mode": "clerk"},
)
```
```javascript JavaScript theme={null}
import { Mixpeek } from 'mixpeek-sdk'
const client = new Mixpeek({ apiKey: 'your-api-key' })
await client.apps.update({
appId: 'app_abc123',
authConfig: { mode: 'clerk' },
})
```
```bash cURL theme={null}
curl -X PATCH https://api.mixpeek.com/v1/apps/$APP_ID \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"auth_config": {"mode": "clerk"}}'
```
</CodeGroup>
Setting mode to `"clerk"` automatically:
* Injects `window.__MIXPEEK__` with Clerk configuration
* Loads the `/_auth/sdk.js` script into your app
* Enforces authentication on all `/api` proxy requests
<Note>
You do not need to install or import any auth libraries. The canvas runtime handles everything.
</Note>
***
## Using MixpeekAuth in Your App
The SDK exposes `window.MixpeekAuth` with the following API:
| Property / Method | Type | Description |
| ------------------------------- | -------------- | ------------------------------------- |
| `MixpeekAuth.onReady(callback)` | `(fn) => void` | Called when auth is initialized |
| `MixpeekAuth.isSignedIn` | `boolean` | Whether the current user is signed in |
| `MixpeekAuth.user` | `object` | `{ id, email, name, avatar_url }` |
| `MixpeekAuth.showSignIn()` | `() => void` | Opens the Clerk sign-in modal |
| `MixpeekAuth.showSignUp()` | `() => void` | Opens the Clerk sign-up modal |
| `MixpeekAuth.signOut()` | `() => void` | Signs out and reloads the page |
### React Example
```jsx theme={null}
// src/App.jsx — auth-gated Canvas app
import { useState, useEffect } from 'react'
export default function App() {
const [user, setUser] = useState(null)
const [ready, setReady] = useState(false)
useEffect(() => {
window.MixpeekAuth.onReady(() => {
setReady(true)
if (window.MixpeekAuth.isSignedIn) {
setUser(window.MixpeekAuth.user)
}
})
}, [])
if (!ready) return <p>Loading...</p>
if (!user) {
return (
<div>
<h1>Welcome</h1>
<p>Sign in to access this app.</p>
<button onClick={() => window.MixpeekAuth.showSignIn()}>
Sign In
</button>
<button onClick={() => window.MixpeekAuth.showSignUp()}>
Sign Up
</button>
</div>
)
}
return (
<div>
<h1>Hello, {user.name}</h1>
<p>{user.email}</p>
<button onClick={() => window.MixpeekAuth.signOut()}>
Sign Out
</button>
{/* Your app content here */}
</div>
)
}
```
***
## Auth Endpoints
The canvas runtime exposes three auth endpoints on your app's domain:
| Endpoint | Method | Description |
| --------------- | ------ | --------------------------------------------------------------- |
| `/_auth/me` | GET | Returns the current user's profile |
| `/_auth/users` | GET | Lists all users for this app (requires auth) |
| `/_auth/sdk.js` | GET | The auth SDK script (auto-loaded, but you can load it manually) |
<Tip>
Use `/_auth/me` from your frontend to verify auth state on page load without relying solely on the client-side SDK.
</Tip>
***
## API Access and Billing
All `/api` requests are authenticated with the **app owner's API key** — your end-users never need their own Mixpeek credentials.
* Usage is billed to the organization that owns the Canvas app
* The `/api` proxy supports all Mixpeek API methods: retrievers, collections, documents, and marketplace
* End-users interact with your app; your API key handles the backend calls transparently
<Warning>
Because all API usage is billed to your organization, monitor your usage in Studio to avoid unexpected costs from high-traffic apps.
</Warning>
***
## User Storage
When users sign up through your app, their metadata (email, name, avatar) is stored automatically. Each app maintains its own user list, scoped independently from other apps.
Access user data via the `/_auth/users` endpoint:
```bash theme={null}
curl https://your-app.mxp.co/_auth/users \
-H "Authorization: Bearer $API_KEY"
```
Response:
```json theme={null}
[
{
"id": "user_abc123",
"email": "jane@example.com",
"name": "Jane Smith",
"avatar_url": "https://..."
}
]
```
***
## Auth Providers
By default, Clerk authentication supports:
* **Google** — OAuth sign-in
* **GitHub** — OAuth sign-in
* **Email / password** — standard credentials
Configure allowed providers via `auth_config.clerk_allowed_providers`:
<CodeGroup>
```python Python theme={null}
client.apps.update(
app_id="app_abc123",
auth_config={
"mode": "clerk",
"clerk_allowed_providers": ["google", "github"],
},
)
```
```bash cURL theme={null}
curl -X PATCH https://api.mixpeek.com/v1/apps/$APP_ID \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"auth_config": {
"mode": "clerk",
"clerk_allowed_providers": ["google", "github"]
}
}'
```
</CodeGroup>
<Note>
Omit `clerk_allowed_providers` to enable all default providers (Google, GitHub, and email/password).
</Note>
***
## Related
* [User Management](/docs/canvas/apps/users) — invite members, assign roles, remove users
* [Apps overview](/docs/canvas/apps)
* [Deploy from Code](/docs/canvas/apps/deploy)
* [Custom Domains](/docs/canvas/apps/domains)
# Build Logs
Source: https://docs.mixpeek.com/docs/canvas/apps/build-logs
Stream real-time build output during deploys and diagnose failures with persistent log history.
## Overview
Every Canvas deploy produces a detailed build log — from downloading your bundle through validation, asset upload, and completion. Logs stream in real time via Server-Sent Events (SSE) and persist in the deploy record for later review.
***
## Streaming logs
Subscribe to live build output during a deploy:
```bash theme={null}
curl -N https://api.mixpeek.com/v1/apps/$APP_ID/deploys/$DEPLOY_ID/logs/stream \
-H "Authorization: Bearer $API_KEY"
```
The response is an SSE stream (`text/event-stream`). Each event contains a JSON log entry:
```
data: {"ts": 1711741200.123, "line": "Downloading bundle from S3..."}
data: {"ts": 1711741201.456, "line": "Extracting bundle..."}
data: {"ts": 1711741202.789, "line": "Validating bundle (index.html, size)..."}
data: {"ts": 1711741203.012, "line": "Uploading 14 assets to S3..."}
data: {"ts": 1711741205.345, "line": "Deploy complete!"}
event: done
data: {}
```
### Consuming in JavaScript
```javascript theme={null}
const response = await fetch(
`https://api.mixpeek.com/v1/apps/${appId}/deploys/${deployId}/logs/stream`,
{ headers: { Authorization: `Bearer ${apiKey}` } }
)
const reader = response.body.getReader()
const decoder = new TextDecoder()
while (true) {
const { done, value } = await reader.read()
if (done) break
const text = decoder.decode(value)
for (const line of text.split('\n')) {
if (line.startsWith('data: ')) {
const entry = JSON.parse(line.slice(6))
console.log(entry.line)
}
}
}
```
<Note>
Use `fetch` + `ReadableStream` instead of `EventSource` — fetch supports `Authorization` headers, while `EventSource` does not.
</Note>
***
## Reconnecting mid-deploy
If you disconnect and reconnect, the SSE endpoint replays all previously emitted log lines (from MongoDB) before switching to live streaming. No log lines are lost on reconnect.
***
## Log lifecycle
| Deploy stage | Log lines emitted |
| ------------ | -------------------------------------------------------------- |
| `queued` | None yet |
| `building` | Real-time build progress (download, extract, validate, upload) |
| `complete` | All lines + "Deploy complete!" |
| `failed` | All lines + "Deploy failed: " |
After a deploy reaches a terminal state (`complete` or `failed`), the full log is persisted in the deploy record and can be retrieved at any time.
***
## Git deploy logs
Deploys triggered via GitHub push include additional log lines from the build process:
```
Cloning https://github.com/org/repo (branch: main)...
Installing dependencies (npm ci)...
npm warn deprecated ...
Building project (npm run build)...
vite v5.4.2 building for production...
✓ 42 modules transformed.
Zipping dist/...
Uploading bundle to S3...
Deploy complete!
```
Real `npm ci` and `npm run build` output is streamed line-by-line, so you see the same output you'd see locally.
***
## Studio
In the App details page, the **Deploy** panel shows a terminal-style log viewer during active deploys. Logs auto-scroll as new lines arrive and remain viewable after completion.
***
## Related
* [Deploy from Code](/docs/canvas/apps/deploy)
* [GitHub Integration](/docs/canvas/apps/github)
* [CLI: `mixpeek apps logs`](/docs/canvas/apps/cli)
# CLI
Source: https://docs.mixpeek.com/docs/canvas/apps/cli
Deploy, stream logs, manage previews, and run a local dev server from the command line with @mixpeek/cli.
## Overview
The Mixpeek CLI (`@mixpeek/cli`) lets you deploy Canvas apps, stream build logs, list preview environments, and run a local dev server — all from your terminal.
***
## Installation
```bash theme={null}
npm install -g @mixpeek/cli
```
***
## Authentication
```bash theme={null}
mixpeek login
```
This stores your API key locally for use in subsequent commands. You can also pass `--api-key` to any command.
***
## Commands
### `mixpeek apps deploy`
Build your app locally and deploy it to Canvas.
```bash theme={null}
cd my-app
mixpeek apps deploy --app-id $APP_ID
```
What it does:
1. Detects your package manager (npm, yarn, pnpm)
2. Runs `npm run build` (or equivalent)
3. Zips the `dist/` output directory
4. Uploads the bundle to Mixpeek
5. Triggers a deploy
Options:
| Flag | Description | Default |
| --------------- | ------------------------------ | -------------------- |
| `--app-id` | App ID to deploy to | Required |
| `--environment` | Target environment | `production` |
| `--message` | Commit message for the version | Auto-generated |
| `--api-key` | API key (overrides stored key) | From `mixpeek login` |
<Note>
The CLI builds `dist/` (not `src/`). Make sure your `build` script outputs to `dist/`, `build/`, or `out/`.
</Note>
***
### `mixpeek apps logs`
Stream real-time build logs for an active deploy.
```bash theme={null}
mixpeek apps logs --app-id $APP_ID --deploy-id $DEPLOY_ID
```
Log lines are printed with timestamps and colored output. The stream closes automatically when the deploy completes or fails.
***
### `mixpeek apps preview`
List active preview deployments for an app.
```bash theme={null}
mixpeek apps preview --app-id $APP_ID
```
Output:
```
Preview Environments:
preview-42 → https://pr-42-my-app.mxp.co
preview-17 → https://pr-17-my-app.mxp.co
```
***
### `mixpeek apps dev`
Start a local development server with Mixpeek environment variables pre-configured.
```bash theme={null}
mixpeek apps dev --app-id $APP_ID
```
This spawns your project's dev server (e.g., `npm run dev`) with `VITE_MIXPEEK_API_URL` set, so API proxy calls work locally.
***
## CI/CD integration
Use the CLI in GitHub Actions or any CI pipeline:
```yaml theme={null}
# .github/workflows/deploy.yml
name: Deploy to Canvas
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npm install -g @mixpeek/cli
- run: mixpeek apps deploy --app-id ${{ secrets.APP_ID }} --api-key ${{ secrets.MIXPEEK_API_KEY }} --message "Deploy from CI (${{ github.sha }})"
```
***
## Related
* [Deploy from Code](/docs/canvas/apps/deploy)
* [Build Logs](/docs/canvas/apps/build-logs)
* [Preview Deploys](/docs/canvas/apps/preview-deploys)
# Deploy from Code
Source: https://docs.mixpeek.com/docs/canvas/apps/deploy
Upload a zip of your frontend code and Mixpeek builds and hosts it automatically. Deploy to staging or production with full versioning.
## Overview
Deploy any React app, vanilla JS site, or static bundle to a Canvas app. Upload a `.zip` of your build output and Mixpeek handles hosting, CDN distribution, and versioning — no infrastructure needed.
Every deploy creates an immutable version record with a commit message, asset manifest, and content hashes. See [Version History](/docs/canvas/apps/versions) for the full version control system.
***
## Quickstart
<Steps>
<Step title="Build your frontend">
Build your app to a static output directory.
```bash theme={null}
# React / Vite
npm run build
# Output: dist/
# Next.js (static export)
npm run build && npm run export
# Output: out/
```
</Step>
<Step title="Zip the output directory">
```bash theme={null}
zip -r my-app.zip dist/
```
The zip should contain an `index.html` at the root (or inside a single top-level folder). Mixpeek auto-detects the entry point.
</Step>
<Step title="Get a presigned upload URL">
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy/upload-url \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"filename": "my-app.zip", "content_type": "application/zip"}'
```
Response:
```json theme={null}
{
"upload_url": "https://s3.us-east-2.amazonaws.com/...",
"bundle_s3_key": "app-builds/<app_id>/<token>/my-app.zip",
"expires_in": 3600
}
```
Save `upload_url` for the next step and `bundle_s3_key` for the deploy request.
</Step>
<Step title="Upload the zip">
```bash theme={null}
curl -X PUT "$UPLOAD_URL" \
-H "Content-Type: application/zip" \
--data-binary @my-app.zip
```
</Step>
<Step title="Trigger the deploy">
A commit `message` is required — it describes what changed in this version.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"source": "cli_upload",
"bundle_s3_key": "$BUNDLE_KEY",
"environment": "production",
"message": "Initial deploy",
"source_files": {
"index.html": "<!DOCTYPE html>...",
"src/app.js": "// your source"
}
}'
```
<Warning>
`source_files` is **required** for `cli_upload` deploys. Without it the deploy returns 400.
Pass a `{path: content}` map of your source files so versions can be diffed and re-deployed.
</Warning>
Include `git_commit_sha` / `git_author` for CI/CD traceability.
</Step>
</Steps>
Your app is live at `https://{slug}.mxp.co` within seconds of a successful build.
***
## Environments
Each app supports two independent environments with separate URLs and asset prefixes:
| Environment | URL pattern | Deploy field |
| -------------- | ----------------------- | ----------------------------- |
| **Production** | `{slug}.mxp.co` | `"environment": "production"` |
| **Staging** | `staging-{slug}.mxp.co` | `"environment": "staging"` |
Staging and production can serve different versions simultaneously. Deploy to staging first to test, then promote to production.
### Deploying to staging
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"source": "cli_upload",
"bundle_s3_key": "$BUNDLE_KEY",
"environment": "staging",
"message": "Test new search layout"
}'
```
Your staging build is live at `https://staging-{slug}.mxp.co`.
### Promoting staging to production
Once you've verified staging, promote it to production by deploying the same bundle to the production environment, or use the restore endpoint to point production at the staging version:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/versions/$STAGING_VERSION/restore \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"environment": "production"}'
```
Restore is instant — no rebuild required.
***
## Deploy lifecycle
| Stage | Description |
| ---------- | ----------------------------------------------------------- |
| `queued` | Bundle uploaded, deploy queued |
| `building` | Mixpeek is validating, packaging, and uploading assets |
| `complete` | Deploy complete — new version is live |
| `failed` | Deploy failed — previous version stays live, error returned |
### Check deploy status
`GET /v1/apps/{app_id}/deploys/{deploy_id}` returns the current status of a deploy:
```bash theme={null}
curl https://api.mixpeek.com/v1/apps/$APP_ID/deploys/$DEPLOY_ID \
-H "Authorization: Bearer $API_KEY"
```
Response:
```json theme={null}
{
"deploy_id": "dep_abc123",
"status": "building",
"environment": "production",
"message": "Initial deploy",
"created_at": "2025-01-15T10:30:00Z"
}
```
The `status` field will be one of: `queued`, `building`, `complete`, or `failed`.
### Stream build logs
`GET /v1/apps/{app_id}/deploys/{deploy_id}/logs/stream` returns a Server-Sent Events (SSE) stream of build logs in real time. Use this to monitor progress or debug failed deploys:
```bash theme={null}
curl -N https://api.mixpeek.com/v1/apps/$APP_ID/deploys/$DEPLOY_ID/logs/stream \
-H "Authorization: Bearer $API_KEY"
```
Each SSE event contains a log line from the build process. The stream closes automatically when the deploy reaches `complete` or `failed`.
***
## Multi-file output
Your zip can contain any number of files — HTML, JS, CSS, images, fonts. The only requirement is an `index.html` at the root. All files are uploaded to S3 and served with appropriate cache headers:
* **`index.html`** — no-cache (always fresh)
* **Content-hashed files** (e.g., `app-a1b2c3.js`) — immutable, permanent cache
* **Other assets** — standard cache headers
***
## Canvas SDK
Your app runs inside the Mixpeek canvas runtime. Call Mixpeek APIs through the built-in `/api` proxy — credentials are injected **server-side**, so your API key never reaches the browser:
```jsx theme={null}
// src/App.jsx — no auth headers needed
async function search(query) {
const res = await fetch('/api/v1/retrievers/ret_abc123/execute', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ inputs: { query }, settings: { limit: 20 } }),
})
return res.json()
}
```
<Note>
Use `/api/v1/...` (relative path) instead of `https://api.mixpeek.com/v1/...` — the canvas proxy injects `Authorization` and `X-Namespace` headers automatically, avoiding CORS and keeping API keys out of your bundle.
</Note>
***
## Pre-deploy validation
Mixpeek runs automatic checks on every bundle before deploying:
| Check | What it catches |
| -------------------------- | ----------------------------------------------------------- |
| **index.html exists** | Missing entry point (zip structure wrong) |
| **Script tags present** | Blank page (no executable code) |
| **Asset references valid** | Broken `src`/`href` links in index.html |
| **No bare `process.env`** | Runtime crash in browser (use `window.__MIXPEEK__` instead) |
| **Bundle size \< 50 MB** | Accidentally included `node_modules` or large assets |
| **JS bundles non-empty** | Failed build that produced empty files |
If any check fails, the deploy is rejected with a descriptive error message.
***
## Deploy via Studio
In the App details page, drag & drop your `.zip` file onto the **Deploy** panel and click **Deploy**. The build is queued immediately and status updates in real time.
***
## Rollback
Every deploy is versioned. You have two rollback options:
**Quick rollback** — restore the previous config:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/rollback \
-H "Authorization: Bearer $API_KEY"
```
**Restore any version** — point an environment to a specific version's assets:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/versions/3/restore \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"environment": "production"}'
```
Restore is instant — no rebuild required. All previous assets are retained in S3.
See [Version History](/docs/canvas/apps/versions) for the full version control system including diffs, downloads, and git metadata.
***
## Source files
When you include `source_files` in your deploy request, those files are stored alongside the version record. This enables:
* **Source-level diffs** — compare actual source code between versions, not just built output
* **Download and re-deploy** — download a version's source files and deploy modified code
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"source": "cli_upload",
"bundle_s3_key": "$BUNDLE_KEY",
"environment": "production",
"message": "Add dark mode",
"source_files": {
"src/App.tsx": "import React from ...",
"src/components/Header.tsx": "export function Header() ...",
"src/styles.css": "body { ... }"
}
}'
```
***
## Related
* [Apps overview](/docs/canvas/apps)
* [Version History](/docs/canvas/apps/versions)
* [Custom Domains](/docs/canvas/apps/domains)
* [Deploy App (API)](/docs/api-reference/apps/deploy-app)
* [List Versions (API)](/docs/api-reference/apps/list-versions)
# Custom Domains
Source: https://docs.mixpeek.com/docs/canvas/apps/domains
Point your own subdomain at a Mixpeek App. TLS is provisioned automatically.
## Overview
Every App gets a default URL at `{slug}.mxp.co`. You can also serve it on your own subdomain — for example, `search.yourcompany.com` — by adding a custom domain and creating a DNS CNAME record.
***
## Add a custom domain
<Steps>
<Step title="Open the Domains panel">
In the App details page, click the **Domains** tab.
</Step>
<Step title="Enter your subdomain">
Type the subdomain you want to use (e.g. `search.yourcompany.com`) and click **Add Domain**.
</Step>
<Step title="Add a DNS TXT record for verification">
Add a TXT record to prove domain ownership. The record name and value are returned in the API response:
| Type | Name | Value |
| ---- | ---------------------------------------- | ------------------------------------------------ |
| TXT | `_mixpeek-verify.search.yourcompany.com` | `mixpeek-site-verification={verification_token}` |
</Step>
<Step title="Create a CNAME record">
In your DNS provider, add a `CNAME` record pointing your subdomain to the `cname_target` returned by the API:
| Type | Name | Value |
| ----- | ---------------------- | --------------- |
| CNAME | search.yourcompany.com | `{slug}.mxp.co` |
</Step>
<Step title="Trigger verification">
Call `POST /v1/apps/{app_id}/domains/{domain}/verify` to start DNS polling. Mixpeek polls every 30 minutes for up to 72 hours, then provisions a TLS certificate automatically via Cloudflare.
Domain status changes to `active` once the certificate is issued.
</Step>
</Steps>
***
## Domain statuses
| Status | Meaning |
| ------------------ | ----------------------------------------------------------- |
| `pending` | Domain added, waiting for DNS verification |
| `verifying` | TXT record found, polling for CNAME propagation |
| `provisioning_tls` | CNAME verified, issuing TLS certificate |
| `active` | Domain is live and serving HTTPS |
| `failed` | Verification failed after 72 hours — check your DNS records |
***
## Via API
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key", namespace_id="ns_...")
domain = client.apps.domains.add(
app_id="app_...",
domain="search.yourcompany.com",
)
print(domain.status) # "pending"
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/domains \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{"domain": "search.yourcompany.com"}'
```
</CodeGroup>
***
## Notes
* Only subdomain CNAMEs are supported (not apex/root domains).
* Each domain can be attached to one App at a time.
* Removing a domain does not affect the App's default `{slug}.mxp.co` URL.
***
## Related
* [Apps overview](/docs/canvas/apps)
* [Deployment guide](/docs/canvas/apps/deploy)
* [Add Domain (API)](/docs/api-reference/apps/add-domain)
* [List Domains (API)](/docs/api-reference/apps/list-domains)
# Environment Variables
Source: https://docs.mixpeek.com/docs/canvas/apps/environment-variables
Configure environment variables for your Canvas app — available in both the browser runtime and server functions.
## Overview
Canvas apps have two ways to access configuration:
1. **Browser runtime** — `window.__MIXPEEK__` (injected into every HTML response)
2. **Server functions** — `ctx.env` (available in your handler's context)
***
## Browser runtime variables
Canvas automatically injects a `window.__MIXPEEK__` object into every HTML page. This includes system variables and any custom environment variables you configure.
### System variables (always available)
| Variable | Description |
| -------------------------------- | ------------------------------------------------------------- |
| `window.__MIXPEEK__.apiBaseUrl` | The API proxy URL (`/api`) |
| `window.__MIXPEEK__.appId` | Your app's ID |
| `window.__MIXPEEK__.slug` | Your app's slug |
| `window.__MIXPEEK__.environment` | Current environment (`production`, `staging`, or `preview-N`) |
### Accessing in your app
```jsx theme={null}
// React example
function App() {
const mixpeek = window.__MIXPEEK__
async function search(query) {
const res = await fetch(`${mixpeek.apiBaseUrl}/v1/retrievers/ret_abc/execute`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ inputs: { query } }),
})
return res.json()
}
return <SearchBar onSearch={search} />
}
```
<Note>
Use `window.__MIXPEEK__.apiBaseUrl` (which resolves to `/api`) instead of hardcoding `https://api.mixpeek.com`. The Canvas proxy injects your API key and namespace header server-side, keeping credentials out of the browser.
</Note>
***
## Custom environment variables
Set custom environment variables on your app via the API:
```bash theme={null}
curl -X PATCH https://api.mixpeek.com/v1/apps/$APP_ID \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"build_config": {
"env_vars": {
"FEATURE_FLAG_NEW_UI": "true",
"ANALYTICS_ID": "UA-12345"
}
}
}'
```
Custom env vars are:
* Injected into `window.__MIXPEEK__` for browser access
* Available as `ctx.env` in [server functions](/docs/canvas/apps/server-functions)
* Stored encrypted and never exposed in build logs
***
## Server function environment
Server functions receive all environment variables through `ctx.env`:
```typescript theme={null}
// api/config.ts
export default async function handler(ctx) {
return {
body: {
feature_flag: ctx.env.FEATURE_FLAG_NEW_UI,
api_key_set: !!ctx.env.MIXPEEK_API_KEY,
},
}
}
```
Server functions also have access to system environment variables like `MIXPEEK_API_KEY` and `MIXPEEK_NAMESPACE_ID` — these are injected server-side and never reach the browser.
***
## Environment-specific variables
Since Canvas supports multiple environments (production, staging, preview), you can set different variables per environment by deploying with different configurations.
***
## Related
* [Server Functions](/docs/canvas/apps/server-functions)
* [Deploy from Code](/docs/canvas/apps/deploy)
* [Apps Overview](/docs/canvas/apps)
# GitHub Integration
Source: https://docs.mixpeek.com/docs/canvas/apps/github
Connect a GitHub repo to your Canvas app for automatic deploys on push and preview URLs on pull requests.
## Overview
Connect a GitHub repository to a Canvas app and every push to your branch triggers an automatic build and deploy. Pull requests get [preview URLs](/docs/canvas/apps/preview-deploys) for live review before merging.
***
## Connecting a repo
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/connect-repo \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"repo_url": "https://github.com/your-org/your-repo",
"branch": "main"
}'
```
This configures a GitHub webhook that listens for `push` and `pull_request` events on the specified branch.
***
## Automatic deploys on push
When you push to the connected branch:
1. GitHub sends a `push` webhook to Canvas
2. Canvas clones your repo and checks out the pushed commit
3. Runs `npm ci` to install dependencies
4. Runs `npm run build` to produce the build output
5. Zips the `dist/` directory and deploys it
Build output is streamed in real time — see [Build Logs](/docs/canvas/apps/build-logs).
<Note>
Canvas auto-detects your build output directory. It looks for `dist/`, `build/`, or `out/` in order.
</Note>
***
## What gets built
Canvas runs a standard Node.js build process:
```
git clone <repo_url> --branch <branch> --depth 1
cd <repo>
npm ci
npm run build
```
Your `package.json` must include a `build` script. The build output must contain an `index.html`.
***
## Git metadata in versions
Every deploy from GitHub automatically records git metadata in the version record:
| Field | Description |
| ---------------- | ---------------------------------------- |
| `git_commit_sha` | The commit SHA that triggered the deploy |
| `git_author` | Author of the commit |
| `git_branch` | Branch name |
This metadata is visible in [Version History](/docs/canvas/apps/versions) and in Studio.
***
## Webhook events
| Event | Canvas behavior |
| ----------------------------- | -------------------------------------------------------- |
| `push` to connected branch | Build and deploy to production |
| `pull_request` opened/updated | Build and deploy [preview](/docs/canvas/apps/preview-deploys) |
| `pull_request` closed | Clean up preview environment |
***
## Disconnecting
To stop automatic deploys, remove the webhook from your GitHub repository settings or update the app's configuration to remove the `repo_url`.
***
## Related
* [Deploy from Code](/docs/canvas/apps/deploy)
* [Preview Deploys](/docs/canvas/apps/preview-deploys)
* [Build Logs](/docs/canvas/apps/build-logs)
* [Version History](/docs/canvas/apps/versions)
# Runtime Logs & Analytics
Source: https://docs.mixpeek.com/docs/canvas/apps/logs
View request logs, server function output, and performance metrics for your Canvas app.
## Overview
Canvas captures two types of runtime data for every app:
* **Request logs** — every HTTP request hitting your app (method, path, status, duration)
* **Runtime logs** — `console.log/warn/error/info` output from [server functions](/docs/canvas/apps/server-functions)
Both are stored in ClickHouse with a 30-day retention and queryable via API or Studio.
***
## Request logs
Every request to your Canvas app is logged automatically — no configuration needed.
### Query via API
```bash theme={null}
curl "https://api.mixpeek.com/v1/apps/$APP_ID/logs?log_type=requests&hours=1&limit=100" \
-H "Authorization: Bearer $API_KEY"
```
Response:
```json theme={null}
{
"app_id": "app_1433d93d7a95",
"log_type": "requests",
"entries": [
{
"timestamp": "2026-03-29 21:21:02.750",
"method": "GET",
"path": "/",
"status_code": 200,
"duration_ms": 74,
"user_agent": "Mozilla/5.0 ...",
"ip": "68.175.65.201"
}
]
}
```
### Parameters
| Parameter | Type | Default | Description |
| ---------- | --------- | --------- | ------------------------------------------------------------------ |
| `log_type` | `string` | `runtime` | `runtime` or `requests` |
| `hours` | `integer` | `1` | Lookback window (1–168 hours) |
| `level` | `string` | — | Filter by log level (runtime only): `log`, `warn`, `error`, `info` |
| `limit` | `integer` | `100` | Max entries returned (1–1000) |
***
## Runtime logs
Server function `console` output is automatically captured and stored. Each log entry includes the function route and log level.
### Query via API
```bash theme={null}
# All runtime logs from the last hour
curl "https://api.mixpeek.com/v1/apps/$APP_ID/logs?log_type=runtime&hours=1" \
-H "Authorization: Bearer $API_KEY"
# Only errors
curl "https://api.mixpeek.com/v1/apps/$APP_ID/logs?log_type=runtime&hours=24&level=error" \
-H "Authorization: Bearer $API_KEY"
```
Response:
```json theme={null}
{
"app_id": "app_1433d93d7a95",
"log_type": "runtime",
"entries": [
{
"timestamp": "2026-03-29 22:15:03.456",
"level": "error",
"message": "Failed: connection timeout",
"route_path": "search",
"request_id": "req_abc123"
}
]
}
```
***
## Studio
In Studio, the App details page has an **Analytics** panel with three tabs:
| Tab | What it shows |
| ---------------- | -------------------------------------------------------- |
| **Overview** | Error rates, web vitals, custom events |
| **Runtime Logs** | Server function console output with level badges |
| **Requests** | HTTP request log with method, path, status, and duration |
Both log tabs auto-refresh every 10 seconds and support filtering by time range and log level.
***
## What's captured
### Request log fields
| Field | Description |
| ------------- | -------------------------------- |
| `timestamp` | When the request was received |
| `method` | HTTP method (GET, POST, etc.) |
| `path` | Request path |
| `status_code` | HTTP response status |
| `duration_ms` | Time to process the request (ms) |
| `user_agent` | Client user agent string |
| `ip` | Client IP address |
### Runtime log fields
| Field | Description |
| ------------ | -------------------------------------------- |
| `timestamp` | When the log was emitted |
| `level` | `log`, `warn`, `error`, or `info` |
| `message` | The logged message (truncated to 2000 chars) |
| `route_path` | Which server function produced the log |
| `request_id` | Correlates logs to a specific request |
***
## Retention
All log data is retained for **30 days** and automatically purged after that.
***
## Related
* [Server Functions](/docs/canvas/apps/server-functions)
* [Build Logs](/docs/canvas/apps/build-logs)
* [Deploy from Code](/docs/canvas/apps/deploy)
# Monitoring
Source: https://docs.mixpeek.com/docs/canvas/apps/monitoring
Automatic error tracking, web vitals, and custom event reporting for Canvas apps — with optional Sentry and PostHog integration.
## Overview
Canvas automatically captures client-side errors, web performance metrics, and custom events for every app. Data flows to Mixpeek's analytics backend (ClickHouse) and optionally to your own Sentry or PostHog instance.
***
## Built-in error tracking
Canvas injects a lightweight error boundary that captures:
* **Unhandled exceptions** — `window.onerror` and `unhandledrejection` events
* **Error details** — message, stack trace, source URL, line/column numbers
* **Context** — app ID, user agent, page URL
Errors are sent to the `canvas_errors` ClickHouse table and visible in Studio's Analytics panel.
### Querying errors via API
```bash theme={null}
curl "https://api.mixpeek.com/v1/apps/$APP_ID/analytics/errors?minutes=60" \
-H "Authorization: Bearer $API_KEY"
```
***
## Web vitals
Canvas automatically reports [Core Web Vitals](https://web.dev/vitals/) for every page load:
| Metric | What it measures |
| ---------------------------------- | ------------------- |
| **LCP** (Largest Contentful Paint) | Loading performance |
| **FID** (First Input Delay) | Interactivity |
| **CLS** (Cumulative Layout Shift) | Visual stability |
Vitals are stored in `canvas_vitals` and visible in Studio.
***
## Custom events
Track custom events from your app using the Canvas analytics API:
```javascript theme={null}
// In your app code
fetch('/_canvas/events', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
event_type: 'search',
event_name: 'query_executed',
event_data: JSON.stringify({ query: 'red shoes', results: 42 }),
}),
})
```
***
## Sentry integration
Send errors to your own Sentry project by configuring the monitoring settings:
```bash theme={null}
curl -X PATCH https://api.mixpeek.com/v1/apps/$APP_ID \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"monitoring": {
"sentry_dsn": "https://abc123@o456.ingest.sentry.io/789"
}
}'
```
Canvas injects the Sentry SDK automatically — no code changes needed in your app.
***
## PostHog integration
Enable PostHog product analytics:
```bash theme={null}
curl -X PATCH https://api.mixpeek.com/v1/apps/$APP_ID \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"monitoring": {
"posthog_api_key": "phc_abc123",
"posthog_host": "https://app.posthog.com"
}
}'
```
Canvas injects the PostHog snippet automatically. Page views, sessions, and custom events are tracked.
***
## Studio
The **Analytics** panel in Studio shows:
* **Error rate** — errors per minute over the selected time range
* **Web vitals** — LCP, FID, CLS distributions
* **Custom events** — event counts by type
* **Runtime logs** — server function console output (see [Runtime Logs](/docs/canvas/apps/logs))
* **Request logs** — HTTP request log with status codes and latency
***
## Related
* [Runtime Logs & Analytics](/docs/canvas/apps/logs)
* [Server Functions](/docs/canvas/apps/server-functions)
* [Apps Overview](/docs/canvas/apps)
# Preview Deploys
Source: https://docs.mixpeek.com/docs/canvas/apps/preview-deploys
Get a unique preview URL for every pull request, automatically built and cleaned up when the PR closes.
## Overview
When you connect a GitHub repo to a Canvas app, every pull request gets its own preview deployment at a unique URL. Reviewers can test changes live before merging — no manual deploy needed.
***
## How it works
1. **Open a PR** targeting your app's connected branch
2. GitHub sends a `pull_request` webhook to Canvas
3. Canvas clones the PR's head branch, builds it, and deploys to a preview environment
4. The preview is live at `https://pr-{N}-{slug}.mxp.co`
5. **Push more commits** to the PR — the preview auto-updates
6. **Close or merge the PR** — the preview environment is cleaned up
***
## Preview URLs
Preview deploys follow a predictable URL pattern:
| PR | App slug | Preview URL |
| ---- | ----------- | --------------------------------- |
| #42 | `my-app` | `https://pr-42-my-app.mxp.co` |
| #17 | `taste` | `https://pr-17-taste.mxp.co` |
| #103 | `dashboard` | `https://pr-103-dashboard.mxp.co` |
Preview environments are fully functional — they use the same API proxy, auth configuration, and environment variables as your production app.
***
## Setup
Preview deploys work automatically once your repo is connected. No additional configuration is needed.
<Steps>
<Step title="Connect your repo">
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/connect-repo \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"repo_url": "https://github.com/your-org/your-repo",
"branch": "main"
}'
```
</Step>
<Step title="Open a pull request">
Open a PR targeting the connected branch (e.g., `main`). Canvas automatically detects the PR and starts building.
</Step>
<Step title="Review the preview">
Visit `https://pr-{N}-{slug}.mxp.co` to see the live preview. The URL is also posted as a GitHub deployment status on the PR.
</Step>
</Steps>
***
## Supported webhook events
| Event | Action | Canvas behavior |
| -------------- | ------------- | -------------------------------- |
| `pull_request` | `opened` | Build and deploy preview |
| `pull_request` | `synchronize` | Rebuild preview with new commits |
| `pull_request` | `reopened` | Rebuild preview |
| `pull_request` | `closed` | Clean up preview environment |
***
## Preview vs staging vs production
| Environment | URL pattern | Use case |
| -------------- | ----------------------- | ------------------------- |
| **Production** | `{slug}.mxp.co` | Live site |
| **Staging** | `staging-{slug}.mxp.co` | Manual staging deploys |
| **Preview** | `pr-{N}-{slug}.mxp.co` | Automatic per-PR previews |
Each preview environment is independent — multiple PRs can have active previews simultaneously.
***
## Cleanup
When a PR is closed (merged or not), Canvas automatically:
1. Removes the preview environment from the app's configuration
2. Deletes preview assets from S3
No manual cleanup is needed.
***
## Related
* [Deploy from Code](/docs/canvas/apps/deploy)
* [Build Logs](/docs/canvas/apps/build-logs)
* [GitHub Integration](/docs/canvas/apps/github)
# Server Functions
Source: https://docs.mixpeek.com/docs/canvas/apps/server-functions
Run server-side JavaScript and TypeScript in your Canvas app — access environment variables, key-value storage, and the Mixpeek API without exposing secrets.
## Overview
Server functions let you run backend logic inside your Canvas app. Write TypeScript or JavaScript handlers that execute server-side with access to environment variables, a built-in KV store, and `fetch` — without managing infrastructure.
***
## How it works
1. Include server function source files in your deploy's `source_files` map
2. Files under `api/` become HTTP endpoints at `/functions/`
3. Canvas transpiles TypeScript → JavaScript via esbuild and executes your handler in a sandboxed async context
| Source file | Endpoint |
| -------------------- | --------------------------------- |
| `api/hello.ts` | `GET/POST /functions/hello` |
| `api/search.js` | `GET/POST /functions/search` |
| `api/data/export.ts` | `GET/POST /functions/data/export` |
***
## Writing a server function
Export a default function that receives a context object and returns a response:
```typescript theme={null}
// api/hello.ts
export default async function handler(ctx) {
const name = ctx.req.query.name || 'world'
return {
status: 200,
body: { message: `Hello, ${name}!` },
}
}
```
### Context object
Your handler receives a `ctx` object with:
| Property | Type | Description |
| ----------------- | -------- | -------------------------------------------- |
| `ctx.req.method` | `string` | HTTP method (GET, POST, etc.) |
| `ctx.req.path` | `string` | Request path |
| `ctx.req.query` | `object` | URL query parameters |
| `ctx.req.body` | `any` | Parsed JSON body (for POST/PUT) |
| `ctx.req.headers` | `object` | Request headers (excluding internal headers) |
| `ctx.env` | `object` | App environment variables |
| `ctx.kv` | `object` | Key-value store (see below) |
### Return value
Return an object with:
| Field | Type | Default | Description |
| --------- | -------- | ------- | -------------------------------------------- |
| `status` | `number` | `200` | HTTP status code |
| `body` | `any` | — | Response body (object → JSON, string → text) |
| `headers` | `object` | `{}` | Custom response headers |
***
## Key-value store
Every app has a built-in Redis-backed KV store, accessible in server functions via `ctx.kv`:
```typescript theme={null}
// api/counter.ts
export default async function handler(ctx) {
const current = await ctx.kv.get('visit_count')
const count = parseInt(current || '0') + 1
await ctx.kv.set('visit_count', String(count))
return {
body: { visits: count },
}
}
```
| Method | Signature | Description |
| ------ | ----------------------------------------------------------------- | -------------------------------------------------------- |
| `get` | `kv.get(key: string): Promise<string \| null>` | Read a value |
| `set` | `kv.set(key: string, value: string, ttl?: number): Promise<void>` | Write a value (optional TTL in seconds, default 30 days) |
| `del` | `kv.del(key: string): Promise<void>` | Delete a key |
| `keys` | `kv.keys(): Promise<string[]>` | List all keys for this app |
***
## Calling the Mixpeek API
Use `fetch` to call the Mixpeek API through the Canvas proxy — credentials are injected automatically:
```typescript theme={null}
// api/search.ts
export default async function handler(ctx) {
const { query } = ctx.req.body
const res = await fetch('https://api.mixpeek.com/v1/retrievers/ret_abc123/execute', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${ctx.env.MIXPEEK_API_KEY}`,
'X-Namespace': ctx.env.MIXPEEK_NAMESPACE_ID,
},
body: JSON.stringify({ inputs: { query }, settings: { limit: 10 } }),
})
return { body: await res.json() }
}
```
***
## Deploying with source files
Include your server functions in the `source_files` field when deploying:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"source": "cli_upload",
"bundle_s3_key": "$BUNDLE_KEY",
"environment": "production",
"message": "Add search API route",
"source_files": {
"api/hello.ts": "export default async function handler(ctx) {\n return { body: { message: \"Hello!\" } }\n}",
"api/search.ts": "export default async function handler(ctx) {\n // search logic here\n}"
}
}'
```
***
## Runtime logging
`console.log`, `console.warn`, `console.error`, and `console.info` calls in server functions are captured and sent to the [runtime logs](/docs/canvas/apps/logs) system. View them in Studio or query via API.
```typescript theme={null}
// api/process.ts
export default async function handler(ctx) {
console.log('Processing request', ctx.req.method, ctx.req.path)
try {
const result = await doWork(ctx.req.body)
console.info('Success:', result.id)
return { body: result }
} catch (err) {
console.error('Failed:', err.message)
return { status: 500, body: { error: err.message } }
}
}
```
***
## Limitations
* **Execution timeout:** 30 seconds per request
* **Request body size:** 1 MB max
* **No filesystem access:** Server functions run in a sandboxed context
* **No npm imports:** Only `fetch` and the `ctx` APIs are available in the execution context
* **TypeScript only:** `.ts` and `.js` files are supported (transpiled via esbuild)
***
## Related
* [Deploy from Code](/docs/canvas/apps/deploy)
* [Runtime Logs](/docs/canvas/apps/logs)
* [Environment Variables](/docs/canvas/apps/environment-variables)
# User Management
Source: https://docs.mixpeek.com/docs/canvas/apps/users
Manage users for your Canvas app with Clerk organizations. Invite members, assign roles, and control access — all through the API or Studio.
## Overview
Each Canvas app with Clerk authentication gets a dedicated **Clerk organization**. This organization is the single source of truth for who has access to the app and what role they hold.
User management lets you:
* **List members** of a Canvas app
* **Invite users** by email with a specific role
* **Update roles** (admin or member)
* **Remove users** to revoke access immediately
* **Manage invitations** — list pending and revoke unused invitations
<Note>
User management requires `auth_config.mode` set to `"clerk"`. Apps using `public`, `password`, or other auth modes do not have per-user access control.
</Note>
***
## Enabling User Management
Set your app's auth mode to `clerk`. Mixpeek automatically provisions a Clerk organization for the app:
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
client.apps.update(
app_id="app_abc123",
auth_config={"mode": "clerk"},
)
```
```bash cURL theme={null}
curl -X PATCH https://api.mixpeek.com/v1/apps/$APP_ID \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{"auth_config": {"mode": "clerk"}}'
```
</CodeGroup>
Once enabled, the **Users** tab appears in the Studio app detail page.
***
## Roles
Each member has one of two roles:
| Role | Slug | Description |
| ---------- | ------------ | ------------------------------------- |
| **Member** | `org:member` | Standard access to the app |
| **Admin** | `org:admin` | Full access including user management |
***
## API Reference
All user management endpoints are scoped to a single app and require your API key with namespace header.
### List Members
Returns all users in the app's Clerk organization.
```bash theme={null}
curl https://api.mixpeek.com/v1/apps/$APP_ID/users \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
**Response:**
```json theme={null}
{
"users": [
{
"user_id": "user_abc123",
"membership_id": "mem_xyz789",
"email": "jane@example.com",
"name": "Jane Smith",
"avatar_url": "https://img.clerk.com/...",
"role": "org:admin",
"joined_at": "2026-03-15T10:30:00Z"
}
],
"total": 1
}
```
### Invite a User
Send an email invitation to join the app's organization.
<CodeGroup>
```python Python theme={null}
import httpx
resp = httpx.post(
f"https://api.mixpeek.com/v1/apps/{app_id}/users/invite",
headers={
"Authorization": f"Bearer {api_key}",
"X-Namespace": namespace_id,
},
json={
"email_address": "newuser@example.com",
"role": "org:member",
},
)
print(resp.json())
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/users/invite \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"email_address": "newuser@example.com",
"role": "org:member"
}'
```
</CodeGroup>
**Response:**
```json theme={null}
{
"invitation_id": "inv_abc123",
"email_address": "newuser@example.com",
"role": "org:member",
"status": "pending",
"created_at": "2026-03-27T14:00:00Z"
}
```
### Update a User's Role
Change a member's role between `org:member` and `org:admin`.
```bash theme={null}
curl -X PATCH https://api.mixpeek.com/v1/apps/$APP_ID/users/$USER_ID/role \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{"role": "org:admin"}'
```
### Remove a User
Immediately revokes the user's access to the app.
```bash theme={null}
curl -X DELETE https://api.mixpeek.com/v1/apps/$APP_ID/users/$USER_ID \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
### List Pending Invitations
```bash theme={null}
curl https://api.mixpeek.com/v1/apps/$APP_ID/users/invitations \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
**Response:**
```json theme={null}
{
"invitations": [
{
"invitation_id": "inv_abc123",
"email_address": "pending@example.com",
"role": "org:member",
"status": "pending",
"created_at": "2026-03-27T14:00:00Z"
}
],
"total": 1
}
```
### Revoke an Invitation
Cancel a pending invitation before the recipient accepts it.
```bash theme={null}
curl -X DELETE https://api.mixpeek.com/v1/apps/$APP_ID/users/invitations/$INVITATION_ID \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
***
## Studio
When a Canvas app has `auth_config.mode = "clerk"`, a **Users** tab appears in the app detail page. From there you can:
1. **View members** — see all users with their name, email, avatar, and role
2. **Invite users** — click **Invite**, enter an email and role, then send
3. **Change roles** — use the role dropdown next to any member
4. **Remove users** — click the trash icon with a confirmation dialog
5. **Manage invitations** — view pending invitations and revoke them
***
## How It Works
Each Canvas app maps to a **Clerk organization**. When you enable `auth_config.mode = "clerk"`, Mixpeek auto-provisions a Clerk org and stores the `clerk_org_id` in the app record.
All user management operations go directly to Clerk — there is no separate user database to maintain. Clerk is the single source of truth.
```
Your App (auth_config.mode = "clerk")
└── Clerk Organization (auto-provisioned)
├── Members (org:admin, org:member)
└── Pending Invitations
```
<Warning>
Removing a user revokes access immediately. The user will be signed out on their next request.
</Warning>
***
## Related
* [Authentication](/docs/canvas/apps/authentication) — enable Clerk auth and use the MixpeekAuth SDK
* [Apps overview](/docs/canvas/apps) — Canvas app architecture and quickstart
* [Custom Domains](/docs/canvas/apps/domains) — add your own domain to a Canvas app
# Version History
Source: https://docs.mixpeek.com/docs/canvas/apps/versions
Every change to a Canvas app is versioned with a content hash, commit message, and diffable snapshot — like git for deployed applications.
## Overview
Canvas apps have built-in version control. Every time you **publish** config changes or **deploy** a new code bundle, an immutable version record is created with:
* A **content hash** (SHA-256) — like a git commit SHA
* A **commit message** (required) — describes what changed
* A **full config snapshot** — serialized as diffable JSON files
* **Who** published and **when**
You can list all versions, diff any two versions, download previous bundles, and restore (rollback) to any version instantly.
***
## How versioning works
There are two ways a new version is created:
| Action | What creates the version | What's captured |
| ------------------------------------------ | --------------------------------------------------- | --------------------------------------------------------------- |
| **Publish** (`POST /v1/apps/{id}/publish`) | Config snapshot (sections, meta, auth, theme, etc.) | Content hash of config, message, config as JSON files |
| **Deploy** (`POST /v1/apps/{id}/deploy`) | Code bundle upload (React/JS build) | Asset manifest with file hashes, message, optional source files |
Both paths create a `VersionRecord` in the app's `versions` array. The version number auto-increments.
### Content hashing
When you publish, Mixpeek generates a deterministic SHA-256 hash of your entire config snapshot. This hash changes only when the config actually changes — identical publishes produce the same hash.
```
v1 → hash: d733b11021fa → "Initial release"
v2 → hash: e186b986ff96 → "feat: add custom HTML search interface"
v3 → hash: d733b11021fa → "revert: back to original config" (same hash as v1!)
```
### Config as diffable files
Your app config is serialized into individual JSON files for each top-level key. This makes version diffs meaningful:
```
meta.json → {"title": "Product Search", "logo_url": "..."}
theme.json → {"colors": {"primary": "#FC5185"}}
sections.json → [{"type": "search", ...}]
auth_config.json → {"mode": "clerk", "clerk_allowed_providers": [...]}
custom_html.json → "<div id=\"app\">...</div>"
```
***
## Listing versions
<CodeGroup>
```bash cURL theme={null}
curl https://api.mixpeek.com/v1/apps/$APP_ID/versions \
-H "Authorization: Bearer $API_KEY"
```
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
versions = client.apps.versions.list(app_id="app_abc123")
for v in versions.versions:
print(f"v{v.version}: {v.s3_version_id} — {v.message}")
```
</CodeGroup>
Response:
```json theme={null}
{
"versions": [
{
"version": 2,
"s3_version_id": "e186b986ff96",
"message": "feat: add custom HTML search interface",
"deployed_by": "int_40ed22c147907235",
"deployed_at": "2026-03-27T13:10:00Z",
"environment": "production"
},
{
"version": 1,
"s3_version_id": "d733b11021fa",
"message": "Initial release",
"deployed_by": "int_40ed22c147907235",
"deployed_at": "2026-03-27T13:09:00Z",
"environment": "production"
}
],
"total": 2
}
```
***
## Viewing version details
Get full metadata for a specific version, including the source files snapshot and which environments it's active in:
```bash theme={null}
curl https://api.mixpeek.com/v1/apps/$APP_ID/versions/1 \
-H "Authorization: Bearer $API_KEY"
```
Response:
```json theme={null}
{
"version": 1,
"s3_version_id": "d733b11021fa",
"message": "Initial release",
"deployed_by": "int_40ed22c147907235",
"deployed_at": "2026-03-27T13:09:00Z",
"environment": "production",
"source_files": {
"meta.json": "{\"title\": \"Product Search\"}",
"theme.json": "{\"colors\": {\"primary\": \"#FC5185\"}}",
"sections.json": "[]"
},
"is_active": {
"staging": false,
"production": true
}
}
```
***
## Diffing versions
Compare any two versions to see what changed — like `git diff v1..v2`:
```bash theme={null}
curl https://api.mixpeek.com/v1/apps/$APP_ID/versions/1/diff/2 \
-H "Authorization: Bearer $API_KEY"
```
Response:
```json theme={null}
{
"app_id": "app_abc123",
"from_version": 1,
"to_version": 2,
"summary": {
"added": 1,
"removed": 0,
"modified": 0,
"unchanged": 7
},
"source_diff": {
"custom_html.json": "--- v1/custom_html.json\n+++ v2/custom_html.json\n@@ -0,0 +1 @@\n+\"<div><h1>Custom Search</h1></div>\""
}
}
```
The `summary` counts file-level changes (based on content hashes). The `source_diff` provides unified diffs of the actual content — the same format as `git diff`.
***
## Rollback
### Quick rollback (one level)
Restore the previous published config instantly:
<CodeGroup>
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/rollback \
-H "Authorization: Bearer $API_KEY"
```
```python Python theme={null}
app = client.apps.rollback(app_id="app_abc123")
print(f"Rolled back to v{app.version}")
```
</CodeGroup>
### Restore any version
For deploy-based versions (code bundles), restore any specific version to any environment:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/versions/3/restore \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"environment": "production"}'
```
This is instant — no rebuild required. The previous assets are always retained in S3.
***
## Download a version's bundle
Download the original zip bundle for any deployed version and work locally:
<Steps>
<Step title="Download the bundle">
```bash theme={null}
curl https://api.mixpeek.com/v1/apps/$APP_ID/versions/2/download \
-H "Authorization: Bearer $API_KEY"
```
Response:
```json theme={null}
{
"download_url": "https://s3.amazonaws.com/...",
"version": 2,
"asset_prefix": "apps/my-app/dep_abc123",
"expires_in": 3600
}
```
</Step>
<Step title="Download and extract">
```bash theme={null}
curl -o bundle.zip "$DOWNLOAD_URL"
unzip bundle.zip -d my-app/
```
</Step>
<Step title="Edit files locally">
Make your changes to the extracted source.
</Step>
<Step title="Re-zip and deploy">
```bash theme={null}
cd my-app && zip -r ../updated.zip .
# Upload and deploy as usual (see Deploy from Code)
```
</Step>
</Steps>
***
## Git metadata
When deploying via CI/CD or the CLI, you can attach git metadata to each version for full traceability:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/apps/$APP_ID/deploy \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"source": "cli_upload",
"bundle_s3_key": "$BUNDLE_KEY",
"message": "fix: search results pagination",
"git_commit_sha": "a1b2c3d4e5f6",
"git_commit_message": "fix: search results pagination",
"git_author": "Jane Smith <jane@example.com>"
}'
```
Git metadata appears in version detail responses and Studio's version history panel.
<Tip>
If you connect a GitHub repository via `POST /v1/apps/{id}/connect-repo`, deploys are triggered automatically on push. Git metadata is captured from the webhook payload — no manual fields needed.
</Tip>
***
## Version record fields
| Field | Type | Description |
| -------------------- | -------- | ------------------------------------------------ |
| `version` | `int` | Auto-incrementing version number |
| `s3_version_id` | `string` | Content hash (publish) or S3 version ID (deploy) |
| `asset_prefix` | `string` | S3 path prefix for deployed assets |
| `asset_manifest` | `object` | Map of `{relative_path: {s3_key, hash, size}}` |
| `source_files` | `object` | Map of `{path: content}` for source-level diffs |
| `deployed_by` | `string` | Internal ID of the user who published/deployed |
| `deployed_at` | `string` | ISO 8601 timestamp |
| `environment` | `string` | `"staging"` or `"production"` |
| `message` | `string` | Commit message (required) |
| `build_duration_ms` | `int` | Build time in milliseconds (deploy only) |
| `git_commit_sha` | `string` | Git SHA (if provided) |
| `git_commit_message` | `string` | Git commit message (if provided) |
| `git_author` | `string` | Git author (if provided) |
***
## Related
* [Apps overview](/docs/canvas/apps)
* [Deploy from Code](/docs/canvas/apps/deploy)
* [List Versions (API)](/docs/api-reference/apps/list-versions)
* [Publish App (API)](/docs/api-reference/apps/publish-app)
* [Rollback App (API)](/docs/api-reference/apps/rollback-app)
# Alerts
Source: https://docs.mixpeek.com/docs/enrichment/alerts
Monitor ingested content with retriever-powered alerts and real-time notifications
<Frame>
<img alt="Alerts: batch ingested documents checked against retriever in parallel, matches trigger webhook, Slack, or email notifications" />
</Frame>
Alerts let you attach retriever pipelines to collections so that every ingested document is automatically checked against your search criteria. When matches are found, notifications fire to your configured channels (webhook, Slack, or email).
## How It Works
1. **Create** a retriever that defines your search criteria (e.g., semantic similarity, attribute filters)
2. **Create** an alert referencing that retriever, with notification channels configured
3. **Attach** the alert to a collection via `alert_applications` with input mappings
4. **Ingest** documents — alerts execute automatically during post-processing (Phase 3)
5. **Receive** notifications when matches are found
## Architecture
Alerts execute during the post-processing pipeline after document ingestion completes:
| Phase | System | Purpose |
| ----- | --------------------- | --------------------------------------- |
| 1 | Taxonomies | Vector-based classification |
| 2 | Clusters | Document grouping |
| **3** | **Alerts** | **Retriever execution + notifications** |
| 4 | Retriever Enrichments | Field write-back |
### Parallel Execution
Within a single alert, document-level retriever calls execute **in parallel** as independent Ray tasks. If a batch ingests 100 documents, all 100 retriever calls fan out simultaneously rather than running sequentially. Results are aggregated after all calls complete, and a single notification is sent if any document produced matches.
Multiple alerts on the same collection execute **sequentially** to avoid race conditions in notification delivery.
## Configuration
### Create an Alert
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
alert = client.alerts.create(
name="safety_content_monitor",
description="Detect potentially harmful content on ingestion",
retriever_id="ret_safety_classifier",
notification_config={
"channels": [
{
"channel_type": "webhook",
"config": {"url": "https://your-app.com/webhooks/alerts"}
},
{
"channel_type": "slack",
"config": {"channel": "#content-alerts"}
}
],
"include_matches": True,
"include_scores": True
},
enabled=True
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/alerts \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace" \
-H "Content-Type: application/json" \
-d '{
"name": "safety_content_monitor",
"retriever_id": "ret_safety_classifier",
"notification_config": {
"channels": [
{"channel_type": "webhook", "config": {"url": "https://your-app.com/webhooks/alerts"}},
{"channel_type": "slack", "config": {"channel": "#content-alerts"}}
],
"include_matches": true,
"include_scores": true
},
"enabled": true
}'
```
</CodeGroup>
### Attach to a Collection
Attach alerts to collections via `alert_applications` when creating or updating a collection:
<CodeGroup>
```python Python theme={null}
client.collections.create(
collection_name="user_uploads",
source={"type": "bucket", "bucket_ids": [bucket_id]},
feature_extractor={
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {"text": "content"},
},
alert_applications=[
{
"alert_id": alert.alert_id,
"execution_mode": "on_ingest",
"input_mappings": [
{
"input_key": "query_text",
"source": {
"source_type": "document_field",
"path": "content"
}
}
]
}
]
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/collections \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "user_uploads",
"source": {"type": "bucket", "bucket_ids": ["bucket_id"]},
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {"text": "content"}
},
"alert_applications": [
{
"alert_id": "alert_id",
"execution_mode": "on_ingest",
"input_mappings": [
{
"input_key": "query_text",
"source": {"source_type": "document_field", "path": "content"}
}
]
}
]
}'
```
</CodeGroup>
### Input Mappings
Input mappings connect document fields to retriever input parameters:
| Source Type | Description | Example |
| ---------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------- |
| `document_field` | Extract value from the ingested document using a dot-notation path | `{"source_type": "document_field", "path": "metadata.category"}` |
| `constant` | Pass a fixed value to the retriever | `{"source_type": "constant", "value": "safety_check"}` |
### Execution Modes
| Mode | Behavior |
| ----------- | ----------------------------------------------------------- |
| `on_ingest` | Execute automatically when documents are ingested (default) |
| `scheduled` | Execute on a schedule (does not trigger on ingest) |
| `on_demand` | Execute only when manually triggered |
## Notification Channels
### Webhook
```json theme={null}
{
"channel_type": "webhook",
"config": {"url": "https://your-app.com/webhooks/alerts"}
}
```
The webhook receives a JSON payload with alert details, matched documents, and scores.
### Slack
```json theme={null}
{
"channel_type": "slack",
"config": {"channel": "#alerts-channel"}
}
```
### Email
```json theme={null}
{
"channel_type": "email",
"config": {"to": ["alerts@your-company.com"]}
}
```
## Monitoring Executions
Track alert execution history to monitor performance and debug issues:
<CodeGroup>
```python Python theme={null}
executions = client.alerts.list_executions(alert_id="alert_id")
for execution in executions.results:
print(f"Execution {execution.execution_id}: "
f"triggered={execution.triggered}, "
f"matches={execution.match_count}, "
f"duration={execution.duration_ms}ms")
```
```bash cURL theme={null}
curl https://api.mixpeek.com/v1/alerts/ALERT_ID/executions \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
</CodeGroup>
## Comparison with Other Enrichment Types
| Feature | Taxonomies | Clusters | Alerts | Retriever Enrichments |
| --------- | ----------------------------- | ------------------------------- | --------------------------------- | ------------------------------------------ |
| Purpose | Vector-based classification | Document grouping | Notifications on match | Arbitrary retriever pipelines |
| Output | Label + score fields | Cluster assignments | Webhook/Slack/email notifications | Configurable field write-back |
| Phase | 1 | 2 | 3 | 4 |
| Execution | Parallel per document | Batch | Parallel per document | Sequential per document |
| Use cases | Face matching, entity linking | Segmentation, pattern discovery | Content monitoring, safety checks | LLM classification, cross-collection joins |
# Clusters
Source: https://docs.mixpeek.com/docs/enrichment/clusters
Group documents by semantic similarity or metadata attributes, then label, visualize, and enrich
<Frame>
<img alt="Cluster visualization showing document groupings with centroids and member assignments" />
</Frame>
Clusters automatically group documents into meaningful categories. Define what to cluster on, pick an algorithm, execute, and get back labeled groups you can visualize, enrich into collections, or promote to taxonomies.
## Two Clustering Types
Mixpeek supports two fundamentally different ways to cluster documents:
<CardGroup>
<Card title="Vector (Semantic)" icon="brain">
Groups documents by **embedding similarity** — what they mean, not what metadata they have. Uses vector embeddings from any extractor (text, image, multimodal) and supports 8 algorithms.
Best for: topic discovery, content deduplication, visual similarity, finding themes across modalities.
</Card>
<Card title="Attribute (Metadata)" icon="tags">
Groups documents by **metadata field values** — like a `GROUP BY` on structured columns. No embeddings needed; operates directly on payload fields.
Best for: categorical grouping, hierarchical organization by brand/category/status, faceted analytics.
</Card>
</CardGroup>
### Vector Clustering
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"cluster_name": "product_topics",
"collection_ids": ["col_products"],
"cluster_type": "vector",
"vector_config": {
"feature_uris": ["mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"],
"clustering_method": "hdbscan",
"algorithm_params": { "min_cluster_size": 10, "min_samples": 5 },
"preprocessing_steps": [
{ "method": "whitening" },
{ "method": "umap", "n_components": 50, "n_neighbors": 30 }
]
},
"llm_labeling": {
"provider": "openai",
"model_name": "gpt-4o-mini"
}
}'
```
### Attribute Clustering
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"cluster_name": "product_categories",
"collection_ids": ["col_products"],
"cluster_type": "attribute",
"attribute_config": {
"attributes": ["category", "brand"],
"hierarchical_grouping": true
}
}'
```
With `hierarchical_grouping: true`, this creates nested groups: "Electronics" containing "Apple", "Samsung", etc. Without it, you get flat groups like "Electronics\_Apple", "Electronics\_Samsung".
## Algorithms
Vector clustering supports 8 algorithms. Pick based on whether you know how many clusters to expect:
| Algorithm | Best When | Key Parameters |
| -------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------ |
| **HDBSCAN** | You don't know the number of clusters. Handles variable density, auto-detects noise. | `min_cluster_size`, `min_samples` |
| **K-Means** | You know K. Fast, spherical clusters. | `n_clusters`, `max_iter` |
| **DBSCAN** | You want density-based grouping with a fixed distance threshold. | `eps`, `min_samples` |
| **Agglomerative** | You want hierarchical merging with a specific linkage strategy. | `n_clusters`, `linkage` (ward/complete/average/single) |
| **Spectral** | Clusters have complex, non-convex shapes. | `n_clusters_spectral` |
| **Gaussian Mixture** | You need soft (probabilistic) assignments. | `n_components_gmm` |
| **Mean Shift** | You want automatic cluster count via bandwidth-based mode finding. | bandwidth params |
| **OPTICS** | Similar to DBSCAN but handles varying density better. | `eps`, `min_samples` |
<Tip>
Start with **HDBSCAN** if you don't know how many clusters to expect. Use **K-Means** when you have a target count and want fast results.
</Tip>
## Multi-Feature Strategy
When clustering on multiple embeddings (e.g., text + image), choose how to combine them:
| Strategy | How It Works | Use When |
| --------------------------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| **`concatenate`** (default) | Fuses all embeddings into a single vector, then clusters once. Supports per-feature weights. | Features are complementary and you want one set of clusters. |
| **`independent`** | Runs separate clustering per feature. Produces one output per modality. | You want to compare how text clusters vs image clusters differ. |
| **`weighted`** | Auto-learns optimal feature weights via Bayesian optimization. | You're not sure which modality matters more — let the algorithm decide. |
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"cluster_name": "multimodal_themes",
"collection_ids": ["col_ads"],
"cluster_type": "vector",
"vector_config": {
"feature_uris": [
"mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding"
],
"multi_feature_strategy": "weighted",
"clustering_method": "hdbscan",
"algorithm_params": { "min_cluster_size": 15 }
}
}'
```
## Visualization Dimensions
In Studio, the cluster scatter plot encodes three visual dimensions so you can explore cluster structure at a glance:
| Visual Dimension | What It Represents | Controlled By |
| ------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| **Position (X, Y)** | Semantic proximity — nearby points are similar in embedding space | First two coordinates from dimensionality reduction (UMAP, PCA, or t-SNE) |
| **Color** | Cluster membership — each cluster gets a distinct color | Automatic assignment based on cluster ID |
| **Dot size** | Depth (Z axis) — larger dots have higher Z values, creating a depth cue | Third coordinate when `dimension_reduction.components` is set to `3` |
To enable the size dimension, set 3 components in your dimensionality reduction config:
```json theme={null}
{
"dimension_reduction": {
"method": "umap",
"components": 3
}
}
```
<Info>
Without the third component, all dots render at the same size. With it, the Z value is linearly mapped to dot radius (10px–50px), so visually prominent points sit "closer" in the third principal axis.
</Info>
## Centroid Methods
Control how cluster centers are calculated:
| Method | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------ |
| **`mean`** (default) | Average of all member vectors. Smooth, stable centroids. |
| **`median`** | Median vector. More robust to outliers than mean. |
| **`medoid`** | The actual cluster member closest to the center. Most interpretable — the centroid is a real document. |
## Preprocessing
High-dimensional embeddings (1408d, 3072d) benefit from preprocessing before density-based algorithms. The `preprocessing_steps` field accepts an ordered list:
<AccordionGroup>
<Accordion title="Whitening (ZCA)">
Decorrelates embedding dimensions, removing redundant structure that causes density-based algorithms to over-fragment.
```json theme={null}
{ "method": "whitening", "regularization": 1e-5 }
```
</Accordion>
<Accordion title="UMAP reduction">
Reduces dimensionality while preserving neighborhood structure. Critical for HDBSCAN on high-dimensional data.
```json theme={null}
{ "method": "umap", "n_components": 50, "n_neighbors": 30, "min_dist": 0.0, "metric": "cosine" }
```
</Accordion>
<Accordion title="Chained (recommended for high-dim)">
Whitening + UMAP together typically improves HDBSCAN cluster purity by 15–30% on embeddings above 1000 dimensions.
```json theme={null}
{
"preprocessing_steps": [
{ "method": "whitening", "regularization": 1e-5 },
{ "method": "umap", "n_components": 50, "n_neighbors": 30, "min_dist": 0.0 }
]
}
```
</Accordion>
</AccordionGroup>
## Execution Modes
| Mode | What It Does | When To Use |
| -------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| **`full`** (default) | Clusters all documents from scratch. | Initial run, or when you want fresh clusters. |
| **`assign`** | Assigns new documents to existing centroids without re-clustering. O(n×k). | Streaming ingestion — run `full` periodically, `assign` for new docs in between. |
| **`composite`** | Clusters the **centroids** from prior executions together. | Cross-modality comparison, temporal drift detection, parameter tuning. |
### Incremental Assignment
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters/{cluster_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"mode": "assign",
"assignment_threshold": 0.5
}'
```
Documents below the `assignment_threshold` cosine similarity are marked as noise (`cluster_id = -1`).
### Composite Clustering
Clusters centroids from prior runs to reveal higher-order patterns. A run with 10,000 documents and 50 clusters contributes only 50 vectors, so composite execution is fast.
**Map your library along multiple dimensions at once.** A common analyst question — "how do my *visual-style* groups relate to my *messaging-theme* groups?" — is expressed with exactly this machinery, no tagging required:
1. Run **several independent clusterings on the same collection** (one per dimension — e.g. one over visual embeddings, one over transcript embeddings). Each is its own cluster resource; a collection can have as many as you need.
2. **Composite them**: pass the runs as `source_execution_ids` with `mode: "composite"`. The composite groups the centroids from every input run, so groups that land together across dimensions surface as one higher-order pattern (e.g. a "street-interview format" centroid clustering next to a "risk-reversal claim" centroid = a combination worth briefing).
3. Read the result on the cluster's **visualization map** in Studio (lasso a region to search inside it, or create a retriever scoped to a group directly from the map).
A first-class per-document cross-tab (counts of *group A × group B* membership) isn't a built-in view yet — the composite map shows how the groupings relate at the pattern level, and each input clustering keeps its own per-document groups.
<CodeGroup>
```bash cURL theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters/{cluster_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"mode": "composite",
"source_execution_ids": ["run_abc123", "run_def456"]
}'
```
```python Python theme={null}
composite = mp.clusters.execute(
cluster_id="cluster_id",
mode="composite",
source_execution_ids=[run_a.run_id, run_b.run_id],
)
```
</CodeGroup>
## Hierarchical Sub-Clustering
Enable recursive sub-clustering when top-level clusters are too broad. Each cluster with enough members is further divided using UMAP + HDBSCAN.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"cluster_name": "content_hierarchy",
"collection_ids": ["col_videos"],
"cluster_type": "vector",
"vector_config": {
"feature_uris": ["mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding"],
"clustering_method": "hdbscan",
"algorithm_params": { "min_cluster_size": 50, "min_samples": 10 },
"hierarchical": true,
"max_hierarchy_depth": 3
}
}'
```
Sub-clusters get IDs like `cl_0_sub_1_sub_0`. Each centroid includes `parent_cluster_id`, `child_cluster_ids`, and `hierarchy_level`. Example: "sports" → "basketball" → "NBA highlights".
## Quality Metrics
Every execution returns metrics that tell you whether your clusters are meaningful:
| Metric | Range | What It Means |
| ------------------------- | ------- | ---------------------------------------------------------------------------- |
| `silhouette_score` | -1 to 1 | Cluster separation quality. Above 0.5 is good. |
| `mean_cosine_to_centroid` | 0 to 1 | Average assignment confidence. Higher = tighter clusters. |
| `noise_ratio` | 0 to 1 | Fraction classified as noise. High values suggest parameters are too strict. |
| `cluster_size_entropy` | 0 to 1 | How balanced cluster sizes are. 1.0 = perfectly even. |
| `should_recluster` | 0 or 1 | Automatic recommendation based on metric thresholds. |
Per-member similarity is also tracked:
| Field | Level | Description |
| ------------------------------- | -------- | --------------------------------------------- |
| `cosine_similarity_to_centroid` | Member | How well this document fits its cluster (0–1) |
| `mean_cosine_similarity` | Centroid | Average member similarity (cohesion) |
| `min_cosine_similarity` | Centroid | Weakest member (indicates outliers) |
## LLM Labeling
Generate human-readable names, summaries, and keywords for each cluster. Control which document fields the LLM sees using **input mappings**.
### Text-Only Labeling
<CodeGroup>
```bash cURL theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"cluster_name": "script_archetypes",
"collection_ids": ["col_ad_scripts"],
"cluster_type": "vector",
"vector_config": {
"feature_uris": ["mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"],
"clustering_method": "kmeans",
"kmeans_parameters": { "n_clusters": 20 }
},
"llm_labeling": {
"provider": "openai",
"model_name": "gpt-4o-mini",
"labeling_inputs": {
"input_mappings": [
{ "input_key": "text", "source_type": "payload", "path": "headline" },
{ "input_key": "text", "source_type": "payload", "path": "primary_text" },
{ "input_key": "text", "source_type": "payload", "path": "description" }
]
}
}
}'
```
```python Python theme={null}
cluster = mp.clusters.create(
cluster_name="script_archetypes",
collection_ids=["col_ad_scripts"],
cluster_type="vector",
vector_config={
"feature_uris": ["mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"],
"clustering_method": "kmeans",
"kmeans_parameters": {"n_clusters": 20},
},
llm_labeling={
"provider": "openai",
"model_name": "gpt-4o-mini",
"labeling_inputs": {
"input_mappings": [
{"input_key": "text", "source_type": "payload", "path": "headline"},
{"input_key": "text", "source_type": "payload", "path": "primary_text"},
{"input_key": "text", "source_type": "payload", "path": "description"},
]
},
},
)
```
</CodeGroup>
### Multimodal Labeling
Send images or video alongside text for richer labels. Use a vision-capable model:
<CodeGroup>
```bash cURL theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"cluster_name": "scene_themes",
"collection_ids": ["col_ad_scenes"],
"cluster_type": "vector",
"vector_config": {
"feature_uris": ["mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding"],
"clustering_method": "kmeans",
"kmeans_parameters": { "n_clusters": 25 }
},
"llm_labeling": {
"provider": "google",
"model_name": "gemini-2.5-flash-preview-04-17",
"labeling_inputs": {
"input_mappings": [
{ "input_key": "text", "source_type": "payload", "path": "headline" },
{ "input_key": "text", "source_type": "payload", "path": "primary_text" },
{ "input_key": "image_url", "source_type": "blob", "path": "document_blobs.0.url" }
]
}
}
}'
```
```python Python theme={null}
cluster = mp.clusters.create(
cluster_name="scene_themes",
collection_ids=["col_ad_scenes"],
cluster_type="vector",
vector_config={
"feature_uris": ["mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding"],
"clustering_method": "kmeans",
"kmeans_parameters": {"n_clusters": 25},
},
llm_labeling={
"provider": "google",
"model_name": "gemini-2.5-flash-preview-04-17",
"labeling_inputs": {
"input_mappings": [
{"input_key": "text", "source_type": "payload", "path": "headline"},
{"input_key": "text", "source_type": "payload", "path": "primary_text"},
{"input_key": "image_url", "source_type": "blob", "path": "document_blobs.0.url"},
]
},
},
)
```
</CodeGroup>
### Input Mapping Reference
| Field | Description |
| ------------- | ------------------------------------------------------------------------------------------------------ |
| `input_key` | Key the LLM receives: `text`, `image_url`, `video_url`, `audio_url` |
| `source_type` | Where to pull the value: `payload` (document fields), `blob` (stored assets), `literal` (static value) |
| `path` | Dot-notation path into the document (for `payload` and `blob`) |
| `override` | Static value (only with `literal` source type) |
<Info>
Without `labeling_inputs`, the full document payload is serialized as JSON. Input mappings let you send only the fields that matter.
</Info>
### Custom Prompts and Response Shapes
Override the default prompt for domain-specific labels:
```json theme={null}
{
"llm_labeling": {
"provider": "openai",
"model_name": "gpt-4o",
"custom_prompt": "Analyze these ad clusters and label each creative archetype (e.g. 'UGC Testimonial', 'Problem-Solution Demo').",
"response_shape": {
"label": "string",
"keywords": ["string"],
"sentiment": "positive | negative | neutral",
"target_audience": "string"
}
}
}
```
### Labeling Settings
| Setting | Default | Description |
| -------------------------------- | ----------- | ---------------------------------------------------------------- |
| `max_samples_per_cluster` | auto (3–20) | Representative documents sent to the LLM per cluster. |
| `sample_text_max_length` | — | Truncate text inputs to this character length. |
| `include_summary` | `true` | Generate a longer summary alongside the label. |
| `include_keywords` | `true` | Generate keyword tags for each cluster. |
| `use_embedding_dedup` | `false` | Merge similar labels across clusters using embedding similarity. |
| `embedding_similarity_threshold` | 0.8 | Threshold for label dedup. |
| `cache_ttl_seconds` | 604800 | Cache labels for 7 days. Set to 0 to disable. |
## Enrichment
Write cluster membership back into your source collections:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/clusters/{cluster_id}/enrich" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"run_id": "run_xyz789",
"target_collection_id": "col_products_enriched",
"fields": ["cluster_id", "label", "summary", "keywords"]
}'
```
This writes `cluster_id` and labels into document payloads, enabling cluster-based filters and facets in retrievers.
## Execution & Triggers
* **Manual**: `POST /v1/clusters/{id}/execute`
* **Async job**: `POST /v1/clusters/{id}/execute/submit`
* **Automated**: create cron, interval, or event-based triggers under `/v1/clusters/triggers`
* Every run yields a `run_id` and exposes status via `GET /v1/clusters/{id}/executions`
## Artifacts
| Artifact | Endpoint | Contents |
| -------------- | ------------------------------------------------------- | ------------------------------------------------------------------ |
| Centroids | `/executions/{run_id}/artifacts?include_centroids=true` | Cluster ID, centroid vectors, counts, labels, summaries, keywords |
| Members | `/executions/{run_id}/artifacts?include_members=true` | Point IDs, reduced coordinates (`x`, `y`, `z`), cluster assignment |
| Streaming data | `/executions/{run_id}/data` | Stream centroids and members for visualization |
## Management
| Operation | Endpoint |
| ------------------ | ---------------------------------- |
| Inspect definition | `GET /v1/clusters/{id}` |
| List clusters | `POST /v1/clusters/list` |
| Execution history | `GET /v1/clusters/{id}/executions` |
| Delete | `DELETE /v1/clusters/{id}` |
## Best Practices
1. **Start with HDBSCAN + whitening + UMAP** for vector clustering on high-dimensional embeddings.
2. **Prototype on samples** — tune parameters with a small `sample_size` before running at scale.
3. **Use incremental assignment for streaming data** — `full` periodically, `assign` for new documents in between.
4. **Monitor quality metrics** — check `silhouette_score` and `noise_ratio` after each run. Recluster when `should_recluster` fires.
5. **Use attribute clustering for categorical grouping** — don't force embeddings when metadata fields already capture the structure.
6. **Try multi-feature `weighted` strategy** when combining modalities — let Bayesian optimization find the right blend.
7. **Enable 3-component dimensionality reduction** to get the depth (size) dimension in Studio visualizations.
## Clusters vs Taxonomies vs Alerts
| I want to… | Use |
| -------------------------------------------- | ------------------------------------ |
| Discover what categories exist in my data | **Clusters** |
| Apply known categories to new documents | [Taxonomies](/docs/enrichment/taxonomies) |
| Get notified when something specific appears | [Alerts](/docs/enrichment/alerts) |
| Turn discovered groups into reusable labels | Clusters → promote to taxonomy |
# Retriever Enrichments
Source: https://docs.mixpeek.com/docs/enrichment/retriever-enrichments
Run retriever pipelines on documents at ingestion time
<Frame>
<img alt="Retriever enrichments: document ingested, inputs mapped to retriever, pipeline executes, selected fields written back to document" />
</Frame>
Retriever enrichments let you attach arbitrary retriever pipelines to collections. When documents are ingested, the configured retrievers execute against each document and write selected result fields back to the document. This enables LLM classification, cross-collection joins, and multi-stage enrichment without building custom extractor plugins.
## How It Works
1. **Attach** a retriever enrichment to a collection with input mappings and write-back field configuration
2. **Ingest** documents via batch processing as usual
3. **Post-processing** executes the retriever for each document, mapping document fields to retriever inputs
4. **Write-back** extracts specified fields from retriever results and writes them to the document
Retriever enrichments run in **Phase 4** of post-processing by default (after taxonomies, clusters, and alerts), but you can configure them to run in any phase.
## Configuration
Each retriever enrichment has three main sections:
### Input Mappings
Map document fields or constant values to retriever input parameters:
```json theme={null}
{
"input_mappings": [
{
"input_key": "query",
"source": {
"source_type": "document_field",
"path": "title"
}
},
{
"input_key": "collection_id",
"source": {
"source_type": "constant",
"value": "col_reference_data"
}
}
]
}
```
### Write-Back Fields
Configure which retriever result fields to write back to documents:
```json theme={null}
{
"write_back_fields": [
{
"source_field": "category",
"target_field": "_enrichment_category",
"mode": "first"
},
{
"source_field": "related_items",
"target_field": "_related_ids",
"mode": "all_as_array"
}
]
}
```
**Write-back modes:**
| Mode | Behavior |
| -------------- | -------------------------------------------------------------- |
| `first` | Write value from the first result only (default) |
| `all_as_array` | Collect values from all results into a list |
| `concat` | Concatenate string values from all results with ", " separator |
### Execution Control
| Field | Description | Default |
| ----------------- | ------------------------------------------- | -------------------- |
| `execution_phase` | Post-processing phase (1-4) | 4 (Enrichment) |
| `priority` | Priority within phase (higher = runs first) | 0 |
| `scroll_filters` | Filter which documents to enrich | None (all documents) |
| `enabled` | Whether enrichment is active | true |
## Example: LLM Classification at Ingestion
Attach a retriever with an `llm_enrich` stage to classify documents as they're ingested:
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
# Update collection to add retriever enrichment
client.collections.update(
collection_id="col_articles",
retriever_enrichments=[
{
"retriever_id": "ret_classifier",
"input_mappings": [
{
"input_key": "query",
"source": {
"source_type": "document_field",
"path": "content"
}
}
],
"write_back_fields": [
{
"source_field": "category",
"target_field": "_llm_category",
"mode": "first"
},
{
"source_field": "sentiment",
"target_field": "_llm_sentiment",
"mode": "first"
}
],
"enabled": True
}
]
)
```
</CodeGroup>
## Example: Cross-Collection Join
Use a retriever enrichment to join data from a reference collection:
<CodeGroup>
```python Python theme={null}
client.collections.update(
collection_id="col_products",
retriever_enrichments=[
{
"retriever_id": "ret_brand_lookup",
"input_mappings": [
{
"input_key": "query",
"source": {
"source_type": "document_field",
"path": "brand_name"
}
}
],
"write_back_fields": [
{
"source_field": "brand_logo_url",
"target_field": "_brand_logo",
"mode": "first"
},
{
"source_field": "brand_category",
"target_field": "_brand_category",
"mode": "first"
}
],
"enabled": True
}
]
)
```
</CodeGroup>
<Note>
Retriever enrichments execute sequentially within each collection to avoid race conditions. For collections with many documents, enrichment time scales linearly with document count.
</Note>
## Comparison with Other Enrichment Types
| Feature | Taxonomies | Clusters | Alerts | Retriever Enrichments |
| --------- | ----------------------------- | ------------------------------- | --------------------- | ------------------------------------------ |
| Purpose | Vector-based classification | Document grouping | Notifications | Arbitrary retriever pipelines |
| Output | Label + score fields | Cluster assignments | Webhook notifications | Configurable field write-back |
| Phase | 1 | 2 | 3 | 4 (default) |
| Execution | Parallel per document | Batch | Parallel per document | Sequential per document |
| Use cases | Face matching, entity linking | Segmentation, pattern discovery | Content monitoring | LLM classification, cross-collection joins |
# Taxonomies
Source: https://docs.mixpeek.com/docs/enrichment/taxonomies
Enrich documents with similarity-based joins
Taxonomies let you attach structured metadata to documents by matching them against a reference collection. They are implemented as retriever-powered joins and can run on demand or be materialized into collections. Taxonomies are warehouse-native enrichment: the multimodal equivalent of a SQL JOIN, linking documents to canonical entities via embedding similarity rather than key equality.
<Frame>
<img alt="From clusters to taxonomies: documents are embedded into a shared vector space, unsupervised clustering discovers natural structure, an LLM labels each cluster (product reviews, technical docs, legal contracts, support tickets), and stable clusters are promoted into reusable hierarchical taxonomy nodes. The taxonomy then auto-classifies new documents at ingest, becomes search filters at query time, and reclassifies existing documents retroactively when the hierarchy changes." />
</Frame>
## Taxonomy Types
| Type | Structure | When to Use |
| ------------ | ----------------------------------- | ---------------------------------------------------- |
| Flat | Single-level reference collection | Face enrollment, entity linking, simple lookups |
| Hierarchical | Parent/child nodes with inheritance | Org charts, product categories, multi-level labeling |
Each node references a collection, retriever, and list of enrichment fields. Child nodes inherit parent properties automatically.
### Flat Taxonomy: Product Catalog Recognition
<Frame>
<img alt="Flat taxonomy showing multimodal documents matched against a product catalog reference collection" />
</Frame>
In a flat taxonomy, documents from any modality (video, image, audio, text) are matched against a single reference collection. Each document uses its appropriate feature embedding (CLIP for visual, text embeddings for audio transcripts) to find the best match. Enrichment fields (SKU, category, price) are attached when similarity exceeds the threshold.
### Hierarchical Taxonomy: Media Content Classification
<Frame>
<img alt="Hierarchical taxonomy showing media content classification with multiple levels from brand to campaign" />
</Frame>
In a hierarchical taxonomy, documents traverse multiple levels of progressive refinement. Starting from a broad brand classification (1 node), through content category (2 nodes), sport/style type (4 nodes), audience segmentation (5 nodes), to specific campaigns (6 nodes). Each level narrows the classification using different multimodal features—CLIP for brand detection, scene classification for categories, activity detection for sport types, demographic models for audiences, and campaign-specific patterns at the final level. Documents inherit all properties from parent nodes as they traverse down the tree.
## Execution Modes
| Mode | Description | Use Case |
| ------------- | --------------------------------------------------------------------------- | ------------------------------------------------------ |
| `on_demand` | Enrich documents at query time inside a retriever (`taxonomy_enrich` stage) | Exploratory workflows, testing, dynamic reference data |
| `materialize` | Batch enrichment after extraction; results persisted in the collection | Production search, low-latency retrieval, analytics |
| `retroactive` | Apply taxonomy to existing documents in a collection | Backfilling, taxonomy updates, schema migrations |
Configure execution mode via a collection's `taxonomy_applications` array or by adding a taxonomy stage to a [retriever](/docs/retrieval/retrievers).
### How Hierarchical Taxonomies Execute
Hierarchical taxonomies are executed like **Common Table Expressions (CTEs)** in SQL—each level builds on the results of the previous level, creating a recursive evaluation chain from root to leaf nodes.
```
Level 1 (Root) → Match against brand collection
↓ passes matched docs
Level 2 → Match against category collection (filtered by L1 result)
↓ passes matched docs
Level 3 → Match against subcategory collection (filtered by L2 result)
↓ passes matched docs
Level N (Leaf) → Final enrichment fields attached
```
At each level:
1. Documents that matched the parent node are passed down
2. The child node's retriever executes against its reference collection
3. Enrichment fields from matching nodes are accumulated
4. Only documents exceeding the similarity threshold continue to child nodes
This CTE-style execution ensures that a document classified as "Nike → Athletic → Running" inherits enrichment fields from all three levels, not just the leaf node.
### Application Methods
Hierarchical taxonomies can be applied through three methods:
| Method | When It Runs | Use Case |
| ---------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| **On-demand** | Query time, as a [retriever stage](/docs/retrieval/retrievers#apply-stages) | Dynamic classification, A/B testing taxonomy versions, low-volume queries |
| **Materialized** | During collection processing (post-extraction) | Production search requiring low latency, analytics dashboards |
| **Retroactive** | Manually triggered via API | Backfilling existing documents, applying updated taxonomy versions |
**On-demand** enrichment adds a `taxonomy_enrich` stage to your retriever pipeline:
```json theme={null}
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "tax_org_roles",
"min_score": 0.4
}
}
}
```
**Materialized** enrichment runs automatically after document extraction completes. Configure it in the collection's `taxonomy_applications`:
```json theme={null}
{
"taxonomy_applications": [
{
"taxonomy_id": "tax_product_hierarchy",
"execution_mode": "materialize"
}
]
}
```
**Retroactive** enrichment applies a taxonomy to documents already in the collection:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/collections/<collection_id>/apply-taxonomy" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"taxonomy_id": "tax_product_hierarchy",
"filter": {
"field": "metadata.needs_reclassification",
"operator": "eq",
"value": true
}
}'
```
Use retroactive application when:
* You've updated a taxonomy and need to reclassify existing documents
* You're migrating from a flat taxonomy to a hierarchical one
* You've added new reference data to taxonomy collections
## Internals: JOIN Stage
Taxonomies reuse the `join@v1` stage under the hood:
* **Direct join** – key-based match (`join_type: "direct"`).
* **Retriever join** – similarity match using a nested retriever (`join_type: "retriever"`).
* **Join strategies** – `replace`, `enrich`, `left`, or `append` control how fields merge.
Parallel execution (`asyncio.gather`) makes retrieval joins 10–50× faster than sequential lookups.
## Create a Flat Taxonomy
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/taxonomies" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"taxonomy_name": "employee_faces",
"taxonomy_type": "flat",
"retriever_id": "ret_face_matcher",
"input_mappings": {
"query_embedding": "mixpeek://face_identity_extractor@v1/insightface__arcface"
},
"source_collection": {
"collection_id": "col_employee_embeddings",
"enrichment_fields": [
{ "field_path": "metadata.name", "merge_mode": "enrich" },
{ "field_path": "metadata.department", "merge_mode": "enrich" }
]
}
}'
```
## Create a Hierarchical Taxonomy
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/taxonomies" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"taxonomy_name": "org_roles",
"taxonomy_type": "hierarchical",
"retriever_id": "ret_face_matcher",
"input_mappings": {
"query_embedding": "mixpeek://face_identity_extractor@v1/insightface__arcface"
},
"hierarchy": [
{
"node_id": "employees",
"collection_id": "col_employee_embeddings",
"enrichment_fields": ["metadata.employee_id", "metadata.department"]
},
{
"node_id": "executives",
"collection_id": "col_executives",
"retriever_id": "ret_executive_face",
"parent_node_id": "employees",
"enrichment_fields": ["metadata.executive_level", "metadata.budget_authority"]
}
]
}'
```
Hierarchical nodes inherit parent enrichment properties; children can override or extend them.
## Attach to a Collection
```json theme={null}
{
"taxonomy_applications": [
{
"taxonomy_id": "tax_employee_faces",
"execution_mode": "materialize"
},
{
"taxonomy_id": "tax_org_roles",
"execution_mode": "on_demand"
}
]
}
```
* Materialized enrichment updates documents \~30 seconds after ingestion completes (debounced to avoid thrashing).
* On-demand enrichment keeps documents untouched; retrievers call the taxonomy join at query time.
## Test On Demand
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/taxonomies/<taxonomy_id>/enrich" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"source_documents": [
{
"document_id": "doc_scene_123",
"mixpeek://face_identity_extractor@v1/insightface__arcface": [0.12, 0.34, ...]
}
],
"mode": "on_demand"
}'
```
## Inference Strategies
* **Manual** – Define nodes explicitly (IDs, collections, retrievers).
* **Schema-based** – Infer nodes from existing collection schemas (planned).
* **Cluster-based** – Create nodes from clustering output.
* **LLM-based** – Generate hierarchical structure from sample documents.
Combining strategies is encouraged: bootstrap via inference, then fine-tune manually.
## Monitoring
* List taxonomies: `POST /v1/taxonomies/list`
* Inspect hierarchy and node metadata: `GET /v1/taxonomies/{id}?expand_nodes=true`
* Track materialized enrichment progress via webhook events (`collection.documents.written`)
* Use retriever analytics to ensure taxonomy stages don’t dominate latency.
## Best Practices
1. **Start flat** for quick wins; layer hierarchies once value is proven.
2. **Keep enrichment minimal**—copy only fields needed at query time.
3. **Cache taxonomy stages** in retrievers when reference collections rarely change.
4. **Version taxonomies** (via snapshots) before major structural changes.
5. **Combine with clusters** to discover candidate nodes and measure coverage.
Taxonomies let you inject domain knowledge into multimodal search—link documents to canonical entities without relying on brittle key joins.
# Claude Code Skill
Source: https://docs.mixpeek.com/docs/integrations/developer-tools/claude-skill
Stand up a complete Mixpeek namespace, buckets, collections, retrievers, taxonomies, clusters, alerts, and triggers from a single slash command
The `/mixpeek` Claude Code skill is a setup wizard that turns a plain-English description of your data into a fully-configured Mixpeek workspace. Run it once, answer nine questions, and every resource is created for you via the API.
<Info>
**What is a Claude Code skill?** Skills are slash commands that extend [Claude Code](https://claude.ai/code) — Anthropic's CLI for AI-assisted development. A skill is a markdown file saved to `~/.claude/commands/` that gives Claude a specialized prompt. Install once, use from any session.
</Info>
***
## Install
One-liner install from the public Gist:
```bash theme={null}
mkdir -p ~/.claude/commands && curl -o ~/.claude/commands/mixpeek.md \
https://gist.githubusercontent.com/esteininger/95a3d92dbae12177367cb8c13126f029/raw/mixpeek.md
```
Or copy the full skill content below manually into `~/.claude/commands/mixpeek.md`:
<Accordion title="View full skill file — copy this into ~/.claude/commands/mixpeek.md">
````markdown theme={null}
---
description: Set up Mixpeek resources from scratch — namespace, buckets, collections, retrievers, taxonomies, clusters, alerts, triggers, and webhooks — via a guided interview about your data and goals
allowed-tools: Bash
argument-hint: [setup|status] [--api-key KEY]
---
# /mixpeek — Mixpeek Resource Setup Wizard
You are a Mixpeek setup assistant. Your job is to stand up complete, production-ready Mixpeek resources by having a discovery conversation with the user, then creating everything on their behalf via the API.
---
## Step 1 — API Key
The user's request: **$ARGUMENTS**
Check if an API key was passed in the arguments. Otherwise check the environment:
```bash
echo "${MIXPEEK_API_KEY:-not_set}"
```
If no key is found, ask:
> "What's your Mixpeek API key? You can find it at https://studio.mixpeek.com → Settings → API Keys."
Store as `API_KEY`. All requests go to `https://api.mixpeek.com`.
---
## Step 2 — Discovery Interview
Ask these questions conversationally. You can batch related ones. Listen carefully — answers drive every resource decision.
---
### DATA SECTION
**Q1 — What data?**
"Describe your data in plain English. What are the items?
*Examples: 'product catalog', 'security camera frames', 'support tickets', 'PDF contracts', 'social media posts with images'*"
**Q2 — Multiple datasets?**
"Do you have more than one dataset? (e.g., products AND customer reviews AND vendor images)
If yes, describe each one separately — I'll create a separate bucket and collections for each."
**Q3 — Schema per dataset**
"For each dataset, list the field names and their types:
- text / string — names, descriptions, titles, content
- image — URLs pointing to photos or images
- video — URLs pointing to video files
- audio — URLs pointing to audio files
- float / number — prices, scores, ratings
- integer / count — quantities, IDs, counts
- boolean — flags like in_stock, is_active
- date — ISO date strings
*Example: name (text), description (text), photo_url (image), price (float), in_stock (boolean)*"
**Q4 — Data location**
"Where does this data live?
- **URLs** — I have HTTP/HTTPS links to each item
- **S3** — AWS S3 bucket (provide bucket name + prefix)
- **Google Drive** — folder ID or URL
- **SharePoint / OneDrive** — site URL + folder path
- **Snowflake** — database.schema.table
- **Upload later** — I'll push data via API after setup"
---
### RETRIEVAL SECTION
**Q5 — Search & retrieval goals**
"What kinds of queries do you want to run? (pick all that apply)
a) **Semantic text search** — 'find items matching a text query'
b) **Image search by text** — 'find images that match a text description'
c) **Visual similarity** — 'find images/videos similar to this image'
d) **Cross-modal** — 'query with text and match against both text and image embeddings'
e) **Filtered search** — 'search + filter by field values (e.g., category=electronics, price<100)'
f) **Question answering** — 'ask natural language questions, get synthesized answers'
g) **Re-ranking** — 'use a cross-encoder to improve result ordering'"
---
### CLASSIFICATION SECTION
**Q6 — Taxonomy / classification?**
"Do you want to automatically classify or tag your documents with labels?
- **Flat taxonomy** — each document gets one or more labels from a flat list (e.g., IAB content categories, product types, sentiment labels). You provide example items per label as a reference collection.
- **Hierarchical taxonomy** — labels have a parent-child structure (e.g., Electronics → Smartphones → iPhone). The hierarchy can be explicit or inferred from your data.
- **None** — skip classification"
If yes: "What are the labels you want to assign? List them (e.g., 'electronics, clothing, food, sports') — or describe the hierarchy."
---
### CLUSTERING SECTION
**Q7 — Clustering / grouping?**
"Do you want to automatically group similar items together?
- **Vector clustering** — group by semantic/visual similarity using embeddings. Algorithm options:
- `hdbscan` — auto-detects number of clusters (best for unknown structure)
- `kmeans` — you specify number of clusters K
- `agglomerative` — hierarchical bottom-up grouping
- **Attribute clustering** — group by metadata field values (e.g., group by category + brand, creating 'Electronics > Apple', 'Electronics > Samsung', etc.)
- **None** — skip clustering
If clustering: Should clusters have **LLM-generated labels** (e.g., 'High-Performance Laptops' instead of 'Cluster 0')? If yes, which model? (gpt-4o-mini recommended, or claude-3-5-haiku)
Should cluster labels be written back to the source documents as enrichment fields?"
---
### AUTOMATION SECTION
**Q8 — Scheduled automation?**
"Do you want any recurring automated operations?
- **Re-cluster on a schedule** — re-run clustering daily/hourly as new data arrives
- **Re-run taxonomy enrichment on a schedule** — re-classify documents periodically
- **None** — trigger manually
If yes: how often? (hourly / every 6 hours / daily at midnight / custom cron like '0 2 * * *')"
---
### ALERTS & WEBHOOKS SECTION
**Q9 — Monitoring & alerts?**
"Do you want to be notified when specific content is found or when jobs complete?
- **Content alerts** — run a retriever query on new documents; notify if matches exceed a threshold (e.g., 'alert when prohibited content is detected', 'alert when competitor mentions appear')
- **Job completion webhooks** — get notified when batches, clusters, or taxonomy jobs complete
- **None** — skip notifications
If alerts: describe what to watch for and provide a webhook URL to receive notifications.
If webhooks: provide a URL and select event types (batch.completed, cluster.execution.completed, alert.triggered, etc.)"
---
## Step 3 — Design the Resource Plan
Use the user's answers to determine exactly what to create. Apply these rules:
### Namespace Extractors
- Any dataset has text fields → `text_extractor@v1`
- Any dataset has image fields → `image_extractor@v1`
- Any dataset has video fields → `image_extractor@v1` (video frames are images)
- Include all that apply
### Buckets (one per dataset)
Map field types to bucket schema types:
- text/string/description/title/content → `"type": "string"`
- image/photo/picture (URL) → `"type": "image"`
- video (URL) → `"type": "string"` (stored as URL reference)
- float/number/price/score → `"type": "float"`
- integer/count/quantity → `"type": "integer"`
- boolean → `"type": "string"` (serialize as "true"/"false")
- date/datetime → `"type": "string"` (ISO-8601 format)
### Collections (one per extractor type per dataset)
- Text field(s) in dataset → `{dataset}-text` collection with `text_extractor@v1`, `input_mappings: {"text": "field_name"}`
- Image field in dataset → `{dataset}-images` collection with `image_extractor@v1`, `input_mappings: {"image": "image_url_field"}`
- `field_passthrough`: all fields except the extractor input (those are stored as payload)
### Retrievers (from Q5)
- Semantic text search → `feature_search` stage, `input_mode: "text"`, text_extractor URI
- Image search by text → `feature_search` stage, `input_mode: "text"`, image_extractor URI
- Visual similarity → `feature_search` stage, `input_mode: "content"`, image_extractor URI, `value: "{{INPUT.image_url}}"`
- Cross-modal → `feature_search` stage with multiple searches (text + image URIs), fusion: "rrf"
- Filtered search → add `attribute_filter` stage after feature_search
- Q&A → `feature_search` + `llm_filter` stages chained
- Re-ranking → add `rerank` stage after feature_search
Default feature URIs (may be overridden post-batch):
- Text: `mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1`
- Image: `mixpeek://image_extractor@v1/google_siglip_base_v1`
**Always auto-detect actual URIs** from the collection's `vector_indexes` before creating retrievers.
### Taxonomies (from Q6)
Flat taxonomy needs:
- A **reference collection** — embeddings of the label examples (created from a label bucket)
- A **retriever** that searches the reference collection
- A **source collection** — the collection to enrich with labels
- `input_mappings` — how to extract the query from source documents
Hierarchical taxonomy:
- Same structure, but `taxonomy_type: "hierarchical"` with `hierarchy` dict (child_collection_id → parent_collection_id)
- Or use `inference_strategy: "llm"` with `inference_collections` to auto-infer hierarchy
### Clusters (from Q7)
Vector cluster: `cluster_type: "vector"`, `vector_config: {feature_uris: [...], clustering_method: "hdbscan"|"kmeans", ...}`
Attribute cluster: `cluster_type: "attribute"`, `attribute_config: {attributes: ["field1", "field2"], hierarchical_grouping: true|false}`
LLM labeling: include `llm_labeling: {enabled: true, model_name: "gpt-4o-mini-2024-07-18", provider: "openai"}`
Enrich source: `enrich_source_collection: true` to write cluster_id/label back to documents
### Triggers (from Q8)
For clusters: `action_type: "cluster"`, `action_config: {cluster_id: "..."}`, `trigger_type: "cron"|"interval"`
For taxonomy enrichment: `action_type: "taxonomy_enrichment"`, `action_config: {taxonomy_id: "...", collection_id: "..."}`
Cron schedule: `schedule_config: {cron_expression: "0 2 * * *", timezone: "UTC"}`
Interval: `schedule_config: {interval_seconds: 3600}` (hourly)
### Alerts (from Q9)
Alert references a retriever (the search logic lives there). When the retriever returns results, the alert fires.
Notification channels:
- Inline webhook: `{channel_type: "webhook", config: {url: "https://..."}}`
- Slack: `{channel_type: "slack", config: {channel: "#alerts"}}`
- Email: `{channel_type: "email", config: {to: ["admin@example.com"]}}`
### Webhooks (from Q9)
`POST /v1/organizations/webhooks/` with `webhook_name`, `event_types`, `channels: [{channel_type: "webhook", config: {url: "..."}}]`
Event types: `object.created`, `collection.documents.written`, `cluster.execution.completed`, `cluster.execution.failed`, `trigger.execution.completed`, `trigger.execution.failed`, `alert.triggered`, `taxonomy.created`
---
## Step 4 — Show the Plan & Confirm
Present a clear resource tree before creating anything:
```
📋 MIXPEEK SETUP PLAN — {project-name}
══════════════════════════════════════════════════════
NAMESPACE: {project-name}
Extractors: text_extractor@v1, image_extractor@v1
DATASET 1: {dataset1-name}
BUCKET: {dataset1-name}-data
Schema: field1 (string), field2 (image), field3 (float)
COLLECTION: {dataset1-name}-text
Extractor: text_extractor@v1 ← {text_field}
Passthrough: field1, field2, field3
COLLECTION: {dataset1-name}-images
Extractor: image_extractor@v1 ← {image_field}
Passthrough: field1, field2, field3
RETRIEVER: {project-name}-search
Stage 1: feature_search (text + image, RRF)
Input: query (text)
TAXONOMY: {project-name}-categories [if classification requested]
Type: flat
Labels: electronics, clothing, food, ...
Source: {collection-id}
CLUSTER: {project-name}-vector-clusters [if vector clustering requested]
Algorithm: hdbscan
Feature: text_extractor URI
LLM labels: enabled (gpt-4o-mini)
Enrich source: yes → cluster_id, cluster_label
TRIGGER: daily-recluster [if automation requested]
Action: cluster → {cluster-id}
Schedule: cron "0 2 * * *" (daily at 2am UTC)
ALERT: {alert-name} [if monitoring requested]
Retriever: {retriever-id}
Notify: webhook → https://your-endpoint.com/hook
WEBHOOK: job-notifications [if webhooks requested]
Events: cluster.execution.completed, batch.completed
URL: https://your-endpoint.com/events
══════════════════════════════════════════════════════
```
Ask: **"Does this look right? (yes / adjust X / skip Y)"**
Wait for confirmation. Let the user adjust before creating.
---
## Step 5 — Create the Resources
Use Python 3 with `httpx` (fallback to `requests` if needed). Run each as an inline script. Capture IDs from outputs.
### 5a — Namespace
```bash
python3 - <<'PYEOF'
import httpx, json, sys
API_KEY = "REPLACE_API_KEY"
BASE = "https://api.mixpeek.com"
PROJECT = "REPLACE_PROJECT_NAME"
extractors = [
{"feature_extractor_name": "text_extractor", "version": "v1"},
# {"feature_extractor_name": "image_extractor", "version": "v1"},
]
headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
resp = httpx.post(f"{BASE}/v1/namespaces", headers=headers, json={
"namespace_name": PROJECT,
"feature_extractors": extractors,
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
data = resp.json()
print(f"namespace_id={data['namespace_id']}")
PYEOF
```
Capture `namespace_id`. All subsequent requests include `X-Namespace: {namespace_id}`.
### 5b — Bucket (repeat for each dataset)
```bash
python3 - <<'PYEOF'
import httpx, json, sys
API_KEY = "REPLACE_API_KEY"
BASE = "https://api.mixpeek.com"
NS_ID = "REPLACE_NAMESPACE_ID"
DATASET = "REPLACE_DATASET_NAME"
headers = {
"Authorization": f"Bearer {API_KEY}",
"X-Namespace": NS_ID,
"Content-Type": "application/json",
}
schema_properties = {
# "field_name": {"type": "string"},
# "image_url": {"type": "image"},
# "price": {"type": "float"},
}
resp = httpx.post(f"{BASE}/v1/buckets", headers=headers, json={
"bucket_name": f"{DATASET}-data",
"bucket_schema": {"properties": schema_properties},
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
print(f"bucket_id={resp.json()['bucket_id']}")
PYEOF
```
### 5c — Data Source Setup (if not manual upload)
**S3 sync:**
```bash
python3 - <<'PYEOF'
import httpx, json
headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
conn_resp = httpx.post(f"{BASE}/v1/organizations/connections", headers=headers, json={
"name": "s3-source",
"provider_type": "s3",
"provider_config": {
"bucket": "REPLACE_S3_BUCKET",
"region": "us-east-1",
"prefix": "",
},
"test_before_save": True,
})
print("connection:", conn_resp.json().get("connection_id"))
headers["X-Namespace"] = NS_ID
sync_resp = httpx.post(f"{BASE}/v1/buckets/{BUCKET_ID}/syncs", headers=headers, json={
"connection_id": conn_resp.json()["connection_id"],
"source_path": "optional/prefix/",
"sync_mode": "continuous",
"polling_interval_seconds": 3600,
})
print("sync_id:", sync_resp.json().get("sync_config_id"))
PYEOF
```
**If URLs (manual):** tell the user to `POST /v1/buckets/{bucket_id}/objects` with:
```json
{
"field1": "value",
"blobs": [
{"property": "image_url", "type": "image", "data": "https://..."},
{"property": "description", "type": "text", "data": "text content here"}
]
}
```
### 5d — Collections (repeat for each extractor type per dataset)
**Text collection:**
```bash
python3 - <<'PYEOF'
import httpx, json, sys
resp = httpx.post(f"{BASE}/v1/collections", headers=headers, json={
"collection_name": f"{DATASET}-text",
"source": {"type": "bucket", "bucket_ids": [BUCKET_ID]},
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {"text": "REPLACE_TEXT_FIELD"},
"parameters": {},
"field_passthrough": ["REPLACE_ALL_OTHER_FIELDS"],
},
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
print(f"collection_id={resp.json()['collection_id']}")
PYEOF
```
**Image collection:**
```bash
python3 - <<'PYEOF'
import httpx, json, sys
resp = httpx.post(f"{BASE}/v1/collections", headers=headers, json={
"collection_name": f"{DATASET}-images",
"source": {"type": "bucket", "bucket_ids": [BUCKET_ID]},
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": {"image": "REPLACE_IMAGE_URL_FIELD"},
"parameters": {},
"field_passthrough": ["REPLACE_ALL_OTHER_FIELDS"],
},
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
print(f"collection_id={resp.json()['collection_id']}")
PYEOF
```
### 5e — Retrievers
**Semantic text search:**
```bash
python3 - <<'PYEOF'
import httpx, json, sys
TEXT_URI = "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
resp = httpx.post(f"{BASE}/v1/retrievers", headers=headers, json={
"retriever_name": f"{PROJECT}-search",
"collection_identifiers": [TEXT_COLLECTION_ID],
"stages": [{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": TEXT_URI,
"query": {"input_mode": "text", "value": "{{INPUT.query}}"},
"top_k": 10,
}],
"final_top_k": 5,
"fusion": "rrf",
"collection_identifiers": [TEXT_COLLECTION_ID],
},
},
}],
"input_schema": {"query": {"type": "text", "description": "Search query"}},
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
print(f"retriever_id={resp.json()['retriever']['retriever_id']}")
PYEOF
```
**Cross-modal (text query → text + image results, RRF fusion):**
```bash
python3 - <<'PYEOF'
import httpx, json, sys
TEXT_URI = "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
IMAGE_URI = "mixpeek://image_extractor@v1/google_siglip_base_v1"
ALL_COLLECTIONS = [TEXT_COLLECTION_ID, IMAGE_COLLECTION_ID]
resp = httpx.post(f"{BASE}/v1/retrievers", headers=headers, json={
"retriever_name": f"{PROJECT}-multimodal",
"collection_identifiers": ALL_COLLECTIONS,
"stages": [{
"stage_name": "multimodal_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{"feature_uri": TEXT_URI, "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 10},
{"feature_uri": IMAGE_URI, "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 10},
],
"final_top_k": 5,
"fusion": "rrf",
"collection_identifiers": ALL_COLLECTIONS,
},
},
}],
"input_schema": {"query": {"type": "text"}},
})
print(f"retriever_id={resp.json()['retriever']['retriever_id']}")
PYEOF
```
**Q&A retriever (retrieve + LLM synthesize):**
```bash
python3 - <<'PYEOF'
import httpx, json, sys
TEXT_URI = "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
resp = httpx.post(f"{BASE}/v1/retrievers", headers=headers, json={
"retriever_name": f"{PROJECT}-qa",
"collection_identifiers": [TEXT_COLLECTION_ID],
"stages": [
{
"stage_name": "retrieve_context",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": TEXT_URI, "query": {"input_mode": "text", "value": "{{INPUT.question}}"}, "top_k": 10}],
"final_top_k": 10,
"fusion": "rrf",
"collection_identifiers": [TEXT_COLLECTION_ID],
},
},
},
{
"stage_name": "synthesize_answer",
"stage_type": "transform",
"config": {
"stage_id": "llm_filter",
"parameters": {
"prompt": "Using only the retrieved documents, answer concisely: {{INPUT.question}}",
"model": "gpt-4o-mini",
"output_field": "answer",
},
},
},
],
"input_schema": {"question": {"type": "text", "description": "Question to answer from the corpus"}},
})
print(f"retriever_id={resp.json()['retriever']['retriever_id']}")
PYEOF
```
### 5f — Batch Processing
Trigger each collection separately to start feature extraction:
```bash
python3 - <<'PYEOF'
import httpx, json
for col_id in [TEXT_COLLECTION_ID]: # add IMAGE_COLLECTION_ID if applicable
r = httpx.post(f"{BASE}/v1/collections/{col_id}/trigger", headers=headers, json={}, timeout=30)
data = r.json()
print(f" {col_id}: {r.status_code} → batch_id={data.get('batch_id')} objects={data.get('object_count')}")
PYEOF
```
### 5g — Taxonomy (flat)
```bash
python3 - <<'PYEOF'
import httpx, json, sys
# Step 1: Reference bucket for label examples
ref_resp = httpx.post(f"{BASE}/v1/buckets", headers=headers, json={
"bucket_name": f"{PROJECT}-taxonomy-labels",
"bucket_schema": {"properties": {"label_name": {"type": "string"}, "description": {"type": "string"}}},
})
ref_bucket_id = ref_resp.json()["bucket_id"]
# Step 2: Upload label examples
LABELS = [
# {"label_name": "electronics", "description": "consumer electronics and gadgets",
# "blobs": [{"property": "description", "type": "text", "data": "consumer electronics and gadgets"}]}
]
for label in LABELS:
httpx.post(f"{BASE}/v1/buckets/{ref_bucket_id}/objects", headers=headers, json=label)
# Step 3: Reference collection
ref_col_resp = httpx.post(f"{BASE}/v1/collections", headers=headers, json={
"collection_name": f"{PROJECT}-taxonomy-reference",
"source": {"type": "bucket", "bucket_ids": [ref_bucket_id]},
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": {"text": "description"},
"parameters": {},
"field_passthrough": ["label_name"],
},
})
ref_col_id = ref_col_resp.json()["collection_id"]
# Step 4: Taxonomy retriever
tax_ret_resp = httpx.post(f"{BASE}/v1/retrievers", headers=headers, json={
"retriever_name": f"{PROJECT}-taxonomy-matcher",
"collection_identifiers": [ref_col_id],
"stages": [{"stage_name": "label_search", "stage_type": "filter", "config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 3}],
"final_top_k": 1,
"collection_identifiers": [ref_col_id],
},
}}],
"input_schema": {"query": {"type": "text"}},
})
tax_ret_id = tax_ret_resp.json()["retriever_id"]
# Step 5: Create taxonomy
tax_resp = httpx.post(f"{BASE}/v1/taxonomies", headers=headers, json={
"taxonomy_name": f"{PROJECT}-categories",
"description": "Automatically classify documents into predefined categories",
"config": {
"taxonomy_type": "flat",
"retriever_id": tax_ret_id,
"input_mappings": [{"input_key": "query", "source_type": "payload", "path": "REPLACE_TEXT_FIELD"}],
"source_collection": {
"collection_id": TEXT_COLLECTION_ID,
# enrichment_fields: only include if those fields already exist in the source schema
},
},
})
if tax_resp.status_code != 200:
print(f"ERROR {tax_resp.status_code}: {tax_resp.text}", file=sys.stderr); sys.exit(1)
print(f"taxonomy_id={tax_resp.json()['taxonomy_id']}")
PYEOF
```
### 5h — Clusters
**Vector cluster (HDBSCAN + LLM labels):**
```bash
python3 - <<'PYEOF'
import httpx, json, sys
TEXT_URI = "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
resp = httpx.post(f"{BASE}/v1/clusters", headers=headers, json={
"cluster_name": f"{PROJECT}-semantic-groups",
"collection_ids": [TEXT_COLLECTION_ID],
"cluster_type": "vector",
"vector_config": {
"feature_uris": [TEXT_URI],
"clustering_method": "hdbscan",
},
"llm_labeling": {"enabled": True, "provider": "openai", "model_name": "gpt-4o-mini-2024-07-18"},
"enrich_source_collection": True,
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
data = resp.json()
print(f"cluster_id={data['cluster_id']}")
exec_resp = httpx.post(f"{BASE}/v1/clusters/{data['cluster_id']}/execute", headers=headers, json={})
print(f"execution_task_id={exec_resp.json().get('task_id')}")
PYEOF
```
### 5i — Triggers
**Daily re-cluster (cron):**
```bash
python3 - <<'PYEOF'
import httpx, json
resp = httpx.post(f"{BASE}/v1/triggers", headers=headers, json={
"action_type": "cluster",
"action_config": {"cluster_id": CLUSTER_ID},
"trigger_type": "cron",
"schedule_config": {"cron_expression": "0 2 * * *", "timezone": "UTC"},
"description": "Re-cluster daily at 2am UTC",
})
# NOTE: POST /v1/triggers returns 201 Created
if resp.status_code not in (200, 201):
print(f"ERROR {resp.status_code}: {resp.text}")
else:
print(f"trigger_id={resp.json()['trigger_id']}")
PYEOF
```
### 5j — Alerts
```bash
python3 - <<'PYEOF'
import httpx, json
resp = httpx.post(f"{BASE}/v1/alerts", headers=headers, json={
"name": f"{PROJECT}-content-monitor",
"description": "Alert when specific content is detected in new documents",
"retriever_id": ALERT_RETRIEVER_ID,
"enabled": True,
"notification_config": {
"channels": [{"channel_type": "webhook", "config": {"url": "REPLACE_WEBHOOK_URL"}}],
"include_matches": True,
"include_scores": True,
},
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
print(f"alert_id={resp.json()['alert_id']}")
PYEOF
```
### 5k — Webhooks
```bash
python3 - <<'PYEOF'
import httpx, json
org_headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
resp = httpx.post(f"{BASE}/v1/organizations/webhooks/", headers=org_headers, json={
"webhook_name": f"{PROJECT}-job-notifications",
"event_types": [
"cluster.execution.completed",
"cluster.execution.failed",
"trigger.execution.completed",
"trigger.execution.failed",
"alert.triggered",
"collection.documents.written",
],
"channels": [{"channel_type": "webhook", "config": {"url": "REPLACE_WEBHOOK_URL"}}],
"enabled": True,
})
if resp.status_code != 200:
print(f"ERROR {resp.status_code}: {resp.text}", file=sys.stderr); sys.exit(1)
print(f"webhook_id={resp.json()['webhook_id']}")
PYEOF
```
---
## Step 6 — Auto-Detect Feature URIs
After triggering the collection, confirm the actual feature URIs registered:
```bash
python3 - <<'PYEOF'
import httpx, json
resp = httpx.get(f"{BASE}/v1/collections/{COLLECTION_ID}", headers=headers)
for vi in resp.json().get("vector_indexes", []):
print(f" vector: {vi.get('vector_name')} uri: {vi.get('feature_uri')}")
PYEOF
```
If the detected URI differs from the default, patch the retriever stages accordingly.
---
## Step 7 — Final Summary
After everything is created, output a complete summary:
```
✅ MIXPEEK SETUP COMPLETE — {project-name}
┌──────────────────────────────────────────────────────────┐
│ Namespace: {namespace_id} │
│ Bucket: {bucket_id} │
│ Collection: {text_col_id} (text embeddings) │
│ Collection: {image_col_id} (image embeddings) │
│ Retriever: {retriever_id} (semantic search) │
│ Taxonomy: {taxonomy_id} (flat categories) │
│ Cluster: {cluster_id} (vector HDBSCAN) │
│ Trigger: {trigger_id} (daily re-cluster) │
│ Alert: {alert_id} (content monitor) │
│ Webhook: {webhook_id} (job notifications) │
└──────────────────────────────────────────────────────────┘
📡 SEARCH YOUR DATA (once batch completes):
curl -X POST https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{"inputs": {"query": "your search here"}, "settings": {"limit": 5}}'
📚 DOCS: https://docs.mixpeek.com
```
---
## Error Handling
For any non-200 response:
1. Print the full error body
2. Explain what went wrong in plain English
3. Suggest the fix
Common errors:
- `401` → bad/missing API key
- `409 Conflict` → name already taken → ask user for a new name or offer to use the existing resource
- `422 Unprocessable Entity` → bad request body → show the exact validation error field
- `429 Too Many Requests` → wait 5s, retry once
- `400` on taxonomy with `input_mappings` → check that `path` field exists in source document payload
---
## Key API Reference
| Resource | Create | List | Execute |
|----------|--------|------|---------|
| Namespace | `POST /v1/namespaces` | `POST /v1/namespaces/list` | — |
| Bucket | `POST /v1/buckets` | `POST /v1/buckets/list` | — |
| Bucket Sync | `POST /v1/buckets/{id}/syncs` | `POST /v1/buckets/{id}/syncs/list` | `POST /v1/buckets/{id}/syncs/{sid}/trigger` |
| Collection | `POST /v1/collections` | `POST /v1/collections/list` | `POST /v1/collections/{id}/trigger` |
| Retriever | `POST /v1/retrievers` | `POST /v1/retrievers/list` | `POST /v1/retrievers/{id}/execute` |
| Taxonomy | `POST /v1/taxonomies` | `POST /v1/taxonomies/list` | `POST /v1/collections/{id}/apply-taxonomy` |
| Cluster | `POST /v1/clusters` | `POST /v1/clusters/list` | `POST /v1/clusters/{id}/execute` |
| Trigger | `POST /v1/triggers` | `POST /v1/triggers/list` | `POST /v1/triggers/{id}/execute` |
| Alert | `POST /v1/alerts` | `POST /v1/alerts/list` | — |
| Webhook | `POST /v1/organizations/webhooks/` | `POST /v1/organizations/webhooks/list` | — |
All requests except webhooks require `Authorization: Bearer {api_key}`.
All requests except namespace creation and webhooks require `X-Namespace: {namespace_id}`.
````
</Accordion>
<Tip>
After saving the file, restart Claude Code. The `/mixpeek` command will appear in tab-complete.
</Tip>
***
## Usage
```
/mixpeek
```
Or pass your API key directly to skip the first prompt:
```
/mixpeek sk-mxp-...
```
***
## What It Asks
<AccordionGroup>
<Accordion title="Q1 — What data?" icon="database">
Describe your dataset in plain English.
*Examples: "product catalog with photos and descriptions", "security camera footage", "support tickets", "PDF contracts"*
</Accordion>
<Accordion title="Q2 — Multiple datasets?" icon="layer-group">
If you have more than one dataset (e.g., products AND customer reviews AND vendor images), describe each separately. The skill creates a dedicated bucket and collection set for each.
</Accordion>
<Accordion title="Q3 — Schema" icon="table">
For each dataset, list field names and types:
| Type | Examples |
| ----------------- | ------------------------------------ |
| `text` / `string` | names, descriptions, titles, content |
| `image` | URLs to photos |
| `video` | URLs to video files |
| `float` | prices, scores, ratings |
| `integer` | quantities, IDs, counts |
| `boolean` | in\_stock, is\_active |
| `date` | ISO date strings |
</Accordion>
<Accordion title="Q4 — Where does the data live?" icon="cloud">
* **URLs** — HTTP/HTTPS links to each item
* **S3** — AWS S3 bucket with optional prefix
* **Google Drive** — folder ID or URL
* **SharePoint / OneDrive** — site URL + folder path
* **Snowflake** — database.schema.table
* **Upload later** — set up the schema now, push data later via API
</Accordion>
<Accordion title="Q5 — Search & retrieval goals" icon="magnifying-glass">
Pick all that apply: semantic text search, image search by text, visual similarity, cross-modal, filtered search, question answering, re-ranking.
</Accordion>
<Accordion title="Q6 — Classification / taxonomy?" icon="sitemap">
Flat (label list) or hierarchical (parent-child structure). You provide example items per label; the skill creates the reference collection and wiring automatically.
</Accordion>
<Accordion title="Q7 — Clustering / grouping?" icon="circle-nodes">
Vector clustering (hdbscan / kmeans / agglomerative) or attribute clustering (group by field values). Optional LLM-generated cluster labels and enrichment back to source documents.
</Accordion>
<Accordion title="Q8 — Scheduled automation?" icon="clock">
Re-cluster or re-classify on a schedule. Supports cron expressions and interval-based triggers.
</Accordion>
<Accordion title="Q9 — Monitoring & alerts?" icon="bell">
Content alerts (notify when a retriever query matches new documents) and job completion webhooks.
</Accordion>
</AccordionGroup>
***
## Resources Created
| Resource | What it does |
| ----------------- | ------------------------------------------------------------- |
| **Namespace** | Isolated workspace; one per project |
| **Bucket(s)** | Raw data storage with typed schema |
| **Collection(s)** | Processing pipeline — one per extractor type per dataset |
| **Batch** | Triggers feature extraction across all bucket objects |
| **Retriever(s)** | Multi-stage search pipeline matching your retrieval goals |
| **Taxonomy** | Flat or hierarchical classifier applied to documents |
| **Cluster** | Groups similar documents; supports LLM-generated labels |
| **Trigger** | Scheduled re-clustering or taxonomy enrichment |
| **Alert** | Fires a webhook when a retriever query matches new content |
| **Webhook** | Event notifications for job completion, object creation, etc. |
***
## Requirements
* [Claude Code](https://claude.ai/code) installed
* A Mixpeek API key from [studio.mixpeek.com](https://studio.mixpeek.com) → Settings → API Keys
* Python 3 with `httpx` (`pip install httpx`)
***
## Next Steps
<CardGroup>
<Card title="Core Concepts" icon="book" href="/overview/concepts">
Understand namespaces, collections, and documents
</Card>
<Card title="Feature Extractors" icon="microchip" href="/processing/feature-extractors">
Choose the right extractor for your data type
</Card>
<Card title="Retriever Stages" icon="filter" href="/retrieval/stages/overview">
Build custom multi-stage search pipelines
</Card>
<Card title="MCP Server" icon="plug" href="/integrations/developer-tools/mcp-server">
Connect Claude to Mixpeek via MCP for ongoing management
</Card>
</CardGroup>
# JavaScript SDK
Source: https://docs.mixpeek.com/docs/integrations/developer-tools/javascript-sdk
Official TypeScript/JavaScript SDK for the Mixpeek API
<Note>
The Mixpeek JavaScript SDK is auto-generated from our OpenAPI specification and always stays in sync with the latest API features.
</Note>
## Features
* **100% Type-Safe** - Built with TypeScript for complete type safety
* **Runtime Validation** - Zod schemas for request/response validation
* **Modern** - Supports both CommonJS and ESM
* **Promise-Based** - Clean async/await API
* **Developer-Friendly** - Intuitive method names and excellent autocomplete
## Installation
<CodeGroup>
```bash npm theme={null}
npm install mixpeek
```
```bash yarn theme={null}
yarn add mixpeek
```
```bash pnpm theme={null}
pnpm add mixpeek
```
</CodeGroup>
## Quick Start
```typescript theme={null}
import { Mixpeek } from 'mixpeek';
// Initialize the client
const client = new Mixpeek({
apiKey: process.env.MIXPEEK_API_KEY,
namespace: 'my-namespace'
});
// List collections
const collections = await client.collections.listCollections();
console.log('Collections:', collections);
// Create a collection
const newCollection = await client.collections.createCollection({
alias: 'my-collection',
description: 'My first collection'
});
// Execute a retriever
const results = await client.retrievers.executeRetriever({
retrieverId: 'ret_abc123',
query: 'find relevant documents'
});
```
## Configuration
### Environment Variables
```bash theme={null}
# Required
MIXPEEK_API_KEY=mxp_sk_your_api_key_here
# Optional
MIXPEEK_BASE_URL=https://api.mixpeek.com # Default
MIXPEEK_NAMESPACE=default # Default
```
### Constructor Options
```typescript theme={null}
const client = new Mixpeek({
apiKey: 'sk_...', // Required (or set MIXPEEK_API_KEY)
baseUrl: 'https://...', // Optional: custom API endpoint
namespace: 'my-namespace', // Optional: namespace for isolation
timeout: 30000, // Optional: request timeout in ms
axiosConfig: { // Optional: additional axios config
// Any axios configuration options
}
});
```
<Tip>
You can create API keys in the Mixpeek dashboard under Organization Settings.
</Tip>
## Core Operations
### Collections
```typescript theme={null}
// Create a collection
const collection = await client.collections.createCollection({
alias: 'my-collection',
description: 'Store multimodal documents',
metadata: { project: 'demo' }
});
// Get a collection
const retrieved = await client.collections.getCollection({
collectionIdentifier: 'my-collection'
});
// List all collections
const allCollections = await client.collections.listCollections();
// Delete a collection
await client.collections.deleteCollection({
collectionIdentifier: 'my-collection'
});
```
### Retrievers
```typescript theme={null}
// Create a retriever
const retriever = await client.retrievers.createRetriever({
retrieverName: 'semantic-search',
description: 'Search across all documents',
collectionIdentifiers: ['my-collection'],
stages: [
{
type: 'embed',
model: 'openai-text-embedding-3-small'
},
{
type: 'vector_search',
top_k: 10
}
]
});
// Execute a retriever
const results = await client.retrievers.executeRetriever({
retrieverId: retriever.retrieverId,
query: 'find relevant documents about AI'
});
// List retrievers
const retrievers = await client.retrievers.listRetrievers();
```
### Documents
```typescript theme={null}
// Upload documents to a collection
const documents = await client.documents.uploadDocuments({
collectionId: 'col_abc123',
documents: [
{
url: 's3://bucket/video.mp4',
metadata: { title: 'Demo Video' }
},
{
url: 's3://bucket/image.jpg',
metadata: { title: 'Demo Image' }
}
]
});
// Search documents
const searchResults = await client.documents.searchDocuments({
collectionId: 'col_abc123',
query: 'search query',
limit: 20
});
```
### Buckets (Object Storage)
```typescript theme={null}
// Create a bucket
const bucket = await client.buckets.createBucket({
alias: 'my-bucket',
provider: 's3',
credentials: {
accessKeyId: 'YOUR_ACCESS_KEY',
secretAccessKey: 'YOUR_SECRET_KEY',
region: 'us-east-1'
}
});
// List buckets
const buckets = await client.buckets.listBuckets();
```
## Error Handling
```typescript theme={null}
try {
const collection = await client.collections.getCollection({
collectionIdentifier: 'non-existent'
});
} catch (error) {
if (error.response) {
// API error
console.error('Status:', error.response.status);
console.error('Message:', error.response.data?.error?.message);
} else if (error.request) {
// Network error
console.error('Network error:', error.message);
} else {
// Other error
console.error('Error:', error.message);
}
}
```
## TypeScript Support
The SDK is built with TypeScript and provides full type definitions:
```typescript theme={null}
import { Mixpeek, MixpeekOptions } from 'mixpeek';
import type {
Collection,
Retriever,
CreateCollectionRequest,
CreateRetrieverRequest
} from 'mixpeek';
// All types are fully typed
const options: MixpeekOptions = {
apiKey: 'sk_...',
namespace: 'default'
};
const client = new Mixpeek(options);
// TypeScript will autocomplete and type-check all methods
const collection: Collection = await client.collections.createCollection({
alias: 'typed-collection'
// TypeScript will suggest all available fields
});
```
## Advanced Usage
### Custom Axios Configuration
```typescript theme={null}
const client = new Mixpeek({
apiKey: 'sk_...',
axiosConfig: {
timeout: 60000,
headers: {
'X-Custom-Header': 'value'
},
proxy: {
host: 'proxy.example.com',
port: 8080
}
}
});
```
### Updating Configuration
```typescript theme={null}
const client = new Mixpeek({ apiKey: 'sk_old' });
// Update API key
client.setApiKey('sk_new');
// Update namespace
client.setNamespace('new-namespace');
// Get current configuration
const config = client.getConfig();
console.log(config);
// { apiKey: 'sk_new...', baseUrl: '...', namespace: 'new-namespace' }
```
## Framework Examples
### Next.js
```typescript theme={null}
// app/api/search/route.ts
import { Mixpeek } from 'mixpeek';
import { NextResponse } from 'next/server';
const client = new Mixpeek({
apiKey: process.env.MIXPEEK_API_KEY!,
namespace: process.env.MIXPEEK_NAMESPACE
});
export async function POST(request: Request) {
const { query } = await request.json();
try {
const results = await client.retrievers.executeRetriever({
retrieverId: 'ret_abc123',
query
});
return NextResponse.json(results);
} catch (error) {
return NextResponse.json(
{ error: 'Search failed' },
{ status: 500 }
);
}
}
```
### Express
```typescript theme={null}
import express from 'express';
import { Mixpeek } from 'mixpeek';
const app = express();
const client = new Mixpeek({
apiKey: process.env.MIXPEEK_API_KEY,
namespace: process.env.MIXPEEK_NAMESPACE
});
app.post('/api/search', async (req, res) => {
try {
const results = await client.retrievers.executeRetriever({
retrieverId: 'ret_abc123',
query: req.body.query
});
res.json(results);
} catch (error) {
res.status(500).json({ error: 'Search failed' });
}
});
app.listen(3000);
```
## Resources
<CardGroup>
<Card title="NPM Package" icon="npm" href="https://www.npmjs.com/package/mixpeek">
View on NPM
</Card>
<Card title="GitHub Repository" icon="github" href="https://github.com/mixpeek/javascript-sdk">
Source code and issues
</Card>
<Card title="API Reference" icon="code" href="/api-reference">
Complete API documentation
</Card>
<Card title="OpenAPI Spec" icon="file-code" href="https://api.mixpeek.com/docs/openapi.json">
OpenAPI specification
</Card>
</CardGroup>
## Next Steps
<CardGroup>
<Card title="Quickstart" icon="rocket" href="/overview/quickstart">
Get started with Mixpeek
</Card>
<Card title="Python SDK" icon="python" href="/integrations/developer-tools/python-sdk">
Use the Python SDK
</Card>
<Card title="MCP Server" icon="robot" href="/integrations/developer-tools/mcp-server">
Use Mixpeek with AI assistants
</Card>
<Card title="Examples" icon="book-open" href="https://mixpeek.com/recipes">
View example implementations
</Card>
</CardGroup>
# MCP Server
Source: https://docs.mixpeek.com/docs/integrations/developer-tools/mcp-server
Connect Claude and AI assistants to Mixpeek via the Model Context Protocol
The Mixpeek MCP server lets AI assistants like Claude manage your entire Mixpeek workflow — creating namespaces, uploading files, building search pipelines, and querying results — all through natural language.
<Info>
**What is MCP?** The [Model Context Protocol](https://modelcontextprotocol.io) is an open standard that lets AI assistants connect to external tools and data sources. Instead of copy-pasting API calls, you describe what you want and Claude handles the rest.
</Info>
## Choose Your Server
MCP clients have context limits. Instead of loading all 43 tools, pick the scoped server that matches your workflow:
<CardGroup>
<Card title="Ingestion Server" icon="upload">
**20 tools** — Buckets, collections, and documents.
Best for data pipelines and content upload workflows.
`https://mcp.mixpeek.com/ingestion/mcp`
</Card>
<Card title="Retrieval Server" icon="magnifying-glass">
**11 tools** — Retrievers, agents, and search.
Best for RAG applications, search UIs, and agent workflows.
`https://mcp.mixpeek.com/retrieval/mcp`
</Card>
<Card title="Admin Server" icon="gear">
**17 tools** — Namespaces, taxonomies, and clusters.
Best for platform administration and enrichment.
`https://mcp.mixpeek.com/admin/mcp`
</Card>
<Card title="Full Platform" icon="layer-group">
**48 tools** — Everything in one server.
Best for power users who need all capabilities.
`https://mcp.mixpeek.com/mcp`
</Card>
</CardGroup>
<Tip>
Need just one retriever? Use the [Per-Retriever Server](#retriever-server) — it exposes a single typed `search` tool with parameters generated from your retriever's input schema.
</Tip>
***
## Setup
Add to your Claude Desktop or Claude Code config. Replace `YOUR_API_KEY` with your key from the [Mixpeek dashboard](https://mixpeek.com/dashboard).
<Tabs>
<Tab title="Ingestion">
```json theme={null}
{
"mcpServers": {
"mixpeek-ingestion": {
"url": "https://mcp.mixpeek.com/ingestion/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
</Tab>
<Tab title="Retrieval">
```json theme={null}
{
"mcpServers": {
"mixpeek-retrieval": {
"url": "https://mcp.mixpeek.com/retrieval/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
</Tab>
<Tab title="Admin">
```json theme={null}
{
"mcpServers": {
"mixpeek-admin": {
"url": "https://mcp.mixpeek.com/admin/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
</Tab>
<Tab title="Full Platform">
```json theme={null}
{
"mcpServers": {
"mixpeek": {
"url": "https://mcp.mixpeek.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
</Tab>
</Tabs>
### Stdio (local development)
Run the server locally with an optional `--scope` flag:
```bash theme={null}
# Full server (default)
python -m mixpeek_mcp.main --transport stdio
# Scoped servers
python -m mixpeek_mcp.main --transport stdio --scope ingestion
python -m mixpeek_mcp.main --transport stdio --scope retrieval
python -m mixpeek_mcp.main --transport stdio --scope admin
```
***
## Ingestion Server — 18 tools
Manage buckets, collections, and documents. Use this server when building data ingestion pipelines.
<AccordionGroup>
<Accordion title="Bucket Management (6 tools)" icon="bucket">
Buckets store your raw files (videos, images, documents) before processing.
| Tool | Description |
| --------------- | --------------------------------------------------- |
| `create_bucket` | Create a new bucket for file storage and processing |
| `list_buckets` | List all buckets in a namespace |
| `get_bucket` | Get details of a specific bucket |
| `update_bucket` | Update bucket configuration |
| `delete_bucket` | Delete a bucket and all its objects |
| `upload_object` | Upload an object (file) to a bucket from a URL |
</Accordion>
<Accordion title="Collection Management (7 tools)" icon="layer-group">
Collections define how your data is processed — which feature extractor runs, what embeddings are generated. Each collection has exactly **one** feature extractor.
| Tool | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------- |
| `create_collection` | Create a collection with a feature extractor, source config, and optional taxonomy/cluster/alert applications |
| `list_collections` | List all collections in a namespace |
| `get_collection` | Get collection details |
| `update_collection` | Update collection configuration |
| `clone_collection` | Clone an existing collection with optional overrides |
| `trigger_collection` | Trigger the processing pipeline on bucket objects |
| `delete_collection` | Delete a collection and all its documents |
**Available feature extractors:**
| Extractor | Description |
| -------------------------- | --------------------------------- |
| `text_extractor` | Text embeddings (multilingual E5) |
| `image_extractor` | Image embeddings (CLIP, SigLIP) |
| `multimodal_extractor` | Text + image joint embeddings |
| `face_identity_extractor` | Face detection and recognition |
| `document_graph_extractor` | Document structure extraction |
| `sentiment_classifier` | Sentiment analysis |
| `web_scraper` | Web page content extraction |
| `course_content_extractor` | Video/course content processing |
</Accordion>
<Accordion title="Document Management (5 tools)" icon="file">
Documents are the processed records stored in your namespace with extracted features and embeddings.
| Tool | Description |
| ----------------- | ------------------------------------------- |
| `create_document` | Create a new document in a collection |
| `list_documents` | List documents in a collection with filters |
| `get_document` | Get a specific document by ID |
| `update_document` | Update a document's data |
| `delete_document` | Delete a document from a collection |
</Accordion>
</AccordionGroup>
**Example prompts:**
```
"Create a bucket called marketing-videos, then create a collection with the
multimodal extractor and trigger processing"
"Upload this CSV to my data-imports bucket: https://example.com/products.csv"
```
***
## Retrieval Server — 11 tools
Search and query your data. Use this server for RAG applications, search UIs, and agent workflows.
<AccordionGroup>
<Accordion title="Retriever Management (7 tools)" icon="filter">
Retrievers are multi-stage search pipelines. Chain stages together to search, filter, rerank, and enrich results.
| Tool | Description |
| ------------------- | ------------------------------------------------------- |
| `create_retriever` | Create a multi-stage search pipeline |
| `list_retrievers` | List all retrievers in a namespace |
| `get_retriever` | Get retriever configuration and all stages |
| `update_retriever` | Update retriever metadata (name, description, tags) |
| `clone_retriever` | Clone an existing retriever with optional modifications |
| `execute_retriever` | Execute a retriever with inputs and get search results |
| `delete_retriever` | Delete a retriever |
**29+ available stages** across 5 categories:
| Category | Stage IDs |
| ---------- | -------------------------------------------------------------------------------------------------- |
| **Filter** | `feature_search`, `attribute_filter`, `llm_filter`, `query_expand`, `agent_search` |
| **Sort** | `sort_relevance`, `sort_attribute`, `rerank`, `mmr`, `score_normalize` |
| **Reduce** | `limit`, `group_by`, `aggregate`, `summarize`, `sample`, `deduplicate`, `cluster` |
| **Apply** | `json_transform`, `api_call`, `web_search`, `sql_lookup`, `cross_compare`, `unwind`, `rag_prepare` |
| **Enrich** | `llm_enrich`, `document_enrich`, `taxonomy_enrich`, `code_execution`, `web_scrape` |
Stages support template variables: `{{INPUT.field}}`, `{{DOC.field}}`, `{{STAGE.field}}`, `{{CONTEXT.field}}`.
</Accordion>
<Accordion title="Agent & Conversation (3 tools)" icon="robot">
Conversational AI sessions with retriever-backed responses.
| Tool | Description |
| ---------------------- | --------------------------------------------------- |
| `create_agent_session` | Create a new conversational agent session |
| `send_agent_message` | Send a message to an agent session and get response |
| `get_agent_history` | Get conversation history for an agent session |
</Accordion>
<Accordion title="Search (1 tool)" icon="magnifying-glass">
| Tool | Description |
| ------------------ | ----------------------------------------------------------------------------------- |
| `search_namespace` | Search across all resources in a namespace (buckets, collections, retrievers, etc.) |
</Accordion>
</AccordionGroup>
**Example prompts:**
```
"Create a retriever that does a feature search on my product-demos collection,
reranks the top 50 down to 10 with Cohere, and adds a 2-sentence summary"
"Execute my product-search retriever with query 'red running shoes under $100'"
```
***
## Admin Server — 14 tools
Manage namespaces, taxonomies, and clusters. Use this server for platform administration and data enrichment.
<AccordionGroup>
<Accordion title="Namespace Management (5 tools)" icon="folder">
Namespaces are isolated workspaces. Each namespace maps to its own vector namespace in [MVS](https://mixpeek.com/mvs).
| Tool | Description |
| ------------------ | --------------------------------------------------------------- |
| `create_namespace` | Create a new workspace for organizing collections and resources |
| `list_namespaces` | List all namespaces in your organization |
| `get_namespace` | Get namespace details by ID or name |
| `update_namespace` | Update namespace configuration |
| `delete_namespace` | Delete a namespace and all its resources |
</Accordion>
<Accordion title="Taxonomy Management (5 tools)" icon="sitemap">
Taxonomies are hierarchical classification systems you can apply to documents.
| Tool | Description |
| ------------------ | ---------------------------------------------- |
| `create_taxonomy` | Create a hierarchical classification taxonomy |
| `list_taxonomies` | List all taxonomies |
| `get_taxonomy` | Get taxonomy details |
| `execute_taxonomy` | Apply taxonomy classification to document data |
| `delete_taxonomy` | Delete a taxonomy |
</Accordion>
<Accordion title="Cluster Management (4 tools)" icon="circle-nodes">
Clusters group similar documents together for discovery and organization.
| Tool | Description |
| ----------------- | ------------------------------------------ |
| `create_cluster` | Create a document clustering configuration |
| `list_clusters` | List all clusters |
| `execute_cluster` | Execute clustering algorithm on collection |
| `delete_cluster` | Delete a cluster configuration |
</Accordion>
</AccordionGroup>
**Example prompts:**
```
"Create a new namespace called production-catalog"
"Create an IAB taxonomy for content classification, then run it on my articles collection"
"Cluster the documents in my product-images collection into 10 groups"
```
***
## Retriever Server
The Retriever MCP server is a lightweight server scoped to a **single retriever**. It reads your retriever's `input_schema` at startup and generates a typed `search` tool whose parameters match exactly — so the AI assistant knows what inputs are available without any guesswork.
### Tools
| Tool | Description |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `search` | Execute the retriever. Parameters are generated from the retriever's `input_schema` — including the correct field names, types, required flags, enums, and descriptions. Pagination parameters (`page`, `page_size`) are added automatically. |
| `describe` | Returns structured metadata: retriever ID, name, collections, input fields, and stage configuration. |
| `explain` | Returns a human-readable explanation of the pipeline: what each stage does, in what order. |
### How Dynamic Schema Works
When the server starts, it fetches your retriever's configuration and converts its `input_schema` into a JSON Schema for the `search` tool. For example, if your retriever has:
```json theme={null}
{
"input_schema": {
"query": { "type": "text", "required": true, "description": "Search query" },
"category": {
"type": "string",
"required": false,
"enum": ["electronics", "clothing", "home"],
"description": "Filter by category"
}
}
}
```
The `search` tool will expose `query` (required string) and `category` (optional enum) as typed parameters — plus `page` and `page_size` for pagination.
<Note>
If your retriever's `input_schema` has a field named `page` or `page_size`, the pagination parameters are automatically renamed to `_pagination_page` and `_pagination_page_size` to avoid conflicts.
</Note>
### Setup
<Tabs>
<Tab title="Claude Desktop (stdio)">
```json theme={null}
{
"mcpServers": {
"my-search": {
"command": "mixpeek-mcp-retriever",
"args": [
"--retriever-id", "ret_abc123",
"--namespace-id", "ns_xyz789",
"--api-key", "YOUR_API_KEY"
]
}
}
}
```
Requires the `mixpeek-mcp-retriever` CLI installed:
```bash theme={null}
pip install mixpeek # includes the CLI
```
</Tab>
<Tab title="HTTP (deployed)">
Run the server as an HTTP endpoint for remote access:
```bash theme={null}
mixpeek-mcp-retriever \
--retriever-id ret_abc123 \
--namespace-id ns_xyz789 \
--api-key YOUR_API_KEY \
--transport http \
--port 8081
```
Then connect your MCP client to `http://your-host:8081`.
**HTTP Endpoints:**
| Method | Path | Description |
| ------ | ------------- | ------------------------------------------------ |
| `GET` | `/health` | Health check with retriever info |
| `GET` | `/tools` | List all 3 tools with schemas |
| `POST` | `/tools/call` | Execute a tool (requires `Authorization` header) |
</Tab>
<Tab title="Environment Variables">
All settings can be configured via environment variables (prefixed with `RETRIEVER_MCP_`):
```bash theme={null}
export RETRIEVER_MCP_RETRIEVER_ID=ret_abc123
export RETRIEVER_MCP_NAMESPACE_ID=ns_xyz789
export RETRIEVER_MCP_API_KEY=YOUR_API_KEY
export RETRIEVER_MCP_TRANSPORT=stdio # or "http"
export RETRIEVER_MCP_HTTP_PORT=8081
```
Then just run `mixpeek-mcp-retriever` with no arguments.
</Tab>
</Tabs>
### Example Conversation
Once connected, you can interact naturally:
```
You: "What does this retriever do?"
Claude: [calls describe] → "This is 'Product Search' — it searches your
products collection by text query with an optional category filter,
using feature search → attribute filter → reranking."
You: "Search for wireless headphones under electronics"
Claude: [calls search with query="wireless headphones", category="electronics"]
→ Returns top 10 matching products with scores and metadata.
You: "Explain the pipeline stages"
Claude: [calls explain] → "1. feature_search: embeds your query and finds
the top 50 matches. 2. attribute_filter: filters by category field.
3. rerank: reranks with Cohere down to the top 10."
```
***
## Authentication & Security
All MCP servers use your existing Mixpeek API key with the same permissions as the REST API.
* **HTTP transport:** Pass the API key in the `Authorization: Bearer` header. The server extracts it and injects it into every tool call.
* **Stdio transport:** Set the `MIXPEEK_API_KEY` environment variable or pass `api_key` in tool arguments.
* Same RBAC permissions as the REST API
* Rate limiting per organization
* Audit logging for all operations
* TLS encryption on the hosted server
<Warning>
**Keep your API key secure.** Never commit keys to version control. For the Retriever Server, prefer environment variables over CLI arguments in production.
</Warning>
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Claude / AI App │
└──────────────┬──────────────────────────────────────────┘
│ MCP Protocol (Streamable HTTP / Stdio)
▼
┌─────────────────────────────────────────────────────────┐
│ Mixpeek MCP Server │
│ ┌──────────┐ ┌───────────┐ ┌─────────┐ ┌──────────┐ │
│ │ Full │ │ Ingestion │ │Retrieval│ │ Admin │ │
│ │ / (43) │ │ /ing (18) │ │/ret (11)│ │/adm (14) │ │
│ └────┬─────┘ └─────┬─────┘ └────┬────┘ └────┬─────┘ │
│ └──────────────┴────────────┴───────────┘ │
│ Tool Handlers │
└──────────────┬──────────────────────────────────────────┘
│ ┌──────────────────┐
│ │ Retriever Server │
│ │ (per-retriever) │
│ │ 3 tools │
│ └────────┬─────────┘
└──────────┬─────────────────┘
│ Direct service calls
▼
┌──────────────────────┐
│ Mixpeek Services │
│ MongoDB · MVS │
│ Redis · S3 · Ray │
└──────────────────────┘
```
The scoped servers share the same codebase and tool handlers — scoping controls which tools are registered, not how they execute. Each scoped sub-app is mounted at its path prefix (`/ingestion`, `/retrieval`, `/admin`) while the full server handles root-level requests.
## Troubleshooting
<AccordionGroup>
<Accordion title="Claude can't connect to the MCP server">
* Verify the URL is correct (e.g. `https://mcp.mixpeek.com/ingestion/mcp`)
* Check that the `Authorization` header format is `Bearer YOUR_API_KEY`
* Restart Claude Desktop or Claude Code after changing config
</Accordion>
<Accordion title="Tools return 'Unauthorized' or 'Invalid API key'">
* Verify your API key at [mixpeek.com/dashboard](https://mixpeek.com/dashboard)
* Check that the key has permissions for the namespace you're accessing
* Make sure there are no extra spaces in the key
</Accordion>
<Accordion title="Tool not found (404)">
* You may be calling a tool on the wrong scoped server (e.g. `execute_retriever` on `/ingestion`)
* Check `GET /tools` on the scoped endpoint to see available tools
* Use the full server (`/`) if you need all tools
</Accordion>
<Accordion title="Retriever Server fails to start">
* Ensure `--retriever-id` and `--namespace-id` are correct
* Verify the API key has access to that namespace
* Check that the retriever exists: `GET /v1/retrievers/{id}`
</Accordion>
<Accordion title="Search returns empty results">
* Confirm your collection has processed documents (not just uploaded files)
* Check that the retriever's `feature_uri` matches your collection's extractor
* Try a broader query or remove optional filters
</Accordion>
<Accordion title="Slow response times">
* Large file uploads depend on file size and network
* Multi-stage retrievers with LLM enrichment or reranking take more time
* Check [status.mixpeek.com](https://status.mixpeek.com) for service issues
</Accordion>
</AccordionGroup>
## Next Steps
<CardGroup>
<Card title="Core Concepts" icon="book" href="/overview/concepts">
Understand namespaces, collections, and documents
</Card>
<Card title="Feature Extractors" icon="microchip" href="/processing/feature-extractors">
Choose the right extractor for your data
</Card>
<Card title="Retriever Stages" icon="filter" href="/retrieval/stages/overview">
Learn what each pipeline stage does
</Card>
<Card title="API Reference" icon="code" href="/api-reference">
Full REST API documentation
</Card>
</CardGroup>
# Mixpeek CLI
Source: https://docs.mixpeek.com/docs/integrations/developer-tools/mixpeek-cli
Command-line interface for building, testing, and deploying custom extractors
## Installation
```bash theme={null}
pip install mixpeek
mixpeek --version
```
## Quick Start
```bash theme={null}
# 1. Create plugin
mixpeek plugin init my_extractor --category text
# 2. Edit processors/core.py with your logic
# 3. Test locally
cd my_extractor && mixpeek plugin test
# 4. Publish
mixpeek plugin publish --namespace ns_xxx
```
See [Custom Extractors](/docs/processing/custom-extractors) for full extractor development guide.
## Configuration
```bash theme={null}
export MIXPEEK_API_KEY="mxp_sk_your_api_key"
export MIXPEEK_NAMESPACE="ns_your_namespace"
```
| Option | Environment Variable | Description |
| ------------ | -------------------- | -------------------------------------------------------------------------- |
| `--api-key` | `MIXPEEK_API_KEY` | Your Mixpeek API key |
| `--base-url` | `MIXPEEK_BASE_URL` | API base URL (default: [https://api.mixpeek.com](https://api.mixpeek.com)) |
## Commands
### `mixpeek plugin init`
Create a new plugin from template.
```bash theme={null}
mixpeek plugin init <name> [options]
```
| Option | Description |
| --------------- | ----------------------------------------------------------- |
| `--category` | `text`, `image`, `video`, `audio`, `document`, `multimodal` |
| `--description` | Plugin description |
| `--author` | Author name |
| `--output` | Output directory |
```bash theme={null}
# Examples
mixpeek plugin init sentiment_analyzer --category text
mixpeek plugin init face_detector --category image --description "Detect faces"
```
### `mixpeek plugin test`
Validate and test plugin locally.
```bash theme={null}
mixpeek plugin test [options]
```
| Option | Description |
| --------------- | ------------------------------- |
| `--path` | Plugin directory (default: `.`) |
| `--sample-data` | JSON/CSV file with test data |
| `--verbose` | Detailed output |
**Validates:**
* Structure (manifest.py, pipeline.py exist)
* Schemas (valid Pydantic models)
* Pipeline (`build_steps()` callable)
* Tests (runs pytest if tests/ exists)
```bash theme={null}
# Examples
mixpeek plugin test
mixpeek plugin test --path ./my_extractor --verbose
mixpeek plugin test --sample-data samples.json
```
### `mixpeek plugin publish`
Upload and deploy plugin to Mixpeek.
```bash theme={null}
mixpeek plugin publish [options]
```
| Option | Description |
| ------------- | -------------------------- |
| `--path` | Plugin directory |
| `--namespace` | Target namespace ID |
| `--dry-run` | Validate without uploading |
**What happens:**
1. Validates structure and schemas
2. Runs security scan
3. Creates .tar.gz archive
4. Uploads to S3 via presigned URL
5. Confirms and triggers deployment
```bash theme={null}
# Examples
mixpeek plugin publish
mixpeek plugin publish --namespace ns_abc123 --dry-run
```
### `mixpeek plugin list`
List plugins in namespace.
```bash theme={null}
mixpeek plugin list [options]
```
| Option | Description |
| ------------- | --------------------------------------- |
| `--namespace` | Namespace ID |
| `--source` | `all`, `builtin`, `custom`, `community` |
```bash theme={null}
mixpeek plugin list --source custom
```
## Plugin Structure
```
my_extractor/
├── manifest.py # Metadata + schemas
├── pipeline.py # Batch processing
├── realtime.py # HTTP endpoint (optional, Enterprise)
└── processors/
└── core.py # Your logic
```
### manifest.py
```python theme={null}
from pydantic import BaseModel, Field
from typing import List
class MyInput(BaseModel):
text: str
class MyOutput(BaseModel):
embedding: List[float]
class MyParams(BaseModel):
threshold: float = Field(default=0.5)
metadata = {
"feature_extractor_name": "my_extractor",
"version": "1.0.0",
"description": "My extractor",
"category": "text",
}
input_schema = MyInput
output_schema = MyOutput
parameter_schema = MyParams
supported_input_types = ["text"]
features = [
{
"feature_name": "my_embedding",
"feature_type": "embedding",
"embedding_dim": 384,
"distance_metric": "cosine",
},
]
```
### processors/core.py
```python theme={null}
from dataclasses import dataclass
import pandas as pd
@dataclass
class MyConfig:
threshold: float = 0.5
class MyProcessor:
def __init__(self, config: MyConfig, progress_actor=None):
self.config = config
self._model = None
def _load_model(self):
if self._model is None:
from sentence_transformers import SentenceTransformer
self._model = SentenceTransformer("all-MiniLM-L6-v2")
def __call__(self, batch: pd.DataFrame) -> pd.DataFrame:
self._load_model()
texts = batch["text"].fillna("").tolist()
batch["my_embedding"] = self._model.encode(texts).tolist()
return batch
```
### pipeline.py
```python theme={null}
from typing import Any, Dict, Optional
from engine.plugins.extractors.pipeline import (
PipelineDefinition, ResourceType, RowCondition, StepDefinition, build_pipeline_steps
)
from .manifest import MyParams, metadata
from .processors.core import MyConfig, MyProcessor
def build_steps(
extractor_request: Any,
container: Optional[Any] = None,
base_steps: Optional[list] = None,
**kwargs
) -> Dict[str, Any]:
params = MyParams(**(extractor_request.extractor_config.parameters or {}))
steps = [
StepDefinition(
service_class=MyProcessor,
resource_type=ResourceType.CPU,
config=MyConfig(threshold=params.threshold),
condition=RowCondition.IS_TEXT,
),
]
pipeline = PipelineDefinition(name=metadata["feature_extractor_name"], version=metadata["version"], steps=steps)
return {"steps": (base_steps or []) + build_pipeline_steps(pipeline), "prepare": lambda ds: ds}
```
### realtime.py (Enterprise)
```python theme={null}
from typing import Any, Dict
class RealtimeHandler:
def __init__(self):
self._model = None
def predict(self, request: Dict[str, Any]) -> Dict[str, Any]:
if self._model is None:
from sentence_transformers import SentenceTransformer
self._model = SentenceTransformer("all-MiniLM-L6-v2")
text = request.get("text", "")
embedding = self._model.encode([text])[0].tolist()
return {"embedding": embedding}
```
## Resource Types
| Type | Use For |
| ------------------ | ------------------------------- |
| `ResourceType.CPU` | Text embeddings, classification |
| `ResourceType.GPU` | Local models (Whisper, CLIP) |
| `ResourceType.API` | External APIs (OpenAI, Vertex) |
## Row Conditions
```python theme={null}
RowCondition.IS_TEXT # text/* MIME types
RowCondition.IS_IMAGE # image/* MIME types
RowCondition.IS_VIDEO # video/* MIME types
RowCondition.IS_AUDIO # audio/* MIME types
RowCondition.IS_PDF # application/pdf
RowCondition.ALWAYS # All rows (default)
```
## Security Constraints
Plugins are scanned before deployment. **Forbidden:**
| Pattern | Reason |
| ------------------------- | --------------- |
| `subprocess`, `os.system` | Shell execution |
| `eval`, `exec` | Dynamic code |
| `socket` | Direct network |
| `ctypes` | Memory access |
| `__import__` | Dynamic imports |
## Using Your Plugin
After publishing:
```python theme={null}
client.collections.create(
collection_name="my_collection",
source={"type": "bucket", "bucket_ids": ["bkt_..."]},
feature_extractor={
"feature_extractor_name": "my_extractor",
"version": "1.0.0",
"parameters": {"threshold": 0.7}
}
)
```
## API Reference
| Endpoint | Method | Description |
| -------------------------------------------------- | ------ | -------------------------------- |
| `/v1/namespaces/{id}/plugins/uploads` | POST | Get presigned upload URL |
| `/v1/namespaces/{id}/plugins/uploads/{id}/confirm` | POST | Confirm upload |
| `/v1/namespaces/{id}/plugins` | GET | List plugins |
| `/v1/namespaces/{id}/plugins/{id}` | GET | Get plugin details |
| `/v1/namespaces/{id}/plugins/{id}` | DELETE | Delete plugin |
| `/v1/namespaces/{id}/plugins/{id}/deploy` | POST | Deploy for realtime (Enterprise) |
| `/v1/namespaces/{id}/plugins/{id}/status` | GET | Check deployment status |
## Troubleshooting
| Issue | Solution |
| ------------------- | ------------------------------------------ |
| Plugin not found | Check namespace, wait for deployment |
| Import errors | Ensure `__init__.py` files exist |
| Security scan fails | Remove forbidden patterns |
| Validation errors | Check manifest.py exports metadata/schemas |
Debug mode:
```bash theme={null}
mixpeek plugin test --verbose
mixpeek plugin publish --dry-run
```
# Python SDK
Source: https://docs.mixpeek.com/docs/integrations/developer-tools/python-sdk
Official Python SDK for the Mixpeek API
<Note>
The Mixpeek Python SDK is auto-generated from our OpenAPI specification and always stays in sync with the latest API features.
</Note>
## Installation
Install via pip:
```bash theme={null}
pip install mixpeek
```
**Requirements:** Python 3.9+
## Quick Start
```python theme={null}
import mixpeek
from mixpeek.rest import ApiException
# Configure the client
configuration = mixpeek.Configuration(
host="https://api.mixpeek.com"
)
# Create an API client
with mixpeek.ApiClient(configuration) as api_client:
# Create an instance of the Collections API
collections_api = mixpeek.CollectionsApi(api_client)
try:
# List collections
collections = collections_api.list_collections(
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
print("Collections:", collections)
except ApiException as e:
print(f"Exception: {e}")
```
## Authentication
All API requests require authentication using a Bearer token:
```python theme={null}
authorization = "Bearer mxp_sk_xxxxxxxxxxxxx" # Your API key
x_namespace = "ns_xxxxxxxxxxxxx" # Your namespace ID or custom name
```
<Tip>
You can create API keys in the Mixpeek dashboard under Organization Settings.
</Tip>
## Core Operations
### Collections
```python theme={null}
import mixpeek
from mixpeek.rest import ApiException
configuration = mixpeek.Configuration(host="https://api.mixpeek.com")
with mixpeek.ApiClient(configuration) as api_client:
collections_api = mixpeek.CollectionsApi(api_client)
# Create a collection
create_request = mixpeek.CreateCollectionRequest(
alias="my-collection",
description="Store multimodal documents"
)
collection = collections_api.create_collection(
create_collection_request=create_request,
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
print(f"Created collection: {collection.collection_id}")
# Get a collection
retrieved = collections_api.get_collection(
collection_identifier="my-collection",
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
# List all collections
all_collections = collections_api.list_collections(
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
# Delete a collection
collections_api.delete_collection(
collection_identifier="my-collection",
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
```
### Retrievers
```python theme={null}
import mixpeek
with mixpeek.ApiClient(configuration) as api_client:
retrievers_api = mixpeek.RetrieversApi(api_client)
# Create a retriever
create_request = mixpeek.CreateRetrieverRequest(
retriever_name="semantic-search",
description="Search across all documents",
collection_identifiers=["my-collection"],
stages=[
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://universal_extractor@v1/gemini-embedding-2",
"query": {"input_mode": "text", "value": "{{INPUT.query_text}}"},
"top_k": 10
}],
"final_top_k": 10
}
}
}
]
)
retriever = retrievers_api.create_retriever(
create_retriever_request=create_request,
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
# Execute a retriever
execute_request = mixpeek.ExecuteRetrieverRequest(
query="find relevant documents about AI"
)
results = retrievers_api.execute_retriever(
retriever_id=retriever.retriever_id,
execute_retriever_request=execute_request,
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
print(f"Found {len(results.documents)} results")
```
### Adhoc Retrievers
Execute retrievers on-the-fly without creating them first:
```python theme={null}
import mixpeek
with mixpeek.ApiClient(configuration) as api_client:
adhoc_api = mixpeek.AdhocRetrieversApi(api_client)
# Execute adhoc retriever
adhoc_request = mixpeek.AdhocExecuteRequest(
query="search query",
collection_identifiers=["my-collection"],
stages=[
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://universal_extractor@v1/gemini-embedding-2",
"query": {"input_mode": "text", "value": "{{INPUT.query_text}}"},
"top_k": 10
}],
"final_top_k": 10
}
}
}
]
)
results = adhoc_api.execute_adhoc_retriever(
adhoc_execute_request=adhoc_request,
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
```
### Documents
```python theme={null}
import mixpeek
with mixpeek.ApiClient(configuration) as api_client:
documents_api = mixpeek.DocumentsApi(api_client)
# Upload documents
upload_request = mixpeek.UploadDocumentsRequest(
collection_id="col_abc123",
documents=[
{
"url": "s3://bucket/video.mp4",
"metadata": {"title": "Demo Video"}
},
{
"url": "s3://bucket/image.jpg",
"metadata": {"title": "Demo Image"}
}
]
)
documents = documents_api.upload_documents(
upload_documents_request=upload_request,
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
```
## Standalone Namespaces (Bring Your Own Vectors)
Standalone namespaces let you store and query **precomputed vectors** directly —
no buckets, collections, or extractors required. This is the fastest path for
agents that already have embeddings. The high-level `Mixpeek` client exposes the
MVS primitives ergonomically; every call sends your namespace as the
`X-Namespace` header automatically.
```python theme={null}
from mixpeek import Mixpeek
mp = Mixpeek(api_key="mxp_sk_xxxxxxxxxxxxx", namespace="my-vectors")
# 1. Create a standalone namespace with one or more vector configs.
mp.namespaces.create(
namespace_id="my-vectors",
mode="standalone",
vector_configs=[{"name": "text_8", "dimension": 8, "metric": "cosine"}],
)
# 2. Upsert documents with your own vectors (direct write, no pipeline).
mp.namespaces.documents.upsert(
namespace_id="my-vectors",
documents=[
{
"document_id": "doc-1",
"vectors": {"text_8": [0.11, 0.32, -0.38, -0.41, 0.006, 0.34, -0.40, -0.52]},
"payload": {"body": "hello world", "category": "docs"},
}
],
# Optional: request a write token for read-your-writes consistency.
options={"write_token": True},
)
# 3. Query is unified on retrievers. Create a retriever with a feature_search
# stage and execute it. The X-Namespace header is sent for you.
retriever = mp.retrievers.create(
retriever_name="byov-search",
stages=[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"final_top_k": 5,
"searches": [
{
"feature_uri": "text_8",
"query": {"input_mode": "vector", "value": "{{INPUT.qv}}"},
"top_k": 5,
}
],
},
},
}
],
input_schema={"qv": {"type": "array", "required": True}},
)
results = mp.retrievers.execute(
retriever["retriever_id"],
inputs={"qv": [0.11, 0.32, -0.38, -0.41, 0.006, 0.34, -0.40, -0.52]},
)
```
<Tip>
The removed `POST /v1/search` endpoint has been replaced by this retriever
create + execute flow. `Mixpeek.search(namespace_id=..., queries=[...])` is a
shortcut that builds and runs an ephemeral feature\_search retriever for you.
</Tip>
### Read-your-writes after a direct upsert
A direct upsert is durable immediately, but retriever reads are eventually
consistent by default. To read your own just-written document, pass the
`write_token` from the upsert response back as the `X-Write-Token` header on
execute — it routes the read to the primary shard and bypasses caches:
```python theme={null}
res = mp.namespaces.documents.upsert(
namespace_id="my-vectors",
documents=[{"document_id": "doc-2", "vectors": {"text_8": [...]}, "payload": {}}],
options={"write_token": True},
)
# res["write_token"] -> send as X-Write-Token on the next retriever execute.
```
### Promote to managed (auto-embedding)
Promote a standalone namespace to **managed** mode to map a vector index to an
inference service. After promotion the same `feature_search` stage accepts
`input_mode: "text"` and auto-embeds queries — no client-side embedding needed.
```python theme={null}
import urllib3, json
http = urllib3.PoolManager()
http.request(
"POST",
"https://api.mixpeek.com/v1/namespaces/my-vectors/promote",
headers={"Authorization": "Bearer mxp_sk_xxxxxxxxxxxxx", "Content-Type": "application/json"},
body=json.dumps({
"vector_mappings": [
{"existing_index": "text_8", "inference_service": "intfloat/multilingual-e5-large-instruct"}
]
}).encode(),
)
```
<Note>
Before promotion, a standalone vector index has no inference mapping, so
`input_mode: "text"` / `"content"` queries return an actionable 400 telling you
to promote the namespace or use `input_mode=vector` with a precomputed embedding.
</Note>
## Error Handling
The SDK raises `ApiException` for every non-2xx response:
```python theme={null}
from mixpeek.rest import ApiException
try:
collection = collections_api.get_collection(
collection_identifier="non-existent",
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
except ApiException as e:
print(f"Status code: {e.status}")
print(f"Reason: {e.reason}")
print(f"Body: {e.body}")
# Handle specific error codes
if e.status == 404:
print("Collection not found")
elif e.status == 401:
print("Authentication failed - check your API key")
elif e.status == 403:
print("Access forbidden - check your namespace")
```
## Available APIs
The SDK includes these API classes:
* `AdhocRetrieversApi` - Execute retrievers without saving them
* `AgentSessionsApi` - Manage AI agent sessions
* `AlertsApi` - Configure and manage alerts
* `AnalyticsApi` - Access usage and performance analytics
* `BucketsApi` - Manage object storage buckets
* `CollectionsApi` - Manage document collections
* `DocumentsApi` - Upload and manage documents
* `NamespacesApi` - Manage multi-tenant namespaces
* `OrganizationsApi` - Organization management
* `PluginsApi` - Configure and manage plugins
* `RetrieversApi` - Create and execute retrievers
* `TaxonomiesApi` - Manage classification taxonomies
* `WebhooksApi` - Configure webhook integrations
## Configuration Options
### Custom Host
```python theme={null}
configuration = mixpeek.Configuration(
host="https://custom.api.endpoint.com"
)
```
### Timeouts
```python theme={null}
configuration = mixpeek.Configuration(
host="https://api.mixpeek.com"
)
configuration.timeout = 60 # Set timeout to 60 seconds
```
## Resources
<CardGroup>
<Card title="PyPI Package" icon="python" href="https://pypi.org/project/mixpeek/">
View on PyPI
</Card>
<Card title="GitHub Repository" icon="github" href="https://github.com/mixpeek/python-sdk">
Source code and issues
</Card>
<Card title="API Reference" icon="code" href="/api-reference">
Complete API documentation
</Card>
<Card title="OpenAPI Spec" icon="file-code" href="https://api.mixpeek.com/docs/openapi.json">
OpenAPI specification
</Card>
</CardGroup>
## Next Steps
<CardGroup>
<Card title="Quickstart" icon="rocket" href="/overview/quickstart">
Get started with Mixpeek
</Card>
<Card title="JavaScript SDK" icon="js" href="/integrations/developer-tools/javascript-sdk">
Use the JavaScript/TypeScript SDK
</Card>
<Card title="MCP Server" icon="robot" href="/integrations/developer-tools/mcp-server">
Use Mixpeek with AI assistants
</Card>
<Card title="Examples" icon="book-open" href="https://mixpeek.com/recipes">
View example implementations
</Card>
</CardGroup>
# SDK Overview
Source: https://docs.mixpeek.com/docs/integrations/developer-tools/sdk-usage
Official SDKs for Python, JavaScript/TypeScript, and more
<Note>
Mixpeek provides auto-generated SDKs that are always in sync with our latest API features.
</Note>
## Official SDKs
Choose the SDK that matches your development environment:
<CardGroup>
<Card title="Python SDK" icon="python" href="/integrations/developer-tools/python-sdk">
Python SDK with type hints and typed exceptions
</Card>
<Card title="JavaScript SDK" icon="js" href="/integrations/developer-tools/javascript-sdk">
TypeScript-first SDK for Node.js and browser environments
</Card>
</CardGroup>
## Quick Comparison
| Feature | Python SDK | JavaScript SDK |
| ------------------ | ---------- | ----------------- |
| Package Manager | pip | npm/yarn/pnpm |
| Type Safety | Type hints | Full TypeScript |
| Async Support | ✅ | ✅ (Promise-based) |
| Auto-generated | ✅ | ✅ |
| Runtime Validation | ✅ | ✅ (Zod schemas) |
## Installation
<CodeGroup>
```bash Python theme={null}
pip install mixpeek
```
```bash JavaScript theme={null}
npm install mixpeek
```
</CodeGroup>
## Quick Start Examples
### Python
```python theme={null}
import mixpeek
from mixpeek.rest import ApiException
configuration = mixpeek.Configuration(host="https://api.mixpeek.com")
with mixpeek.ApiClient(configuration) as api_client:
collections_api = mixpeek.CollectionsApi(api_client)
collections = collections_api.list_collections(
authorization="Bearer mxp_sk_xxxxxxxxxxxxx",
x_namespace="ns_xxxxxxxxxxxxx"
)
```
[View full Python SDK documentation →](/docs/integrations/developer-tools/python-sdk)
### JavaScript/TypeScript
```typescript theme={null}
import { Mixpeek } from 'mixpeek';
const client = new Mixpeek({
apiKey: process.env.MIXPEEK_API_KEY,
namespace: 'my-namespace'
});
const collections = await client.collections.listCollections();
```
[View full JavaScript SDK documentation →](/docs/integrations/developer-tools/javascript-sdk)
## Using the REST API Directly
If you prefer to use the REST API without an SDK, you can make direct HTTP requests:
```python theme={null}
import requests
# Create a namespace
response = requests.post(
"https://api.mixpeek.com/v1/namespaces",
headers={
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
},
json={
"namespace_name": "my-workspace",
"description": "Production workspace"
}
)
namespace = response.json()
print(f"Created namespace: {namespace['namespace_id']}")
```
### Example: Direct API with Python
```python theme={null}
import requests
from typing import Optional
class MixpeekClient:
def __init__(self, api_key: str, base_url: str = "https://api.mixpeek.com"):
self.api_key = api_key
self.base_url = base_url
self.headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
def create_collection(
self,
namespace_id: str,
collection_name: str,
feature_extractors: list
) -> dict:
"""Create a new collection with feature extractors."""
response = requests.post(
f"{self.base_url}/v1/collections",
headers={**self.headers, "X-Namespace": namespace_id},
json={
"collection_name": collection_name,
"feature_extractors": feature_extractors
}
)
response.raise_for_status()
return response.json()
def execute_retriever(
self,
namespace_id: str,
retriever_id: str,
inputs: dict,
limit: Optional[int] = None
) -> dict:
"""Execute a retriever to search documents."""
params = {"limit": limit} if limit else {}
response = requests.post(
f"{self.base_url}/v1/retrievers/{retriever_id}/execute",
headers={**self.headers, "X-Namespace": namespace_id},
json={"inputs": inputs},
params=params
)
response.raise_for_status()
return response.json()
# Usage
client = MixpeekClient(api_key="sk_...")
# Create a collection
collection = client.create_collection(
namespace_id="ns_abc123",
collection_name="videos",
feature_extractors=[{
"feature_extractor_name": "multimodal_extractor",
"version": "v1"
}]
)
# Execute a search
results = client.execute_retriever(
namespace_id="ns_abc123",
retriever_id="ret_xyz789",
inputs={"query": "machine learning"},
limit=10
)
```
### Example: Direct API with JavaScript
```javascript theme={null}
class MixpeekClient {
constructor(apiKey, baseUrl = "https://api.mixpeek.com") {
this.apiKey = apiKey;
this.baseUrl = baseUrl;
this.headers = {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json"
};
}
async createCollection(namespaceId, collectionName, featureExtractors) {
const response = await fetch(`${this.baseUrl}/v1/collections`, {
method: "POST",
headers: {
...this.headers,
"X-Namespace": namespaceId
},
body: JSON.stringify({
collection_name: collectionName,
feature_extractors: featureExtractors
})
});
if (!response.ok) {
throw new Error(`API error: ${response.statusText}`);
}
return response.json();
}
async executeRetriever(namespaceId, retrieverId, inputs, limit = null) {
const url = new URL(`${this.baseUrl}/v1/retrievers/${retrieverId}/execute`);
if (limit) url.searchParams.set("limit", limit);
const response = await fetch(url, {
method: "POST",
headers: {
...this.headers,
"X-Namespace": namespaceId
},
body: JSON.stringify({ inputs })
});
if (!response.ok) {
throw new Error(`API error: ${response.statusText}`);
}
return response.json();
}
}
// Usage
const client = new MixpeekClient("sk_...");
// Create a collection
const collection = await client.createCollection(
"ns_abc123",
"videos",
[{
feature_extractor_name: "multimodal_extractor",
version: "v1"
}]
);
// Execute a search
const results = await client.executeRetriever(
"ns_abc123",
"ret_xyz789",
{ query: "machine learning" },
10
);
```
## Next Steps
<CardGroup>
<Card title="Python SDK" icon="python" href="/integrations/developer-tools/python-sdk">
Full Python SDK documentation
</Card>
<Card title="JavaScript SDK" icon="js" href="/integrations/developer-tools/javascript-sdk">
Full JavaScript SDK documentation
</Card>
<Card title="API Reference" icon="code" href="/api-reference">
Explore all API endpoints
</Card>
<Card title="MCP Server" icon="robot" href="/integrations/developer-tools/mcp-server">
Use Mixpeek with AI assistants
</Card>
<Card title="Quickstart" icon="rocket" href="/overview/quickstart">
Get started with Mixpeek
</Card>
<Card title="Examples" icon="book-open" href="https://mixpeek.com/recipes">
View example implementations
</Card>
</CardGroup>
# Search Widget
Source: https://docs.mixpeek.com/docs/integrations/search-widget
Drop-in React component for AI-powered multimodal search — add search to any site in minutes
`@mixpeek/search-js` is a drop-in React component that connects to any published Mixpeek retriever. It provides a full search UI with keyboard shortcuts, filters, AI answers, and streaming results — no backend code required on your side.
<CodeGroup>
```bash npm theme={null}
npm install @mixpeek/search-js
```
```bash yarn theme={null}
yarn add @mixpeek/search-js
```
```bash pnpm theme={null}
pnpm add @mixpeek/search-js
```
</CodeGroup>
## Quick Start
<Steps>
<Step title="Set up your search backend">
Use the [docs search quickstart](#quickstart-docs-search) to provision a complete pipeline in one API call, or [create a retriever manually](/docs/retrieval/retrievers) and [publish it](#publishing-a-retriever).
</Step>
<Step title="Install the widget">
```bash theme={null}
npm install @mixpeek/search-js
```
</Step>
<Step title="Add to your app">
```tsx theme={null}
import { MixpeekSearch } from "@mixpeek/search-js";
import "@mixpeek/search-js/styles.css";
export default function App() {
return (
<MixpeekSearch
projectKey="your-retriever-slug"
placeholder="Search..."
theme="auto"
accentColor="#6366f1"
/>
);
}
```
</Step>
</Steps>
Users press `Cmd+K` (or `Ctrl+K`) to open the search modal. Results stream in from your retriever pipeline.
## Props
| Prop | Type | Default | Description |
| ------------------ | ----------------------------- | ------------- | ---------------------------------------------- |
| `projectKey` | `string` | **required** | Published retriever slug or `ret_sk_*` API key |
| `placeholder` | `string` | `"Search..."` | Input placeholder text |
| `maxResults` | `number` | `10` | Maximum results to show |
| `theme` | `"light" \| "dark" \| "auto"` | `"auto"` | Color theme |
| `accentColor` | `string` | `"#6366f1"` | Accent color (hex) |
| `position` | `"modal" \| "inline"` | `"modal"` | Modal overlay or inline embed |
| `keyboardShortcut` | `boolean` | `true` | Enable Cmd+K / Ctrl+K |
| `showPoweredBy` | `boolean` | `true` | Show "Search by Mixpeek" badge |
| `enableAIAnswer` | `boolean` | `false` | Show AI-generated answer with citations |
| `enableShareLinks` | `boolean` | `false` | Enable shareable search URLs |
| `defaultOpen` | `boolean` | `false` | Start with modal open |
| `defaultFilters` | `Record<string, unknown>` | - | Default filter values on mount |
### Callbacks
| Prop | Type | Description |
| ------------------ | ------------------------------ | ---------------------------------- |
| `onSearch` | `(query: string) => void` | Fires when a search is performed |
| `onResultClick` | `(result, index) => void` | Fires when a result is clicked |
| `onZeroResults` | `(query: string) => void` | Fires when no results are found |
| `onFilterChange` | `(filterInputs) => void` | Fires when filters change |
| `transformResults` | `(results[]) => results[]` | Transform results before rendering |
| `renderResult` | `(result, index) => ReactNode` | Custom result renderer |
## CDN Usage (No Build Step)
For sites without a build system, load the widget via CDN:
```html theme={null}
<link rel="stylesheet" href="https://cdn.mixpeek.com/search/v1/mixpeek-search.css" />
<script
src="https://cdn.mixpeek.com/search/v1/loader.js"
data-project-key="your-retriever-slug"
data-mount="search-container">
</script>
<div id="search-container"></div>
```
## Filters
The widget includes built-in filter components for facets, ranges, and LLM-powered smart filtering.
### Facet Filter
Single or multi-select dropdown:
```tsx theme={null}
<MixpeekSearch projectKey="my-search">
<FacetFilter
field="category"
label="Category"
options={[
{ label: "Electronics", value: "electronics" },
{ label: "Clothing", value: "clothing" },
]}
multiple={true}
/>
</MixpeekSearch>
```
### Range Filter
Numeric min/max slider:
```tsx theme={null}
<RangeFilter
field="price"
label="Price"
min={0}
max={1000}
step={10}
unit="$"
/>
```
### Smart Filter (LLM-based)
Natural language filtering powered by the retriever's LLM filter stage:
```tsx theme={null}
<SmartFilter
label="Describe what you want"
placeholder="e.g. red shoes under $50"
/>
```
## AI Answers
Enable `enableAIAnswer` to show an LLM-generated answer with citations above search results. This requires an `agent_search` or `rag_prepare` stage in your retriever.
```tsx theme={null}
<MixpeekSearch
projectKey="docs-search"
enableAIAnswer={true}
/>
```
## Hooks
Access search state from any child component:
```tsx theme={null}
import { useSearchKit } from "@mixpeek/search-js";
function MyComponent() {
const {
query,
results,
isLoading,
aiAnswer,
isOpen,
open,
close,
search,
filterInputs,
setFilter,
clearFilters,
hasActiveFilters,
} = useSearchKit();
return <div>{results.length} results for "{query}"</div>;
}
```
**Available hooks:** `useSearchKit`, `useSearch`, `useFilters`, `useKeyboardShortcut`, `useRecentSearches`.
## Setting Up the Backend
The widget needs a published retriever to connect to. There are two paths:
### Quickstart: Docs Search
Provision a complete search pipeline in one API call. This creates a namespace, bucket, collection (with web scraper + text embeddings), retriever, and published endpoint automatically:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/quickstart/docs-search \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"site_url": "https://docs.example.com",
"site_name": "Example Docs"
}'
```
The response includes a retriever slug and embed snippet ready to paste:
```json theme={null}
{
"retriever_slug": "example-docs-search",
"api_key": "ret_sk_...",
"embed_snippet": "<MixpeekSearch projectKey=\"example-docs-search\" />"
}
```
**Options:** `enable_code_search` (default: true), `enable_image_search` (default: false), `max_pages` (default: 200), `max_depth` (default: 3).
### Bootstrap CLI
The `@mixpeek/react-searchkit` package includes a CLI to scaffold a retriever with search, filter, and RAG stages:
```bash theme={null}
npx mixpeek-bootstrap --api-key YOUR_API_KEY --slug my-site-search
```
This creates a retriever with `feature_search`, `attribute_filter`, and `rag_prepare` stages pre-configured.
### Manual Setup
For full control, create each resource yourself:
<Steps>
<Step title="Create a namespace and bucket">
Set up storage and enable the feature extractors you need ([guide](/docs/ingestion/namespaces)).
</Step>
<Step title="Create a collection">
Configure a collection with a feature extractor (e.g., `text_extractor`, `web_scraper`, `multimodal_extractor`) and trigger processing on your data ([guide](/docs/ingestion/collections)).
</Step>
<Step title="Create and publish a retriever">
Build a retriever with the stages you need, then publish it to get a slug for the widget ([guide](/docs/retrieval/retrievers)).
</Step>
</Steps>
### Publishing a Retriever
Once you have a retriever, publish it to make it accessible to the widget:
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
result = client.retrievers.publish(
retriever_id="ret_abc123",
public_name="my-search",
)
print(result.public_url) # https://mxp.co/r/my-search
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers/ret_abc123/publish \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"public_name": "my-search"}'
```
</CodeGroup>
The `public_name` becomes the `projectKey` you pass to the widget.
## Authentication
The widget supports two authentication modes:
| Mode | `projectKey` value | When to use |
| ------------------ | --------------------- | ------------------------------------------------ |
| **Public slug** | `"my-retriever-slug"` | Public-facing search (no auth needed) |
| **Scoped API key** | `"ret_sk_..."` | Authenticated access with a `retrieverSlug` prop |
```tsx theme={null}
// Public slug (most common)
<MixpeekSearch projectKey="my-retriever-slug" />
// Scoped API key
<MixpeekSearch projectKey="ret_sk_..." retrieverSlug="my-retriever-slug" />
```
## Examples
The search widget is used on [mixpeek.com](https://mixpeek.com) as the site-wide search in the navigation bar, powered by `@mixpeek/search-js` with the `mixpeek-blog-search` retriever slug.
```tsx theme={null}
<MixpeekSearch
projectKey="mixpeek-blog-search"
placeholder="Search..."
theme={isScrolled ? "light" : "dark"}
accentColor="#6366f1"
maxResults={10}
showPoweredBy={true}
keyboardShortcut={true}
transformResults={(results) =>
results.map((r) => ({
id: r.document_id || r.id,
title: r.title || r.page_title || "Mixpeek",
content: r.content || r.text || "",
page_url: r.page_url || r.url || null,
image_url: r.image_url || null,
score: r.score,
}))
}
/>
```
## Exported Components
For building fully custom search experiences, the package exports composable sub-components:
| Component | Description |
| --------------- | ----------------------------------- |
| `SearchButton` | Standalone search trigger button |
| `SearchModal` | Search modal container |
| `SearchInput` | Input field |
| `SearchResults` | Results list |
| `ResultCard` | Individual result card |
| `AIAnswer` | AI-generated answer panel |
| `FilterPanel` | Filter container |
| `FacetFilter` | Select/multi-select filter |
| `RangeFilter` | Min/max range slider |
| `SmartFilter` | LLM-powered natural language filter |
| `PoweredBy` | "Search by Mixpeek" badge |
| `ShareLink` | Shareable search URL generator |
| `ZeroResults` | Empty state placeholder |
| `IntentCTA` | Enterprise CTA capture |
## Related
<CardGroup>
<Card title="Retrievers" icon="filter" href="/retrieval/retrievers">
Build the retriever pipeline behind your widget
</Card>
<Card title="Retriever Stages" icon="layer-group" href="/retrieval/stages/overview">
Configure search, reranking, and enrichment stages
</Card>
<Card title="Interactions" icon="chart-line" href="/retrieval/interactions">
Use interaction data to improve search relevance
</Card>
<Card title="JavaScript SDK" icon="js" href="/integrations/developer-tools/javascript-sdk">
Full SDK for programmatic access
</Card>
</CardGroup>
# Analytics & Performance
Source: https://docs.mixpeek.com/docs/operations/analytics-overview
Interpret metrics, optimize pipelines, and tune retriever performance
Mixpeek's Analytics API provides granular visibility into retrieval performance, cache efficiency, feature extraction throughput, and inference latency. Use these metrics to identify bottlenecks, validate optimizations, and allocate budgets effectively.
## Analytics Categories
<CardGroup>
<Card title="Retrieval Performance" icon="magnifying-glass-chart">
Track latency, cache hit rates, and stage-level breakdowns for retrievers.
</Card>
<Card title="Infrastructure Metrics" icon="server">
Monitor API response times, Engine throughput, and inference service health.
</Card>
<Card title="Feature Extraction" icon="microchip">
Measure extractor execution time, failure rates, and batch processing efficiency.
</Card>
<Card title="User Signals" icon="chart-line">
Analyze interaction patterns (clicks, long views, feedback) to tune relevance.
</Card>
</CardGroup>
## Key Metrics Explained
### Retrieval Metrics
| Metric | API Endpoint | Use Case |
| ------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **Retriever Performance** | `GET /v1/analytics/retrievers/{retriever_id}/performance` | Overall latency percentiles (p50, p95, p99), cache hit rate, error rate |
| **Stage Breakdown** | `GET /v1/analytics/retrievers/{retriever_id}/stages` | Per-stage execution time to identify slow stages (e.g., LLM generation vs vector search) |
| **Retriever Signals** | `GET /v1/analytics/retrievers/{retriever_id}/signals` | User interactions aggregated by result position, document, or session |
| **Slowest Queries** | `GET /v1/analytics/retrievers/{retriever_id}/slowest` | Identify outlier executions with full input payloads for debugging |
### Infrastructure Metrics
| Metric | API Endpoint | Use Case |
| ------------------------- | ----------------------------------------- | ------------------------------------------------------------- |
| **API Performance** | `GET /v1/analytics/api/performance` | Request latency by endpoint and status code |
| **Engine Performance** | `GET /v1/analytics/engine/performance` | Ray task execution time, queue depth, worker utilization |
| **Inference Performance** | `GET /v1/analytics/inference/performance` | Model latency (embedding, LLM, classification) and throughput |
### Extraction Metrics
| Metric | API Endpoint | Use Case |
| ------------------------- | ----------------------------------------------------------- | --------------------------------------------------------------- |
| **Extractor Performance** | `GET /v1/analytics/extractors/{extractor_name}/performance` | Avg processing time per object, failure rate, retry count |
| **Batch Efficiency** | Derived from extractor + engine metrics | Compare batch sizes vs throughput to optimize batching strategy |
### Cache Metrics
| Metric | API Endpoint | Use Case |
| --------------------- | ------------------------------------- | -------------------------------------------------- |
| **Cache Performance** | `GET /v1/analytics/cache/performance` | Hit rate, average TTL, eviction rate, memory usage |
### Usage & Cost Metrics
| Metric | API Endpoint | Use Case |
| ----------------- | ----------------------------------------------- | ------------------------------------------------------------------- |
| **Usage Summary** | `GET /v1/analytics/usage/summary` | Usage (in dollars) by resource type (API calls, inference, storage) |
| **Org Usage** | `GET /v1/organizations/usage` | Per-org breakdown for billing and quota enforcement |
| **API Key Usage** | `GET /v1/organizations/usage/api-keys/{key_id}` | Attribute costs to specific services or teams |
## Optimization Workflows
### 1. Diagnose Slow Retrievers
**Symptoms:** High p95/p99 latency, user complaints about wait times
**Steps:**
1. Call `/analytics/retrievers/{retriever_id}/performance` to get overall latency distribution
2. Check `cache_hit_rate` – if low, consider increasing `cache_config.ttl_seconds` or caching more stages
3. Call `/analytics/retrievers/{retriever_id}/stages` to identify the bottleneck stage
4. Common culprits:
* **LLM generation stages** – consider smaller models, prompt caching, or async processing
* **Large limit values** – reduce `limit` in early stages and use rerankers
* **Complex filters** – move structured filters before expensive search stages
5. Call `/analytics/retrievers/{retriever_id}/slowest` to inspect outlier queries with full inputs
### 2. Improve Cache Hit Rates
**Target:** >70% cache hit rate for common queries
**Strategies:**
* Enable caching at the retriever level: `cache_config.enabled = true`
* Increase TTL for stable queries: `cache_config.ttl_seconds = 3600`
* Use `cache_stage_names` to selectively cache expensive stages (e.g., web search, LLM)
* Normalize inputs (lowercase, trim whitespace) before hashing for cache keys
* Monitor `/analytics/cache/performance` to track hit rate trends
### 3. Optimize Feature Extraction
**Symptoms:** Batch processing takes hours, high extractor failure rate
**Steps:**
1. Check `/analytics/extractors/{extractor_name}/performance` for avg execution time
2. Compare across extractors – identify if one is significantly slower
3. Strategies:
* **Batch sizing** – 100-1000 objects optimal; adjust based on object size
* **Parallelism** – increase Ray workers or use GPU instances for vision/LLM extractors
* **Retry logic** – review `__missing_features` in documents to identify flaky extractors
* **Model selection** – swap heavy models for distilled versions (e.g., `distilbert` vs `bert-large`)
4. Monitor `/analytics/engine/performance` for worker saturation
### 4. Tune Inference Budgets
**Goal:** Balance cost and latency
**Steps:**
1. Review `/analytics/usage/summary` to see inference spend by model
2. Identify high-volume, low-value queries (e.g., exploratory searches)
3. Apply budget limits at the retriever level:
```json theme={null}
{
"budget_limits": {
"max_inference_calls": 10,
"max_llm_tokens": 2000,
"max_execution_time_ms": 5000
}
}
```
4. Use cheaper models for initial ranking (e.g., `multilingual-e5-base`) and expensive rerankers only for top-K results
### 5. Use User Signals for Relevance Tuning
**Workflow:**
1. Instrument your app to send interactions via `/v1/retrievers/{retriever_id}/interactions`
* `click`, `long_view`, `positive_feedback`, `negative_feedback`
2. Query `/analytics/retrievers/{retriever_id}/signals` to see:
* Click-through rate by result position
* Documents with high negative feedback
* Queries with zero engagement
3. Use insights to:
* Adjust stage ordering (e.g., move filters earlier to reduce noise)
* Update taxonomy mappings or cluster definitions
* Fine-tune reranker prompts or scoring functions
4. A/B test changes by creating retriever variants and comparing signal distributions
## Recommended Dashboards
### Production Health Dashboard
**Metrics to display:**
* API p95 latency (target: \<500ms)
* Cache hit rate (target: >70%)
* Error rate by endpoint (target: \<1%)
* Engine queue depth (alert if >100 pending tasks)
* Inference quota remaining (% of plan)
**Refresh interval:** 1 minute
### Retriever Performance Dashboard
**Metrics to display:**
* Per-retriever p95 latency (grouped by retriever\_id)
* Cache hit rate by retriever
* Top 5 slowest queries (with input previews)
* Stage breakdown for critical retrievers
* Click-through rate trends (weekly aggregation)
**Refresh interval:** 5 minutes
### Cost Optimization Dashboard
**Metrics to display:**
* Inference cost per retriever execution (derived: `inference_calls * model_cost`)
* Storage growth rate (documents + features + cache)
* API key usage breakdown (top consumers)
* Batch processing cost per object (extractor time \* worker cost)
**Refresh interval:** Daily
## Alerting Recommendations
<AccordionGroup>
<Accordion title="Critical: API Error Rate >5%">
**Trigger:** `/analytics/api/performance` shows error rate >5% over 5 min\
**Action:** Check health endpoint `/v1/health`, inspect logs for MVS/Mongo connectivity issues
</Accordion>
<Accordion title="Warning: Retriever p95 Latency >2s">
**Trigger:** Specific retriever p95 >2000ms for 10 min\
**Action:** Review stage breakdown, disable non-critical stages temporarily, increase cache TTL
</Accordion>
<Accordion title="Warning: Cache Hit Rate <50%">
**Trigger:** Cache hit rate drops below 50% for 30 min\
**Action:** Review cache config, check if query patterns changed (seasonality, A/B tests)
</Accordion>
<Accordion title="Info: High Inference Quota Usage">
**Trigger:** 80% of monthly inference quota consumed\
**Action:** Review top consumers via `/analytics/usage/summary`, throttle exploratory retrievers
</Accordion>
<Accordion title="Critical: Engine Queue Depth >500">
**Trigger:** Ray queue depth exceeds 500 pending tasks\
**Action:** Scale Ray workers, pause batch submissions, investigate slow extractors
</Accordion>
</AccordionGroup>
## Analytics API Patterns
### Time-Windowed Queries
Most endpoints accept `start_time` and `end_time` filters (ISO 8601):
```bash theme={null}
GET /v1/analytics/retrievers/{retriever_id}/performance?start_time=2025-10-01T00:00:00Z&end_time=2025-10-31T23:59:59Z
```
### Aggregation Levels
Control granularity with `group_by`:
```bash theme={null}
# Group by day
GET /v1/analytics/api/performance?group_by=day
# Group by retriever_id
GET /v1/analytics/usage/summary?group_by=retriever_id
```
### Tuning Recommendations
The `/analytics/analyze-for-tuning` endpoint provides automated suggestions:
```bash theme={null}
POST /v1/analytics/retrievers/{retriever_id}/analyze
```
**Response includes:**
* Recommended stage order changes
* Cache config suggestions (TTL, stage-level caching)
* Budget limit recommendations
* Model swap suggestions (cost vs latency trade-offs)
## Best Practices
1. **Baseline before optimizing** – capture 7 days of metrics before making changes
2. **Change one variable at a time** – isolate the impact of each optimization
3. **Monitor post-deployment** – use execution IDs to compare before/after distributions
4. **Set SLOs early** – define p95 latency and cache hit targets per retriever tier
5. **Correlate signals with changes** – timestamp config updates and overlay on metric charts
6. **Automate reporting** – schedule weekly summaries via webhooks or export to your BI tool
## Integration with External Tools
### Export to Datadog / Grafana
Use the Analytics API to pull metrics into your existing observability stack:
```python theme={null}
import requests
resp = requests.get(
"https://api.mixpeek.com/v1/analytics/retrievers/ret_123/performance",
headers={"Authorization": "Bearer mxp_sk_...", "X-Namespace": "ns_prod"}
)
data = resp.json()
# Push to Datadog
statsd.gauge("mixpeek.retriever.p95_latency", data["p95_latency_ms"])
statsd.gauge("mixpeek.retriever.cache_hit_rate", data["cache_hit_rate"])
```
### Webhook Alerts
Configure webhooks to push anomaly alerts:
```bash theme={null}
POST /v1/organizations/webhooks
{
"event_types": ["retriever.slow_query", "extractor.failure", "cache.low_hit_rate"],
"url": "https://your-app.com/webhooks/mixpeek",
"secret": "whsec_..."
}
```
## Next Steps
* Explore all analytics endpoints in the [API Reference](/docs/api-reference/analytics/get-retriever-performance)
* Learn how to record [Interactions](/docs/retrieval/interactions) for signal tracking
* Review [Caching Strategies](/docs/overview/caching) for hit rate optimization
* Set up [Webhooks](/docs/operations/webhooks) for automated alerting
# API Keys
Source: https://docs.mixpeek.com/docs/operations/api-keys
Create, scope, rotate, and monitor API keys — permissions, resource scopes, per-key usage, and end-user keys
An API key authenticates every request to Mixpeek. Keys belong to your
**organization** and are created **per user**. This page covers creating keys,
restricting what they can do (permissions and scopes), rotating and revoking
them, monitoring per-key usage, and the special key types that power end-user
multi-tenancy.
For the broader authentication and tenancy model (the `Authorization` and
`X-Namespace` headers, isolation guarantees), see
[Security & Tenancy](/docs/operations/security). For rate-limit tiers and usage
pools, see [Rate Limits & Quotas](/docs/operations/rate-limits-quotas).
<Note>
**Managing keys requires an admin key.** Every endpoint on this page requires
a key with the `admin` [permission](#permissions). Your organization ships
with a protected `admin-key` you can use to mint your first scoped keys.
</Note>
## How a key looks
When you create a key, the **plaintext secret is returned exactly once** — it is
hashed (SHA-256) at rest and can never be read back. Store it immediately in a
secrets manager or environment variable.
| Field | Example | What it is |
| ------------ | ------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `key` | `mxp_sk_9f3a…` (60 chars) | The plaintext secret. **Shown once, on create/rotate.** Send it as `Authorization: Bearer <key>`. |
| `key_prefix` | `mxp_sk_9f3...` | First 10 characters + `...`. A safe display hint so you can tell keys apart in a list — never the full secret. |
| `key_id` | `key_a1b2c3d4e5f6g7h` | Stable public identifier. Used for [usage lookups](#monitor-usage-per-key). |
| `name` | `backend-service` | Human label you choose. Used to reference the key in update/rotate/revoke calls. |
<Warning>
The plaintext `key` is **never retrievable after creation**. If you lose it or
suspect exposure, [rotate](#rotate-expire-and-revoke) the key — don't try to
recover it.
</Warning>
## Create a key
`POST /v1/organizations/users/{user_email}/api-keys` — create a key owned by the
user at `{user_email}` (use your own email to create keys for yourself).
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "backend-service",
"description": "Service account for the ingestion pipeline",
"permissions": ["read", "write"]
}'
```
```python Python theme={null}
import os, requests
resp = requests.post(
"https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys",
headers={"Authorization": f"Bearer {os.environ['MIXPEEK_ADMIN_KEY']}"},
json={
"name": "backend-service",
"description": "Service account for the ingestion pipeline",
"permissions": ["read", "write"],
},
)
new_key = resp.json()["key"] # plaintext — store it now, it won't be shown again
print(new_key)
```
```javascript JavaScript theme={null}
const resp = await fetch(
"https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.MIXPEEK_ADMIN_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
name: "backend-service",
description: "Service account for the ingestion pipeline",
permissions: ["read", "write"],
}),
},
);
const { key } = await resp.json(); // plaintext — store it now
console.log(key);
```
</CodeGroup>
The response is the key's metadata plus the one-time plaintext `key`:
```json theme={null}
{
"key": "mxp_sk_9f3a2c7b1e...KUwq",
"key_id": "key_a1b2c3d4e5f6g7h",
"key_prefix": "mxp_sk_9f3...",
"name": "backend-service",
"permissions": ["read", "write"],
"scopes": [],
"status": "active",
"created_at": "2026-07-07T00:00:00Z",
"created_by": "usr_a1b2c3d4e5f6g7"
}
```
### Request fields
<ParamField type="string">
Human-friendly label (1–100 chars). Used to reference the key in
update/rotate/revoke calls. Must be unique among your active keys.
</ParamField>
<ParamField type="string[]">
Permission levels granted to the key. **Defaults to full `read`+`write`+`delete`
access** — pass a narrower list explicitly when you want a restricted key. See
[Permissions](#permissions).
</ParamField>
<ParamField type="ResourceScope[]">
Resource-level restrictions. **Omitting scopes grants org-wide access.** See
[Restrict a key to specific resources](#restrict-a-key-to-specific-resources).
</ParamField>
<ParamField type="integer">
Per-key requests-per-minute ceiling. Defaults to your plan limit when absent.
See [Per-key rate limits](#per-key-rate-limits).
</ParamField>
<ParamField type="string (ISO 8601)">
UTC timestamp when the key auto-expires. Omit for a non-expiring key.
</ParamField>
<ParamField type="string">
Optional note (≤500 chars) explaining the key's purpose.
</ParamField>
<ParamField type="string">
End-user identifier. Setting this makes the key **user-scoped** for
document-level ACL — see [End-user keys](#end-user-keys-for-multi-tenant-apps).
</ParamField>
<ParamField type="string[]">
CORS allowlist of web origins (exact or wildcard-subdomain). When set, browser
requests must send a matching `Origin` header. See [Restrict a key to browser
origins](#restrict-a-key-to-browser-origins).
</ParamField>
## Permissions
Every key carries one or more of four permission levels. They form a strict
hierarchy — a higher level **implies** all lower ones, so you never list more
than the strongest you need.
| Permission | Implies | Use it for |
| ---------- | ------------------------- | ----------------------------------------------------------------- |
| `read` | — | Dashboards, analytics, read-only search clients |
| `write` | `read` | Ingestion and service accounts that create documents |
| `delete` | `write`, `read` | Full data management (create + remove) |
| `admin` | `delete`, `write`, `read` | Org administration — **managing keys and users requires `admin`** |
```
admin ⊃ delete ⊃ write ⊃ read
```
A route declares the minimum permission it needs, and a key satisfies it if the
key's permission is at least that level. To issue a **read-only key** for an
internal dashboard, create it with `"permissions": ["read"]`:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{ "name": "metrics-dashboard", "permissions": ["read"] }'
```
## Restrict a key to specific resources
Permissions control *what actions* a key can take; **scopes** control *which
resources* it can take them on. A scope is a `ResourceScope`:
```json theme={null}
{
"resource_type": "namespace",
"resource_id": "ns_customer_acme",
"operations": ["read_data", "execute_retriever"]
}
```
<ParamField type="enum">
What kind of resource the scope governs: `namespace`, `collection`, `bucket`,
`retriever`, `cluster`, `taxonomy` (and other [resource
types](#resource-types-and-operations)).
</ParamField>
<ParamField type="string">
A literal ID (`ns_production`) **or a wildcard** — `*` for all, or a prefix
pattern like `ns_customer_*` to match every namespace for a set of tenants.
</ParamField>
<ParamField type="string[]">
A subset of [namespace operations](#resource-types-and-operations) the key may
perform within the scope. Omit (or `null`) to allow any operation its
[permissions](#permissions) already grant.
</ParamField>
Example — a key that can only **read and search** within one customer's
namespace, and nothing else:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "acme-readonly",
"permissions": ["read"],
"scopes": [{
"resource_type": "namespace",
"resource_id": "ns_customer_acme",
"operations": ["read_data", "execute_retriever"]
}]
}'
```
<Warning>
**An empty `scopes` list means org-wide access** — this is the legacy default,
so a key created without scopes can reach every namespace and resource in your
organization. To confine a key, you **must** provide at least one scope.
Setting `scopes` to `[]` on an update *widens* the key back to org-wide.
</Warning>
### Resource types and operations
<AccordionGroup>
<Accordion title="resource_type values">
`organization`, `user`, `api_key`, `namespace`, `collection`, `bucket`,
`retriever`, `cluster`, `taxonomy`, `storage_connection`, `alert`,
`annotation`, `secret`, `webhook`.
</Accordion>
<Accordion title="operations values (within a namespace)">
**Data:** `read_data`, `write_data`, `delete_data` ·
**Retrieval:** `execute_retriever`, `create_retriever`, `delete_retriever` ·
**Jobs:** `execute_job`, `cancel_job` ·
**Clusters:** `create_cluster`, `delete_cluster`, `modify_cluster` ·
**Infrastructure:** `modify_infrastructure`, `manage_permissions`.
Common bundles: read-only `["read_data", "execute_retriever"]`;
data-engineer `["read_data", "write_data", "execute_job"]`.
</Accordion>
</AccordionGroup>
## Per-key rate limits
Set `rate_limit_override` to cap a single key's requests per minute — useful for
throttling a third-party integration below your plan's default:
```bash theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys/backend-service" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{ "rate_limit_override": 120 }'
```
The override interacts with your organization's plan tiers and usage pools,
which are documented in [Rate Limits & Quotas](/docs/operations/rate-limits-quotas).
## Restrict a key to browser origins
Set `allowed_origins` to a CORS allowlist of web origins. When set, any
**browser** request using the key must send an `Origin` header that matches the
list — an exact origin (`https://app.example.com`) or a wildcard subdomain
(`https://*.example.com`) — otherwise the request is rejected with a `403`.
Requests that send no `Origin` header (server-side curl and the SDKs) are
unaffected. This is enforced org-wide, on every route, for any key that carries
an allowlist.
Set it on create, or change it later with `PATCH`. **An empty list clears the
restriction:**
<CodeGroup>
```bash Set origins theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys/web-widget" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{ "allowed_origins": ["https://app.example.com", "https://*.example.com"] }'
```
```bash Clear origins theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys/web-widget" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{ "allowed_origins": [] }'
```
</CodeGroup>
The allowlist carries across [rotation](#rotate-expire-and-revoke) — a rotated
key keeps its origins.
<Warning>
`allowed_origins` is **defense-in-depth, not a standalone security boundary.**
It only constrains requests that send an `Origin` header (i.e. browsers); a
stolen key used from a non-browser client can omit `Origin` and bypass it.
Pair it with least-privilege [permissions](#permissions) and
[scopes](#restrict-a-key-to-specific-resources), and treat any key shipped to
a browser as public.
</Warning>
## Rotate, expire, and revoke
<Steps>
<Step title="Rotate — issue a new secret, invalidate the old one">
`POST /v1/organizations/users/{user_email}/api-keys/{key_name}/rotate`
returns a **new plaintext `key`** and immediately revokes the previous
secret. The `key_id` and settings carry over; only the secret changes.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys/backend-service/rotate" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY"
```
</Step>
<Step title="Expire — set an automatic cutoff">
Set `expires_at` on create or via `PATCH`. Once passed, the key's status
flips to `expired` and it stops authenticating. Expired keys cannot be
reactivated — create or rotate instead.
</Step>
<Step title="Revoke — kill a key immediately">
`DELETE /v1/organizations/users/{user_email}/api-keys/{key_name}` sets the
key's status to `revoked`. It's a soft revoke (the record is retained for
the audit trail), and it cannot be undone.
```bash theme={null}
curl -X DELETE "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys/backend-service" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY"
```
</Step>
</Steps>
A key's `status` is always one of `active`, `revoked`, or `expired`, and
`last_used_at` records the timestamp of its most recent successful request — a
quick way to find stale keys worth revoking. List a user's keys (add
`?include_revoked=true` to see revoked ones) with:
```bash theme={null}
curl "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY"
# -> { "results": [ { "key_id": "...", "name": "...", "status": "active", ... } ], "total": 3 }
```
<Note>
The organization's primary **`admin-key` is protected** — it cannot be
updated, rotated, or deleted (the Studio UI depends on it). Create additional
scoped keys for specific use cases, and if the `admin-key` is ever
compromised, [contact support](https://mixpeek.com/contact).
</Note>
### Audit trail
Every key lifecycle event is written to your organization's audit log with the
acting user and a timestamp. Keys record `created_by` and `revoked_by`
(with `revoked_at`) directly, and the audit log captures the actions
`api_key_created`, `api_key_rotated`, `api_key_revoked`, and
`api_key_scope_updated` — each with the actor, resource, and what changed.
## Monitor usage per key
Attribute traffic and spend to individual keys. Metrics come from the analytics
pipeline (ClickHouse) and default to the **last 7 days** unless you pass `start`
and `end` (ISO 8601).
```bash theme={null}
curl "https://api.mixpeek.com/v1/organizations/api-keys/key_a1b2c3d4e5f6g7h/usage" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY"
```
```json theme={null}
{
"key_id": "key_a1b2c3d4e5f6g7h",
"start": "2026-06-30T00:00:00Z",
"end": "2026-07-07T00:00:00Z",
"total_requests": 14820,
"total_credits": 512,
"unique_endpoints": 6,
"avg_latency_ms": 43.7,
"p95_latency_ms": 118.0
}
```
For a per-endpoint breakdown, call
`GET /v1/organizations/api-keys/{key_id}/usage/endpoints`.
<Info>
Usage endpoints are keyed by **`key_id`** (e.g. `key_a1b2…`), not the key's
`name`. Grab the `key_id` from the list response above. Your billing usage
records also carry the attributing `key_id`, so credit spend reconciles to the
exact key that drove it.
</Info>
## Key types
Most keys you create are **standard** keys, but Mixpeek issues a few specialized
types for different jobs:
| Type | Prefix | Purpose |
| ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------ |
| Standard | `mxp_sk_` | Regular organization key. Everything above. |
| User-scoped | `usr_sk_` | Carries a `principal_id` for [document-level ACL](#end-user-keys-for-multi-tenant-apps). |
| Retriever-scoped | `ret_sk_` | Locked to executing one [published retriever](#retriever-scoped-keys). |
| Marketplace | `sk_marketplace_` | Grants access to a subscribed marketplace retriever. |
| Session | — | Short-lived key Studio mints on login to back the UI. Hidden from key lists; not something you manage. |
### End-user keys for multi-tenant apps
If you're building an app where your own end-users each see only *their* data,
create a key with a `principal_id`. Mixpeek issues a **`usr_sk_`** user-scoped
key, and every document read made with it is automatically filtered to documents
that principal owns or has been granted access to — enforced server-side, so a
tampered client can't widen its own view.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/users/you@acme.com/api-keys" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{ "name": "enduser-jane", "permissions": ["read"], "principal_id": "user_jane_123" }'
```
<Note>
`principal_id` is an identifier for an **end-user in your application**, not a
Mixpeek organization user. This is the building block for row-level security —
see [Document-Level ACL](/docs/operations/document-acl) for the full ownership and
grant model, and [Permissions](/docs/platform/permissions) for how it composes with
external OpenFGA authorization.
</Note>
### Retriever-scoped keys
To embed a single retriever in a browser app or hand it to a customer, mint a
**`ret_sk_`** key scoped to just that retriever — it can execute the retriever
and nothing else, and its scope/permissions are fixed at creation. Only the
retriever's owner can create one.
`POST /v1/retrievers/{retriever_id}/api-keys`:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers/ret_abc123/api-keys" \
-H "Authorization: Bearer $MIXPEEK_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "public-video-search",
"expires_at": "2026-12-31T23:59:59Z",
"allowed_origins": ["https://app.example.com", "https://*.example.com"]
}'
```
`allowed_origins` works exactly as it does on
[standard keys](#restrict-a-key-to-browser-origins) — a CORS allowlist enforced
for browser requests, and the same defense-in-depth caveat applies. Because a
`ret_sk_` key is designed to be embedded in a browser, always pair it with a
short `expires_at` and treat the key as public.
## Manage keys from Studio
Everything on this page is also available without the API: in
[Studio](https://studio.mixpeek.com), go to **Settings → API Keys** to create,
scope, rotate, revoke, and view per-key usage.
## Related
<CardGroup>
<Card title="Security & Tenancy" icon="shield-halved" href="/operations/security">
The auth model, namespace isolation, and secrets vault.
</Card>
<Card title="Rate Limits & Quotas" icon="gauge-high" href="/operations/rate-limits-quotas">
Plan tiers, usage pools, and how `rate_limit_override` fits in.
</Card>
<Card title="Document-Level ACL" icon="user-lock" href="/operations/document-acl">
Row-level security driven by user-scoped `principal_id` keys.
</Card>
<Card title="Permissions" icon="key" href="/platform/permissions">
Retrieval-time authorization and OpenFGA integration.
</Card>
<Card title="API reference — Create key" icon="code" href="/api-reference/organization-api-keys/create-api-key">
Full request/response schema for every key endpoint.
</Card>
<Card title="API reference — Per-key usage" icon="chart-line" href="/api-reference/organization-usage/get-api-key-usage">
Usage and endpoint-breakdown response schemas.
</Card>
</CardGroup>
# Batch ingestion at scale
Source: https://docs.mixpeek.com/docs/operations/batch-ingestion-at-scale
How to size, submit, and monitor large batch ingestions — chunking, concurrency, automatic recovery, and the limits that apply at each plan tier.
Processing a large bucket (100k–5M objects) is a batching decision, not one API call. This page gives you the sizing model the platform itself uses, so you can pick a chunk size and batch count with numbers instead of guesses.
## Two submission paths
**`POST /v1/collections/{collection_id}/trigger`** creates **one batch containing every object** in the source bucket. There is no server-side splitting. Use it for buckets up to a few thousand objects.
**`POST /v1/buckets/{bucket_id}/batches/bulk-submit`** splits the bucket into batches of `chunk_size` objects (default 1,000, maximum 50,000) and submits each one. Use it for everything larger.
<Warning>
Do not trigger a collection over a very large bucket. The single batch it creates
runs as one job with a fixed worker ceiling and a 24-hour execution deadline, so
it under-parallelizes and can time out. Bulk-submit is the large-corpus path.
</Warning>
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
mp = Mixpeek(api_key="YOUR_API_KEY")
result = mp.buckets.batches.bulk_submit(
bucket_id="bkt_123",
collection_ids=["col_abc"],
chunk_size=20000,
)
print(result.batch_ids)
```
```javascript JavaScript theme={null}
import { Mixpeek } from "@mixpeek/sdk";
const mp = new Mixpeek({ apiKey: "YOUR_API_KEY" });
const result = await mp.buckets.batches.bulkSubmit({
bucketId: "bkt_123",
collectionIds: ["col_abc"],
chunkSize: 20000,
});
console.log(result.batchIds);
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/bkt_123/batches/bulk-submit" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace" \
-H "Content-Type: application/json" \
-d '{"collection_ids": ["col_abc"], "chunk_size": 20000}'
```
</CodeGroup>
## Picking a chunk size
Each batch runs as its own job. A job scales its workers with object count — roughly one CPU worker per 500 objects — up to a per-job worker ceiling. A batch of 16,000–20,000 objects saturates one job's parallelism. Larger batches do not run faster; they only raise the cost of a mid-run failure.
Going too small hurts the other way. Every batch pays a cluster cold-start, and only a fixed number of batches run at once (see the tier table). 1,200 batches of 1,000 objects serialize into deep queue waves and spend a large share of wall-clock on startup.
| Corpus size | Suggested `chunk_size` | Resulting batches |
| ----------- | ---------------------- | ----------------- |
| under 5k | one trigger call | 1 |
| 5k – 100k | 5,000–10,000 | 1–20 |
| 100k – 1M | 20,000 | 5–50 |
| 1M+ | 20,000–50,000 | 20–100+ |
Calibrate before you commit: run one small batch (1,000 objects) and one at your target size, compare objects/hour and credits from the [Billing usage page](/docs/platform/billing), then extrapolate. Media mix changes cost more than batch size does.
## Concurrency and queueing
Submission is accept-and-queue. Batches past your concurrency limit park as `QUEUED` and promote automatically, first-in-first-out, as running batches finish. You do not pace submissions yourself.
| Plan | Max objects per batch | Concurrent running batches |
| ---------- | --------------------- | -------------------------- |
| Free | 10,000 | 1 |
| Pro | 50,000 | 5 |
| Team | 500,000 | 10 |
| Enterprise | 5,000,000 | 50 |
Submission returns `429` only when a queue is truly full. Each running job also has a 24-hour execution deadline.
## Automatic recovery
The platform heals transient failures without your involvement:
* Failed tasks retry 3 times with exponential backoff.
* A worker that dies mid-task has its work redelivered to another worker.
* Reapers sweep every 1–10 minutes for orphaned, stuck, or stalled batches. They re-drive recoverable ones (bounded retries) and mark the rest failed.
* Failures tagged safe-to-resubmit are resubmitted automatically every 30 minutes.
* Batches that failed on transient errors retry hourly, up to 3 times, with 1/2/4-hour backoff.
Progress is durable per object, not per batch. Each object is recorded in a processing ledger as it completes, so resubmitting a partially-failed batch skips finished objects and reprocesses only the remainder. You do not pay for extraction twice.
Some failures never self-heal by design: authentication errors, schema validation errors, and hard provider quota exhaustion (for example, an exhausted OpenAI balance). These end the affected batches in `FAILED` or `COMPLETED_WITH_ERRORS` and wait for you to fix the cause, then [resubmit](/docs/api-reference/bucket-batches/submit-batch-for-processing).
## Monitoring a run
Watch progress through the API, not by polling logs:
* `GET /v1/buckets/{bucket_id}/batches/{batch_id}` — status, per-tier progress, heartbeat freshness, `documents_written`, `failed_object_count`, and `error_summary`.
* `GET .../batches/{batch_id}/failed-documents` — every failed object with its error, for targeted retries.
* [Batch diagnostics](/docs/troubleshoot/batch-diagnostics) — deeper triage when a batch misbehaves.
Enable the batch [system alerts](/docs/api-reference/alerts) — `batch_failed`, `batch_stalled`, and `batch_error_rate` — before a large run. They evaluate every 5 minutes and notify in-app plus any webhook, Slack, or email channel you configure. Without them, a failed batch is visible only if you go looking for it.
<Note>
A batch can finish as `COMPLETED_WITH_ERRORS`. Treat that as a work list, not a
verdict: read `failed-documents`, fix the cause, resubmit, and the ledger skips
everything already done.
</Note>
# Using Mixpeek on corporate networks
Source: https://docs.mixpeek.com/docs/operations/corporate-networks
If Mixpeek won't load behind a corporate proxy or firewall, allowlist these domains — a one-page checklist for IT administrators
If Mixpeek Studio loads a blank page, hangs on sign-in, or "goes dead" a few
seconds after opening on your corporate network, the cause is almost always a
**proxy, firewall, or content-security policy blocking outbound requests** the app
needs to make. Mixpeek's own security policy already permits every domain below —
the block is on the network side, so the fix is a network-side allowlist.
This page is written for an IT administrator to action in one read.
<Note>
**The symptom:** the browser loads `studio.mixpeek.com`, but calls to the API are
silently dropped, so the app renders its shell and then does nothing — no data, no
sign-in, no error. That is a blocked `api.mixpeek.com`, not a Mixpeek outage.
</Note>
## Minimum allowlist — the app loads and works
Allowlist these for **outbound HTTPS (443)** from the browser. This is the smallest
set that makes Studio and the API fully functional:
| Domain | Purpose |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `*.mixpeek.com` | Core API (`api.mixpeek.com`), authentication (`auth.mixpeek.com`), and the app itself (`studio.mixpeek.com`) |
| `*.mxp.co` | Mixpeek content-delivery domain the app calls at runtime |
| `js.stripe.com`, `*.stripe.com` | Billing and checkout (Stripe) |
| `*.amazonaws.com` | Media and file delivery — Studio fetches content over signed Amazon S3 URLs |
| `*.googleapis.com` | Media and file delivery — Studio fetches content over signed Google Cloud Storage URLs |
<Warning>
Allowlisting only `*.mixpeek.com` is a common partial fix: the app then **loads
and signs in, but thumbnails, video, and file previews stay blank** because media
is served over signed `*.amazonaws.com` / `*.googleapis.com` URLs. Include the
storage domains, or evaluators will conclude the product is broken when it is only
blocked.
</Warning>
### Scoped storage hosts for strict environments
If your policy does not permit whole-provider wildcards like `*.amazonaws.com`,
replace the two storage entries above with these narrower hosts. They cover all
media delivery and nothing else — no other AWS or Google service:
| Instead of | Allowlist |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `*.googleapis.com` | `storage.googleapis.com` — the single host all signed media resolves to |
| `*.amazonaws.com` | `s3.*.amazonaws.com`, `*.s3.amazonaws.com`, `*.s3.*.amazonaws.com` — Amazon S3 only (path-style and virtual-hosted, all regions), e.g. `your-bucket.s3.us-east-2.amazonaws.com` |
<Note>
Two `googleapis.com` hosts are needed **only if** you sync from Google Drive
(`drive.googleapis.com` and `www.googleapis.com`, for the connector's OAuth scope).
They are **not** required to view media — leave them out of a viewing-only
evaluation. If you bring your own object storage, media resolves to *your* bucket
host instead; allowlist that host.
</Note>
## Optional — enhances the experience, not required
The app is fully usable without these. Allowlist them to enable in-product analytics,
support chat, and error reporting:
| Domain | Purpose |
| --------------------------------------------------------------- | --------------------------- |
| `*.posthog.com` (`us.i.posthog.com`, `us-assets.i.posthog.com`) | Product analytics |
| `*.intercom.io`, `*.intercomcdn.com` | In-app support chat |
| `*.ingest.us.sentry.io` | Client-side error reporting |
| `*.cloudflareinsights.com` | Performance telemetry |
## Protocols and ports
| Requirement | Value |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Protocol | HTTPS |
| Port | 443 |
| WebSockets | Allow `wss://` to the domains above (Studio uses live connections for some features) |
| TLS interception | If your proxy does TLS/SSL inspection, it must present a certificate the browser trusts **and** preserve the allowlist above; deep-inspection proxies frequently strip the calls that break the app |
## How to confirm it's a network block (not Mixpeek)
Have the affected user, on the corporate network:
<Steps>
<Step title="Open the browser developer console">
In Studio, open DevTools (F12 or ⌘⌥I) → **Console** and **Network** tabs, then reload.
</Step>
<Step title="Look for blocked requests">
Requests to `api.mixpeek.com` (or the domains above) showing `blocked`,
`net::ERR_BLOCKED_BY_CLIENT`, a CSP violation, or a proxy error page — rather than
a normal `200`/`4xx` from Mixpeek — confirm the network is dropping them.
</Step>
<Step title="Compare off-network">
The same account on a non-corporate network (or personal hotspot) loading normally
is definitive: the account and the service are fine; the corporate network is the
variable.
</Step>
</Steps>
<Tip>
Fastest path for an evaluation: allowlist the **Minimum** table above and reload.
If sign-in works but media is blank, you're missing the two storage domains.
</Tip>
## Still blocked?
If the domains are allowlisted and Studio still won't load, contact
[support@mixpeek.com](mailto:support@mixpeek.com) with the browser console's Network
tab (screenshot or HAR export) — the blocked request URLs tell us exactly which rule
is still catching them.
# Deployment
Source: https://docs.mixpeek.com/docs/operations/deployment
Run Mixpeek locally, in Kubernetes, or on managed Ray
Mixpeek is split into two deployable components:
* **API Layer** – FastAPI + Celery + Redis connection (HTTP endpoints, task orchestration, webhooks).
* **Engine Layer** – Ray cluster + Ray Serve (extractors, inference, clustering, taxonomy runs).
Shared dependencies: MongoDB, [MVS](https://mixpeek.com/mvs), Redis, and S3-compatible object storage.
## Local Development
`./start.sh` scripts spin up a full stack with Docker Compose:
```
./start.sh api # FastAPI + Celery
./start.sh celery # Celery Beat
./start.sh engine # Ray head + workers
```
Docker Compose services:
* `mongodb` – metadata (`mongodb://localhost:27017`)
* `mvs` – vector storage ([MVS](https://mixpeek.com/mvs))
* `redis` – task queue/cache (`redis://localhost:6379`)
* `localstack` – S3 emulator (`http://localhost:4566`)
Run `curl http://localhost:8000/v1/health` to confirm readiness, then follow the [Quickstart](/docs/overview/quickstart).
## Production Topology (Kubernetes)
```
Namespace: mixpeek-api
├─ fastapi-deployment (ReplicaSet + HPA)
├─ celery-worker-deployment (process tasks)
└─ celery-beat-deployment (1 replica scheduler)
Namespace: mixpeek-engine
├─ ray-head (StatefulSet)
├─ ray-worker-cpu (Autoscaled Deployment)
└─ ray-worker-gpu (Autoscaled Deployment)
Namespace: mixpeek-data
├─ mongodb (StatefulSet, replica set)
├─ mvs (StatefulSet or distributed cluster)
└─ redis (Deployment or Redis Cluster)
```
Recommended node pools:
* **API nodes** – general purpose (e.g., `t3.xlarge`), scale FastAPI/Celery horizontally.
* **CPU workers** – compute-optimized (e.g., `c5.4xlarge`) for text extraction, clustering.
* **GPU workers** – GPU instances (e.g., `p3.2xlarge`) for embeddings, rerankers, video processing.
Expose the API via an ingress or load balancer; keep Ray Serve internal unless exposing custom inference endpoints.
## Managed Ray (Anyscale / Ray Service)
* Deploy the Engine layer via a managed Ray service.
* Point the API layer to the Ray cluster using `ENGINE_API_URL` and Ray job submission credentials.
* Managed Ray handles autoscaling, node health, and GPU provisioning; you manage API + data stores.
## Core Environment Variables
| Service | Key | Description |
| ------- | ------------------------------------------------------------------ | ------------------ |
| API | `MONGO_URI`, `MVS_URL`, `REDIS_URL`, `S3_BUCKET`, `ENGINE_API_URL` | Connectivity |
| Engine | `MONGO_URI`, `MVS_URL`, `S3_BUCKET`, `RAY_memory`, `RAY_num_gpus` | Runtime config |
| Shared | `ENABLE_ANALYTICS`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc. | Optional providers |
Secrets should be injected via Kubernetes secrets, environment managers, or cloud secret stores.
## Health & Verification
* **Endpoint**: `GET /v1/health` – checks Redis, MongoDB, MVS, Celery, Engine, ClickHouse (if enabled).
* **Smoke test**: create namespace → bucket → collection → upload object → submit batch → execute retriever.
* **Tasks**: ensure Celery workers process webhook events, cache invalidations, and maintenance tasks.
## Scaling Guidelines
| Component | Scaling Strategy | Notes |
| -------------- | ------------------------------------------------ | ----------------------------------------------- |
| FastAPI | Horizontal autoscale on CPU/utilization | Stateless, use HPA |
| Celery workers | Scale with queue depth | Prefork pool supports task termination |
| Ray workers | Autoscale CPU/GPU pools | Use Ray autoscaler or managed Ray policies |
| MVS | Scales automatically with object storage backend | Monitor vector count and query latency |
| MongoDB | Use managed cluster (Atlas, DocumentDB) | Ensure indexes on `namespace_id`, `internal_id` |
| Redis | Scale vertically or cluster | Used for task queue + caching |
Monitor Ray dashboard (port 8265) for job status, resource utilization, and Serve deployments.
## Deployment Checklist
1. Provision MongoDB, MVS, Redis, and S3/GCS buckets (with IAM roles).
2. Deploy Ray cluster (head + workers) and confirm job submission works.
3. Deploy FastAPI + Celery services; configure environment variables to point to Ray + data stores.
4. Configure ingress/HTTPS, secrets, and network policies.
5. Run health checks and quickstart workflow to verify end-to-end functionality.
6. Set up observability (logs, metrics, webhooks) and configure backups for MongoDB.
## References
* [Architecture](/docs/overview/architecture) – full system design
* [Observability](/docs/operations/observability) – metrics, logs, dashboards
* [Security](/docs/operations/security) – tenancy, auth, secret management
* [Webhooks](/docs/operations/webhooks) – event processing pipeline
# Document-Level ACL
Source: https://docs.mixpeek.com/docs/operations/document-acl
Row-level security for multi-user applications with automatic access control on documents
## Overview
Document-level ACL (Access Control Lists) lets you build multi-user applications on top of Mixpeek where each end-user only sees the documents they are authorized to access. Instead of managing separate namespaces per user, you store all documents in a single namespace and let Mixpeek enforce read/write permissions automatically.
Key capabilities:
* **Automatic filter injection** — user-scoped API keys transparently filter all reads so users only see their own (or shared) documents
* **Per-document ownership** — every document has an `_acl` object tracking who can read, write, or own it
* **Public documents** — mark a document as `public: true` to make it visible to all users
* **Zero application-side filtering** — your app does not need to add user filters to queries; Mixpeek handles it server-side
## Concepts
### API Key Types
Mixpeek supports two types of API keys:
| Key Type | Prefix | ACL Behavior |
| --------------- | --------- | --------------------------------------------------------- |
| **Org-scoped** | `mxp_sk_` | Full access — bypasses ACL filters entirely |
| **User-scoped** | `usr_sk_` | Restricted — ACL filters injected on every read and write |
Org-scoped keys are for your backend services and admin operations. User-scoped keys are for end-user sessions in your application.
### The `_acl` Object
Every document stores an `_acl` sub-object inside its `_internal` payload:
```json theme={null}
{
"_internal": {
"_acl": {
"owner": "user_123",
"read": ["user_123", "user_456"],
"write": ["user_123"],
"public": false
}
}
}
```
<ParamField type="string">
The `principal_id` of the user who created the document. Only the owner (or an org-scoped key) can modify ACL settings.
</ParamField>
<ParamField type="string[]">
List of `principal_id` values that can read this document.
</ParamField>
<ParamField type="string[]">
List of `principal_id` values that can update this document.
</ParamField>
<ParamField type="boolean">
When `true`, the document is visible to all user-scoped keys regardless of the `read` list.
</ParamField>
## Creating User-Scoped API Keys
Generate a user-scoped key by providing a `principal_id` that represents the end-user in your application (e.g., your internal user ID, email, or UUID).
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
# Use your org-scoped key to create a user-scoped key
client = Mixpeek(api_key="mxp_sk_your-org-key")
user_key = client.organizations.api_keys.create(
principal_id="user_123", # setting principal_id makes the key user-scoped
description="Key for user 123"
)
print(user_key.api_key) # usr_sk_...
```
```javascript JavaScript theme={null}
import { Mixpeek } from 'mixpeek-sdk'
const client = new Mixpeek({ apiKey: 'mxp_sk_your-org-key' })
const userKey = await client.organizations.apiKeys.create({
principalId: 'user_123', // setting principalId makes the key user-scoped
description: 'Key for user 123'
})
console.log(userKey.apiKey) // usr_sk_...
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/organizations/users/{user_email}/api-keys \
-H "Authorization: Bearer mxp_sk_your-org-key" \
-H "Content-Type: application/json" \
-d '{
"principal_id": "user_123",
"description": "Key for user 123"
}'
```
</CodeGroup>
<Note>
Store the returned `usr_sk_` key securely. Pass it to your frontend or mobile app so the end-user's requests are automatically scoped.
</Note>
## How ACL Works
### Automatic ACL on Document Creation
When a user-scoped key creates a document, Mixpeek automatically sets the `_acl`:
* `owner` = the key's `principal_id`
* `read` = `[principal_id]`
* `write` = `[principal_id]`
* `public` = `false`
You do not need to set `_acl` manually — it is populated from the API key's identity.
### Automatic Filter Injection on Reads
Every read operation made with a user-scoped key — including retriever executions, document listings, and searches — has an ACL filter injected server-side. The filter ensures the user only sees documents where:
1. Their `principal_id` is in the `_acl.read` list, **OR**
2. The document has `_acl.public: true`
This happens transparently. Your application code does not need to add any user-specific filters.
### Write Protection
Update and delete operations made with a user-scoped key check the `_acl.write` list. If the user's `principal_id` is not in the write list, the operation returns a `403 Forbidden` error.
### Org-Scoped Key Bypass
Org-scoped keys (`mxp_sk_` prefix) bypass all ACL checks. They can read, write, and modify any document regardless of its `_acl` settings. Use org-scoped keys for backend services, admin dashboards, and data pipelines.
## Managing ACL
Update a document's ACL using the dedicated endpoint. Only the document owner or an org-scoped key can modify ACL settings.
<CodeGroup>
```python Python theme={null}
# Share a document with another user (add read access)
client = Mixpeek(api_key="usr_sk_owner-key")
client.collections.documents.update_acl(
collection_id="col_abc123",
document_id="doc_xyz789",
read=["user_123", "user_456"], # Add user_456 to readers
)
```
```javascript JavaScript theme={null}
const client = new Mixpeek({ apiKey: 'usr_sk_owner-key' })
await client.collections.documents.updateAcl({
collectionId: 'col_abc123',
documentId: 'doc_xyz789',
read: ['user_123', 'user_456'] // Add user_456 to readers
})
```
```bash cURL theme={null}
curl -X PATCH https://api.mixpeek.com/v1/collections/col_abc123/documents/doc_xyz789/acl \
-H "Authorization: Bearer usr_sk_owner-key" \
-H "Content-Type: application/json" \
-d '{
"read": ["user_123", "user_456"],
"write": ["user_123"],
"public": false
}'
```
</CodeGroup>
### Making a Document Public
Set `public: true` to make a document visible to all user-scoped keys in the namespace:
```bash theme={null}
curl -X PATCH https://api.mixpeek.com/v1/collections/col_abc123/documents/doc_xyz789/acl \
-H "Authorization: Bearer usr_sk_owner-key" \
-H "Content-Type: application/json" \
-d '{
"public": true
}'
```
### Revoking Access
Remove a user from the `read` or `write` list by sending the updated list without their `principal_id`:
```bash theme={null}
# Remove user_456 from readers — only user_123 retains access
curl -X PATCH https://api.mixpeek.com/v1/collections/col_abc123/documents/doc_xyz789/acl \
-H "Authorization: Bearer usr_sk_owner-key" \
-H "Content-Type: application/json" \
-d '{
"read": ["user_123"],
"write": ["user_123"]
}'
```
## Examples
### Multi-User Document Search
A SaaS app where each user uploads and searches their own documents, with optional sharing.
<Steps>
<Step title="Create user-scoped keys for each user">
When a user signs up, generate a `usr_sk_` key with their user ID as the `principal_id`.
</Step>
<Step title="Users ingest documents with their key">
Each document is automatically tagged with the user's ACL. User A cannot see User B's documents.
</Step>
<Step title="Users search normally">
Retriever executions with a user-scoped key only return documents the user can access — no extra filters needed.
</Step>
<Step title="Share a document">
User A shares a document with User B by adding User B's `principal_id` to the `read` list via the ACL endpoint.
</Step>
</Steps>
### Public Knowledge Base with Private Uploads
Combine public reference documents with private user uploads in the same namespace:
<CodeGroup>
```python Python theme={null}
# Admin creates public reference docs using org-scoped key
admin_client = Mixpeek(api_key="mxp_sk_org-key")
# Create a document and make it public
doc = admin_client.collections.documents.create(
collection_id="col_kb",
data={"title": "Getting Started Guide", "content": "..."}
)
admin_client.collections.documents.update_acl(
collection_id="col_kb",
document_id=doc.document_id,
public=True,
)
# End-user creates private notes using their user-scoped key
user_client = Mixpeek(api_key="usr_sk_user-key")
user_client.collections.documents.create(
collection_id="col_kb",
data={"title": "My private notes", "content": "..."}
)
# _acl automatically set: owner=user, read=[user], write=[user], public=false
```
</CodeGroup>
When the end-user searches, they see both the public reference docs and their own private notes — but not other users' private documents.
## Backwards Compatibility
Documents created before ACL was enabled do not have an `_acl` field. These documents follow these rules:
| Key Type | Behavior for Documents Without `_acl` |
| --------------- | ------------------------------------------------------------------ |
| **Org-scoped** | Full access — documents are visible and writable as before |
| **User-scoped** | **Not visible** — documents without `_acl` are excluded from reads |
<Warning>
If you adopt user-scoped keys on an existing namespace, pre-existing documents will be invisible to end-users until you set their `_acl` (either individually or via a bulk update). Org-scoped keys continue to work normally.
</Warning>
To backfill ACL on existing documents, use a bulk update with an org-scoped key:
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/collections/col_abc123/documents/bulk-update \
-H "Authorization: Bearer mxp_sk_org-key" \
-H "Content-Type: application/json" \
-d '{
"filter": {},
"update": {
"_internal._acl": {
"owner": "system",
"read": [],
"write": [],
"public": true
}
}
}'
```
## Best Practices
<Tip>
Use your own stable user identifiers (UUIDs, database IDs) as `principal_id` values. Avoid using email addresses, which can change.
</Tip>
<Tip>
Keep org-scoped keys on your backend only. Never expose `mxp_sk_` keys to client-side code. User-scoped `usr_sk_` keys are safe to use in frontend and mobile apps.
</Tip>
<Warning>
ACL is enforced at the Mixpeek API layer. If you have direct access to the underlying MVS or MongoDB instances, ACL filters are not applied. Always route end-user traffic through the Mixpeek API.
</Warning>
## Related
* [Security & Tenancy](/docs/operations/security) — org-level authentication and namespace isolation
* [Filters](/docs/retrieval/filters) — manual query filters (ACL filters are injected automatically on top of these)
* [Namespaces](/docs/ingestion/namespaces) — data isolation boundaries
# Environment Branching
Source: https://docs.mixpeek.com/docs/operations/environment-branching
Clone namespaces, collections, retrievers, and taxonomies to create isolated staging environments and run experiments without re-processing data
## Overview
Branching an AI data environment used to mean one of two things: re-processing your entire corpus (slow, expensive) or experimenting directly in production (dangerous). Mixpeek solves this with a **clone-based branching model** that operates at every layer of the pipeline — namespace, collection, retriever, and taxonomy — so you can create fully isolated environments instantly and promote changes deliberately.
This matters most when:
* You want to test a new retrieval pipeline on production data without affecting live traffic
* You need a staging namespace that mirrors production for QA without re-ingesting everything
* You're evaluating a new embedding model and want side-by-side comparison on the same corpus
* A taxonomy schema is about to change and you need a safe place to validate it first
## Branching Primitives
Mixpeek resources are **immutable by design** — you can't change a collection's feature extractor or a retriever's pipeline stages via PATCH. This preserves execution history, dependent results, and audit trails. The branching mechanism is a first-class `clone` operation available on every resource type.
### Namespace Clone (full environment branch)
The broadest primitive. A namespace clone deep-copies the entire environment: collections (including MVS vectors), retrievers, buckets, and optionally taxonomies — remapping all internal IDs so nothing points back to production.
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
clone = client.namespaces.clone(
"ns_prod",
namespace_name="listings_staging",
include_resources={
"collections": True, # copies MVS vectors — no reprocessing
"retrievers": True, # remaps all collection refs to staging copies
"taxonomies": False # optional: include taxonomy configs
}
)
# The response returns immediately with the new namespace and a clone status.
new_namespace_id = clone["namespace"]["namespace_id"]
print(clone.status) # "cloning" -> completes async to "ready"/"failed"
print(clone.cloned_resources) # what was copied (source_id -> cloned_id per resource)
```
```javascript JavaScript theme={null}
import { Mixpeek } from 'mixpeek-sdk';
const client = new Mixpeek({ apiKey: 'your-api-key' });
const clone = await client.namespaces.clone('ns_prod', {
namespace_name: 'listings_staging',
include_resources: {
collections: true,
retrievers: true,
taxonomies: false,
},
});
const newNamespaceId = clone.namespace.namespace_id;
console.log(clone.status); // "cloning" -> completes async to "ready"
console.log(clone.cloned_resources); // source_id -> cloned_id per resource
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/namespaces/ns_prod/clone \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: ns_prod" \
-H "Content-Type: application/json" \
-d '{
"namespace_name": "listings_staging",
"include_resources": {
"collections": true,
"retrievers": true,
"taxonomies": false
}
}'
```
</CodeGroup>
<Note>
Namespace clone copies MVS vectors directly — your data is **not re-processed**. The clone is isolated: changes to collections, retrievers, or documents in staging have zero effect on production.
</Note>
**What gets copied:**
| Resource | What's cloned | Notes |
| ----------- | -------------------------- | --------------------------------------- |
| Collections | Metadata + MVS vectors | Vectors copied, not recomputed |
| Retrievers | Full pipeline stage config | All collection refs remapped to staging |
| Buckets | Metadata only | S3 objects are not duplicated |
| Taxonomies | Config only (optional) | Retriever refs remapped |
### Collection Clone (swap extractor or source)
Clone a single collection and optionally override its feature extractor or source. This is the entry point for **embedding model experimentation** — run two extractors on the same corpus and compare retrieval quality.
<CodeGroup>
```python Python theme={null}
# Clone with a different embedding model
new_col = client.collections.clone(
"col_properties",
collection_name="properties_siglip_v2",
feature_extractor={
"feature_extractor_name": "image_extractor",
"version": "v2",
"parameters": { "model": "google_siglip_base_v1" }
}
)
# Trigger reprocessing — required when changing the extractor
client.collections.trigger(new_col["collection_id"])
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/collections/col_properties/clone \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "properties_siglip_v2",
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v2",
"parameters": { "model": "google_siglip_base_v1" }
}
}'
```
</CodeGroup>
<Note>
When you clone a collection without changing the feature extractor, vectors are reused. When you **change the extractor**, you must trigger reprocessing — vectors are model-specific and cannot be ported across embedding spaces.
</Note>
### Retriever Clone (pipeline experiment)
Immutable retriever stages mean the safe way to test a new ranking strategy, add a rerank stage, or adjust fusion weights is to clone the retriever with overrides.
<CodeGroup>
```python Python theme={null}
# Clone retriever and add an MMR rerank stage
new_ret = client.retrievers.clone(
"ret_ad_relevance",
body={
"retriever_name": "ad_relevance_mmr_v2",
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"feature_uri": "mixpeek://text_extractor@v1/e5_large",
"query": "{{INPUT.query}}",
"final_top_k": 50
}
}
},
{
"stage_name": "diversify",
"stage_type": "sort",
"config": {
"stage_id": "mmr",
"parameters": { "lambda": 0.7, "top_k": 20 }
}
}
]
}
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers/ret_ad_relevance/clone \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "ad_relevance_mmr_v2",
"stages": [...]
}'
```
</CodeGroup>
### Taxonomy Clone (schema version branch)
Taxonomies are immutable in their core config (`retriever_id`, `input_mappings`, `enrichment_fields`). Clone to branch a schema — swap the backing retriever, adjust the hierarchy, or test a new classification model.
<CodeGroup>
```python Python theme={null}
# Branch a taxonomy to test a new classification retriever
new_tax = client.taxonomies.clone(
taxonomy_identifier="tax_icd_codes",
body={
"taxonomy_name": "icd_codes_llama_v2",
"retriever_id": "ret_llama_classifier" # only changed field
}
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/taxonomies/tax_icd_codes/clone \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"taxonomy_name": "icd_codes_llama_v2",
"retriever_id": "ret_llama_classifier"
}'
```
</CodeGroup>
***
## Common Patterns
### Pattern 1: Staging environment for a production namespace
The most common use case: a full mirror of production where QA teams can validate changes before go-live.
```
prod namespace (ns_prod)
├── col_content_v1 (CLIP embeddings, 2M docs)
├── ret_content_search (semantic + rerank pipeline)
└── tax_iab_v3 (IAB 3.0 taxonomy)
→ clone → staging namespace (ns_staging)
├── col_content_v1_copy (vectors copied, no reprocessing)
├── ret_content_search_copy (points to staging collection)
└── [taxonomies excluded]
```
After cloning, engineers can modify the staging retriever pipeline, run evaluations, and only promote to prod once quality gates pass.
### Pattern 2: Embedding model A/B test
Run two embedding models on the same corpus, then compare retrieval quality with Mixpeek's [Evaluations](/docs/retrieval/evaluations) before committing.
```python theme={null}
# Collection A — existing model (no reprocessing needed)
col_a = "col_listings_clip" # already exists
# Collection B — new model (clone + reprocess)
col_b = client.collections.clone("col_listings_clip", {
"collection_name": "col_listings_siglip",
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"parameters": { "model": "google_siglip_base_v1" }
}
})
client.collections.trigger(col_b["collection_id"])
# Retriever A — current model
ret_a = "ret_property_search"
# Retriever B — points to new collection
ret_b = client.retrievers.clone("ret_property_search", {
"retriever_name": "ret_property_search_siglip",
"collection_identifiers": [col_b["collection_id"]]
})
# Run evaluations side by side
eval_a = client.retrievers.run_evaluation("ret_property_search", dataset_id="eval_ds_001")
eval_b = client.retrievers.run_evaluation(ret_b["retriever_id"], dataset_id="eval_ds_001")
```
### Pattern 3: Retriever pipeline experiment
Test a new retrieval stage (reranker, MMR, query expansion) without touching the live retriever.
```python theme={null}
# Current: bare semantic search
# Experiment: add query expansion + rerank
exp_retriever = client.retrievers.clone("ret_content_search", {
"retriever_name": "ret_content_search_exp_rerank",
"stages": [
{ "stage_type": "filter", "config": { "stage_id": "query_expand", "parameters": {...} } }, # new
{ "stage_type": "filter", "config": { "stage_id": "feature_search", "parameters": {...} } },
{ "stage_type": "sort", "config": { "stage_id": "rerank", "parameters": {...} } } # new
]
})
# Shadow-test: run both retrievers on the same queries, compare metrics
```
### Pattern 4: Taxonomy version checkpoint
Before migrating an IAB or ICD taxonomy schema, snapshot the current version so you can validate in parallel.
```python theme={null}
# Snapshot before migration
checkpoint = client.taxonomies.clone("tax_iab_content", {
"taxonomy_name": "tax_iab_content_v3_snapshot"
})
# Apply to existing docs in the new taxonomy to validate
client.taxonomies.apply(
taxonomy_identifier=checkpoint.taxonomy_id,
collection_id="col_content_sample_100"
)
```
***
## Vertical Examples
<AccordionGroup>
<Accordion title="Adtech — IAB taxonomy migration">
**Problem:** Your IAB 2.2 taxonomy needs to be upgraded to IAB 3.0. You can't migrate production mid-campaign without validating classification quality first.
**Solution:**
1. Clone the production namespace → `ns_adtech_staging`
2. Clone `tax_iab_v2` with the new retriever trained on IAB 3.0 → `tax_iab_v3_candidate`
3. Apply `tax_iab_v3_candidate` to a sample of staging documents
4. Validate label quality against ground truth
5. Promote: update production taxonomy config once quality gates pass
No live campaigns are affected. Staging vectors are reused from production (no reprocessing).
</Accordion>
<Accordion title="Healthcare — clinical classification model upgrade">
**Problem:** You're switching the ICD-10 classification retriever from GPT-4o to a fine-tuned clinical model. Patient record classification in production cannot be interrupted.
**Solution:**
1. Clone `tax_icd10_prod` with the new retriever → `tax_icd10_clinical_v2`
2. Apply to a test collection of de-identified records
3. Compare classification accuracy against the production taxonomy's output on the same records
4. Only swap production once the new model meets or exceeds current accuracy
Both taxonomies run in parallel. There's no downtime and no disruption to the production classification pipeline.
</Accordion>
<Accordion title="Media — search ranking experiment">
**Problem:** Editorial wants to test a diversity-aware ranking algorithm (MMR) on the content search endpoint before rolling out to all users.
**Solution:**
1. Clone `ret_content_search` → `ret_content_search_mmr`
2. Override stages to add an MMR sort step after semantic search
3. Route 10% of internal QA traffic to the experimental retriever (traffic splitting handled in your application layer)
4. Compare CTR, dwell time, and result diversity metrics
5. Promote by cloning the staging retriever config back to production
The collection (and all its vectors) is shared between both retrievers — no extra storage cost.
</Accordion>
<Accordion title="CRE — new embedding model for property images">
**Problem:** A new image embedding model shows better performance on architectural/interior photos. You want to validate before migrating 5M property images.
**Solution:**
1. Clone `col_property_images` with the new extractor → `col_property_images_v2`
2. Trigger reprocessing (required for model changes — vectors are model-specific)
3. Create `ret_property_search_v2` pointing to the new collection
4. Run offline evaluation: compare top-5 retrieval precision on a labeled query set
5. If metrics improve, migrate production: update `ret_property_search` to point to the new collection
The old collection stays active during migration as a fallback. Roll back is instant — just repoint the retriever.
</Accordion>
</AccordionGroup>
***
## Promotion Workflow
Branching is only useful if you have a clear path back to production. The recommended flow:
```
dev branch → staging clone → eval gate → production
↑ |
└──────────── rollback (repoint retriever) ───┘
```
**Promoting a retriever experiment to production:**
1. Run [Evaluations](/docs/retrieval/evaluations) on the experimental retriever
2. If metrics pass, delete the old production retriever (after confirming no dependent published pages)
3. Rename the experimental retriever to the production name via PATCH (name is mutable)
4. Or: update your application to point to the new retriever ID directly
**Rolling back** is always safe — because the old retriever and collection still exist unchanged, you can revert by updating the retriever ID in your application config.
***
## Best Practices
<Tip>
**One namespace per environment** (dev, staging, prod). Clone from prod to create staging rather than maintaining them separately — this guarantees staging always reflects current production data and config.
</Tip>
* **Clone, don't modify.** Resist the urge to patch production resources for "quick experiments." A clone takes seconds and preserves the ability to roll back.
* **Retriever clones are free** — they share the underlying collection (and all its vectors). You only pay for additional MVS storage when collection vectors diverge.
* **Trigger reprocessing only when the extractor changes.** Cloning a collection with the same extractor reuses existing vectors — no GPU time consumed.
* **Use evaluations before promoting.** The [Evaluations API](/docs/retrieval/evaluations) lets you run offline quality checks on any retriever before it touches production traffic.
* **Name branches consistently.** A naming convention like `{resource}_staging`, `{resource}_exp_{date}`, or `{resource}_v{n}` makes it easy to identify which resources are active experiments vs. production.
* **Clean up stale branches.** Delete experimental collections and retrievers after promotion or abandonment. MVS vectors from branched collections consume storage until deleted.
***
## Related
* [Namespaces](/docs/ingestion/namespaces) — isolation boundaries and multi-tenancy
* [Collections](/docs/ingestion/collections) — processing pipelines and lifecycle states
* [Retrievers](/docs/retrieval/retrievers) — pipeline stages and configuration
* [Evaluations](/docs/retrieval/evaluations) — offline quality testing before promotion
# Manifests
Source: https://docs.mixpeek.com/docs/operations/manifests
Declarative resource configuration with YAML manifests
Manifests let you define Mixpeek resources in YAML files and apply them in a single operation. This enables version-controlled, reproducible infrastructure across environments.
## Quick Start
```yaml theme={null}
# mixpeek.yaml — a complete pipeline: ingest → index → search
version: "1.0"
metadata:
name: "video-search-env"
namespaces:
- name: video_search
feature_extractors:
- name: multimodal_extractor
version: v1
buckets:
- name: raw_videos
namespace: video_search
schema:
properties:
video: { type: video }
collections:
- name: video_index
namespace: video_search
source:
type: bucket
bucket: raw_videos
feature_extractor:
feature_extractor_name: multimodal_extractor
version: v1
retrievers:
- name: video_search_tool
namespace: video_search
collections: [video_index]
input_schema:
query:
type: text
required: true
stages:
- stage_name: search
stage_type: filter
config:
stage_id: feature_search
parameters:
searches:
- collection_identifiers: [video_index]
query:
input_mode: text
text: "{{INPUT.query}}"
feature_uri: "mixpeek://multimodal_extractor@v1/embedding"
top_k: 10
final_top_k: 10
```
Every collection needs a `source` and a `feature_extractor`; every retriever needs `namespace`, `collections`, and its `stages` pipeline. `validate` names any missing field, warns on unknown top-level keys (a typo like `retreivers:` won't silently drop your search pipeline), and checks that references (`namespace`, `bucket`, collection names) resolve within the manifest.
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/manifest/apply \
-H "Authorization: Bearer $API_KEY" \
-F "manifest_file=@mixpeek.yaml"
```
## Core Operations
| Operation | Description |
| ------------ | ---------------------------------------------------- |
| **Apply** | Create all resources defined in the manifest |
| **Validate** | Check syntax, schema, and references without changes |
| **Export** | Generate a manifest from existing resources |
| **Diff** | Compare manifest against current state |
## Resource Types
Manifests support these resource types, applied in dependency order:
1. **Namespaces** - Isolation boundaries with feature extractors
2. **Buckets** - Object storage with schemas
3. **Collections** - Document stores with indexing
4. **Taxonomies** - Classification hierarchies
5. **Clusters** - Grouping configurations
6. **Retrievers** - Search pipelines
## Secret References
Use `${{ secrets.NAME }}` to reference organization secrets:
```yaml theme={null}
buckets:
- name: external_data
namespace: my_namespace
sync:
connection_id: ${{ secrets.S3_CONNECTION_ID }}
```
Secrets must exist before applying. Use the validate endpoint to check for missing secrets.
## Dependency Resolution
Resources are created in topological order. The manifest engine:
* Resolves cross-resource references automatically
* Detects circular dependencies
* Rolls back all changes if any creation fails
## Workflows
### Environment Replication
```bash theme={null}
# Export from production
curl https://api.mixpeek.com/v1/manifest/export \
-H "Authorization: Bearer $PROD_KEY" \
-o prod-manifest.yaml
# Apply to staging
curl -X POST https://api.mixpeek.com/v1/manifest/apply \
-H "Authorization: Bearer $STAGING_KEY" \
-F "manifest_file=@prod-manifest.yaml"
```
### Pre-deployment Validation
```bash theme={null}
# Validate without applying
curl -X POST https://api.mixpeek.com/v1/manifest/validate \
-H "Authorization: Bearer $API_KEY" \
-F "manifest_file=@mixpeek.yaml"
# Check what would change
curl -X POST https://api.mixpeek.com/v1/manifest/diff \
-H "Authorization: Bearer $API_KEY" \
-F "manifest_file=@mixpeek.yaml"
```
### CI/CD Integration
```yaml theme={null}
# GitHub Actions example
- name: Deploy Mixpeek Resources
run: |
curl -X POST https://api.mixpeek.com/v1/manifest/apply \
-H "Authorization: Bearer ${{ secrets.MIXPEEK_API_KEY }}" \
-F "manifest_file=@mixpeek.yaml"
```
## References
* [Apply Manifest](/docs/api-reference/manifest/apply-manifest)
* [Validate Manifest](/docs/api-reference/manifest/validate-manifest)
* [Export Manifest](/docs/api-reference/manifest/export-manifest)
* [Diff Manifest](/docs/api-reference/manifest/diff-manifest)
# Observability
Source: https://docs.mixpeek.com/docs/operations/observability
Monitor API, Engine, storage, and asynchronous jobs
Mixpeek provides multiple observability surfaces: health endpoints, task metadata, Ray dashboards, analytics APIs, and webhook histories. Combine them to detect regressions early and debug production issues quickly.
## Health & Status
* **`GET /v1/health`** – checks MongoDB, [MVS](https://mixpeek.com/mvs), Redis, Celery, Engine, and ClickHouse (if analytics enabled). Returns `OK` or `DEGRADED` with per-service errors.
* **Tasks API** – `/v1/tasks/{task_id}` and `/v1/tasks/list` expose status for batches, clustering jobs, taxonomy materialization, and migrations. All tasks use `TaskStatusEnum`.
* **Webhooks** – webhook events recorded in MongoDB provide a durable log of ingestion and enrichment milestones (`collection.documents.written`, etc.).
## Engine Monitoring
* **Ray Dashboard (port 8265)** – view worker health, task timelines, Serve deployments, resource utilization, and logs.
* **Ray logs** – pod logs (Kubernetes) or Ray CLI provide detailed extractor and clustering output (`ray logs <job_id>`).
* **Serve metrics** – per-model latency and request counts; scrape via Prometheus or Ray metrics endpoint.
## Analytics APIs
Enable analytics (`ENABLE_ANALYTICS=true`) to populate ClickHouse-backed metrics:
| Endpoint | Insight |
| ------------------------------------------------- | -------------------------------------------- |
| `/v1/analytics/retrievers/{id}/performance` | Query volume, latency percentiles |
| `/v1/analytics/retrievers/{id}/stages` | Stage-level timing and candidate counts |
| `/v1/analytics/retrievers/{id}/signals` | Cache hits, rerank scores, filter reductions |
| `/v1/analytics/retrievers/{id}/cache-performance` | Hit/miss rates and latency delta |
| `/v1/analytics/retrievers/{id}/slow-queries` | Top slow queries with execution context |
| `/v1/analytics/usage/summary` | Usage and spend (billing support) |
Use these APIs to populate dashboards or feed alerting systems.
## Logging & Tracing
* **API layer** – structured JSON logs include request IDs, namespace, HTTP status, error codes, and downstream latency.
* **Celery workers** – log task execution, retries, and webhook dispatch results.
* **Ray workers** – include extractor metrics, batch IDs, and queue stats; aggregate logs centrally for long-term retention.
* **Correlation** – propagate `x-request-id` from API to Engine jobs via `additional_data.request_id` to stitch traces together.
## Metrics to Track
| Component | Key Metrics |
| --------- | -------------------------------------------------------------------- |
| API | Request rate, p95 latency, error rate, rate-limit hits |
| Celery | Queue depth, task execution time, retry count |
| Ray | Worker utilization (CPU/GPU), job duration, Serve requests in flight |
| MongoDB | Operation latency, primary health, replication lag |
| MVS | Storage usage, search latency, vector count per namespace |
| Redis | Connection count, command latency, cache hit ratio |
Integrate with Prometheus, Datadog, or your preferred metrics stack via existing exporters or custom scrapers.
## Alerting Playbook
1. **Latency spike** → check retriever analytics, stage statistics, and Ray Serve load.
2. **Task backlog** → inspect Celery queue length, Redis health, and Ray worker availability.
3. **Failed enrichment** → query `/v1/tasks/list` for `FAILED`, inspect `error_message`, review webhook events.
4. **Storage saturation** → monitor MVS storage usage and MongoDB disk consumption; scale storage or shard by namespace.
5. **Cache regression** → view cache hit-rate endpoint; adjust TTLs or stage cache configuration.
## Dashboards to Build
* **API dashboard** – health endpoint status, request latency, error breakdown, rate-limit counters.
* **Engine dashboard** – Ray worker utilization, job runtime percentiles, extractor throughput, Serve queue depth.
* **Retrieval performance** – retriever analytics charts (latency, cache hits, slow queries).
* **Storage dashboard** – MongoDB/Redis/MVS metrics for capacity planning.
* **Task tracker** – open tasks by status, median processing times, failure rates.
## Incident Response Tips
* Keep runbooks for common failures (e.g., extractor timeouts, MVS restarts).
* Use webhook history to confirm whether ingestion completed or stalled.
* Capture Ray job IDs from task metadata to replay logs quickly.
* Snapshot retriever and collection configurations when debugging to ensure you’re reproducing the same pipeline.
With health checks, task metadata, analytics APIs, and Ray observability, you can confidently operate Mixpeek in production and catch issues before users notice.
# Rate Limits & Quotas
Source: https://docs.mixpeek.com/docs/operations/rate-limits-quotas
Understand API rate limits, usage pools, and strategies for scaling under constraints
Mixpeek enforces rate limits and quotas to ensure fair resource allocation and system stability. Limits vary by tier (free, pro, enterprise) and are applied per organization, API key, and endpoint.
## Rate Limiting Model
Mixpeek uses **token bucket** rate limiting with per-minute and per-second windows:
<CardGroup>
<Card title="Request Rate Limits" icon="gauge-high">
Maximum requests per second (RPS) or per minute (RPM) per API key.
</Card>
<Card title="Usage & Pools" icon="coins">
Monthly dollar usage pool included with each tier; overage bills at the rate card.
</Card>
<Card title="Concurrent Requests" icon="layer-group">
Maximum simultaneous in-flight requests per organization.
</Card>
<Card title="Resource Quotas" icon="database">
Limits on collections, documents, feature extractors, and batch sizes.
</Card>
</CardGroup>
## Rate Limit Tiers
| Tier | Requests/Min (RPM) | Requests/Sec (RPS) | Concurrent | Burst Allowance |
| -------------- | ------------------ | ------------------ | ---------- | --------------- |
| **Free** | 60 | 10 | 5 | 20 requests |
| **Pro** | 600 | 100 | 50 | 200 requests |
| **Enterprise** | Custom | Custom | Custom | Custom |
**Burst Allowance:** Short spikes above the sustained rate are permitted using banked tokens (refill at steady rate).
## Rate Limit Headers
Every API response includes rate limit metadata:
```http theme={null}
X-RateLimit-Limit: 600 # Max requests per window
X-RateLimit-Remaining: 542 # Requests left in current window
X-RateLimit-Reset: 1698765432 # Unix timestamp when limit resets
X-RateLimit-Window: 60 # Window duration in seconds
```
**When rate limited:**
```http theme={null}
HTTP/1.1 429 Too Many Requests
Retry-After: 15 # Seconds until retry is safe
X-RateLimit-Remaining: 0
{
"success": false,
"status": 429,
"error": {
"message": "Rate limit exceeded",
"type": "TooManyRequestsError",
"details": {
"limit": 600,
"window": "1m",
"retry_after": 15
}
}
}
```
## Usage Pools & Metering
Usage is metered in **dollars**: each feature you enable is priced per modality unit (images, video minutes, document pages, text tokens, crawled web pages). Every tier includes a monthly dollar usage pool — $10 on Build, $100 on Scale — and once the pool is spent, additional usage bills at the rate card. See [Billing](/docs/platform/billing) for the full rate card, fetch it live via `GET /v1/billing/pricing` (no auth required), or quote a workload before running it with `POST /v1/organizations/billing/estimate`.
What's metered:
| What | How it bills |
| ------------------- | ----------------------------------------------------------------------------- |
| **Ingestion** | Per modality unit × features enabled — see the [rate card](/docs/platform/billing) |
| **Storage** | \$0.33/GB-month, all-in |
| **Reads / queries** | Included per tier; \$2 per 1M queries beyond included |
### Monitoring Usage
Check current spend via the Usage API:
```bash theme={null}
GET /v1/organizations/usage
```
The response reports your spend in dollars for the current billing period, broken down by category (ingestion, search, storage), along with your remaining pool balance and the pool reset date. The Studio **Billing** page shows the same numbers with charts and trends.
**Set alerts:**
* 80% of pool spent → warning
* 95% of pool spent → critical
* Pool exhausted → usage continues; overage bills at the [rate card](/docs/platform/billing)
## Resource Quotas
### Per-Organization Limits
| Resource | Free Tier | Pro Tier | Enterprise |
| --------------- | ----------- | -------------- | ---------- |
| **Namespaces** | 1 | 10 | Unlimited |
| **Collections** | 5 | 50 | Unlimited |
| **Buckets** | 5 | 50 | Unlimited |
| **Documents** | 10,000 | 1,000,000 | Unlimited |
| **Retrievers** | 3 | 50 | Unlimited |
| **Taxonomies** | 2 | 20 | Unlimited |
| **Clusters** | 1 | 10 | Unlimited |
| **API Keys** | 2 | 10 | Unlimited |
| **Batch Size** | 100 objects | 10,000 objects | Custom |
### Enforcement
When a quota is exceeded:
```http theme={null}
HTTP/1.1 403 Forbidden
{
"success": false,
"status": 403,
"error": {
"message": "Collection quota exceeded",
"type": "QuotaExceededError",
"details": {
"resource": "collections",
"current": 5,
"limit": 5,
"tier": "free"
}
}
}
```
## Scaling Strategies
### 1. Optimize Request Patterns
**Problem:** Hitting RPM limits during peak traffic
**Solutions:**
* **Batch operations** – use `/batch` endpoints to group objects/documents
* **Cache aggressively** – enable `cache_config` on retrievers to reduce redundant searches
* **Async processing** – submit batches asynchronously, poll task status instead of blocking
* **Load shedding** – deprioritize non-critical operations during peak hours
### 2. Distribute Load Across API Keys
**Problem:** Single API key hitting concurrency limit
**Solutions:**
* Issue separate API keys per service/team
* Use key rotation for different application environments (staging, prod)
* Monitor per-key usage: `GET /v1/organizations/usage/api-keys/{key_id}`
### 3. Reduce Spend
**Problem:** Burning through your monthly usage pool
**Solutions:**
| High-Cost Operation | Optimization |
| ------------------------- | ------------------------------------------------------------------- |
| **LLM generation stages** | Use smaller models (GPT-3.5 Turbo vs GPT-4), reduce `max_tokens` |
| **Frequent reprocessing** | Implement incremental updates instead of full reindexing |
| **Large batch ingestion** | Deduplicate objects before processing, filter out low-value content |
| **Exploratory searches** | Apply pre-filters to reduce search scope, lower `limit` values |
| **Web search stages** | Cache results with long TTL, fallback to internal collections |
### 4. Upgrade Tier
**When to upgrade:**
* Consistently hitting rate limits (>3 429 errors per hour)
* Usage pool >90% spent with 10+ days left in billing cycle
* Need for higher concurrency or batch sizes
* Require custom SLAs or dedicated infrastructure
Contact sales via "Talk to Engineers" CTA for enterprise pricing.
## Handling Rate Limit Errors
### Exponential Backoff
Implement retry logic with exponential backoff:
```python theme={null}
import time
import requests
def api_call_with_retry(url, headers, max_retries=5):
for attempt in range(max_retries):
resp = requests.post(url, headers=headers, json=payload)
if resp.status_code == 429:
retry_after = int(resp.headers.get("Retry-After", 2 ** attempt))
print(f"Rate limited. Retrying in {retry_after}s...")
time.sleep(retry_after)
continue
return resp
raise Exception("Max retries exceeded")
```
### Circuit Breaker Pattern
Prevent cascading failures when rate limits are sustained:
```python theme={null}
class CircuitBreaker:
def __init__(self, failure_threshold=5, timeout=60):
self.failure_count = 0
self.failure_threshold = failure_threshold
self.timeout = timeout
self.last_failure_time = None
self.state = "CLOSED" # CLOSED, OPEN, HALF_OPEN
def call(self, func, *args, **kwargs):
if self.state == "OPEN":
if time.time() - self.last_failure_time > self.timeout:
self.state = "HALF_OPEN"
else:
raise Exception("Circuit breaker is OPEN")
try:
result = func(*args, **kwargs)
if self.state == "HALF_OPEN":
self.state = "CLOSED"
self.failure_count = 0
return result
except RateLimitError:
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = "OPEN"
raise
```
### Graceful Degradation
When rate limited, fall back to cached or reduced-quality results:
```python theme={null}
def search_with_fallback(query):
try:
return mixpeek.retrievers.execute(retriever_id, inputs={"query": query})
except RateLimitError:
# Fallback to cached results or simpler search
return cached_search(query) or simplified_search(query)
```
## Endpoint-Specific Limits
Some endpoints have additional constraints:
| Endpoint | Special Limit | Reason |
| --------------------- | ------------------------------------------ | ----------------------------------------- |
| **Batch Submit** | 1 submission per batch every 60s | Prevents duplicate processing |
| **Cluster Execution** | 1 concurrent execution per cluster | Resource-intensive operation |
| **Web Search Stages** | 10 queries per minute (external API limit) | Third-party rate limit passthrough |
| **LLM Generation** | 100K tokens per minute | Model provider constraint |
| **Document List** | Max 10,000 results per query | Pagination required for large collections |
## Monitoring & Alerting
### Proactive Monitoring
Track these metrics to avoid surprises:
1. **Rate limit utilization** – alert at 80% of RPM limit
2. **Spend rate** – project end-of-month spend based on current trend
3. **Concurrent request count** – alert when approaching tier limit
4. **429 error frequency** – spike indicates need for optimization or upgrade
### Recommended Alerts
```yaml theme={null}
- name: "Rate Limit Warning"
condition: rate_limit_remaining < 20% of limit
action: Log warning, consider caching/batching
- name: "Usage Pool Critical"
condition: usage_pool_remaining < 5% AND days_left > 5
action: Upgrade tier or optimize high-cost operations
- name: "Sustained Rate Limiting"
condition: 429_errors > 10 in 5 minutes
action: Activate circuit breaker, alert on-call engineer
- name: "Quota Breach"
condition: Resource creation fails with QuotaExceededError
action: Archive unused resources or upgrade tier
```
## Best Practices
<AccordionGroup>
<Accordion title="Implement client-side rate limiting">
Don't rely solely on server enforcement. Implement token bucket or leaky bucket algorithms in your client to smooth request distribution and avoid bursts.
</Accordion>
<Accordion title="Cache responses aggressively">
Enable retriever-level caching with appropriate TTLs. For exploratory queries, cache for 5-15 minutes. For stable queries (e.g., product search), cache for hours.
</Accordion>
<Accordion title="Use batch operations">
Single-object operations consume rate limit budget faster. Batch 10-100 operations per request when possible.
</Accordion>
<Accordion title="Monitor per-API-key usage">
Isolate noisy services by assigning separate API keys. Throttle or upgrade only the high-volume keys instead of affecting the entire org.
</Accordion>
<Accordion title="Set budget limits on retrievers">
Configure `budget_limits` to prevent runaway costs from exploratory or LLM-heavy pipelines.
</Accordion>
<Accordion title="Paginate large result sets">
Use `offset` and `limit` parameters instead of requesting thousands of documents at once. This reduces latency and spend.
</Accordion>
</AccordionGroup>
## Enterprise Options
For organizations with sustained high volume:
* **Custom rate limits** – negotiate RPM/RPS based on traffic patterns
* **Reserved capacity** – pre-allocate Engine workers and inference quota
* **Dedicated infrastructure** – isolated MVS cluster, Redis, and Ray head nodes
* **Pool sharing** – share a usage pool across multiple sub-organizations
* **SLA guarantees** – contractual uptime and p99 latency commitments
Contact sales for custom pricing and limits.
## FAQ
<AccordionGroup>
<Accordion title="Do rate limits apply per API key or per organization?">
Rate limits are enforced at the **API key level**, but concurrent request limits apply at the **organization level**. This allows you to distribute load across multiple keys while respecting org-wide concurrency caps.
</Accordion>
<Accordion title="Are retries counted against rate limits?">
Yes. Every request, including retries, counts toward your RPM/RPS limits. Implement exponential backoff to avoid wasting quota on rapid retries.
</Accordion>
<Accordion title="Can I request a temporary quota increase?">
Yes. Contact support with your use case (e.g., annual reindexing, event-driven spike). We can provision temporary pool boosts or rate limit exemptions.
</Accordion>
<Accordion title="What happens if I exceed storage quota?">
New document creation fails with a `QuotaExceededError`. Existing documents remain queryable. Delete unused documents or upgrade tier to restore write access.
</Accordion>
<Accordion title="Do cached retriever responses bill usage?">
No. Cache hits are free and don't bill against your usage pool. Maximize cache hit rate to reduce spend.
</Accordion>
</AccordionGroup>
## Next Steps
* Monitor usage via [Organization Usage API](/docs/api-reference/organization-usage/get-org-usage)
* Review [Analytics Overview](/docs/operations/analytics-overview) for cost optimization strategies
* Configure [Webhooks](/docs/operations/webhooks) to alert on quota thresholds
* Optimize retriever performance with [Caching Strategies](/docs/overview/caching)
# Security & Tenancy
Source: https://docs.mixpeek.com/docs/operations/security
Authentication, authorization, isolation, and operational safeguards
Mixpeek enforces organization-level authentication, namespace isolation, and stage-level validation across the entire stack. This page summarizes the security model and operational protections you should configure in production.
## Authentication
* **Header**: `Authorization: Bearer <api_key>`
* API keys belong to an organization; keys can be rotated, revoked, or scoped per environment.
* Sensitive operations (e.g., creating namespaces, rotating keys) require elevated permissions.
See [API Keys](/docs/operations/api-keys) for the full lifecycle: creating keys, setting `read`/`write`/`delete`/`admin` permissions and resource scopes, rotating and revoking, per-key rate limits, usage attribution, and end-user (`principal_id`) keys.
## Namespace Isolation
* **Header**: `X-Namespace: <namespace_id or namespace_name>`
* Every MongoDB query filters on `namespace_id`; indexes ensure isolation at scale.
* [MVS](https://mixpeek.com/mvs) uses one namespace per namespace (`ns_<namespace_id>`); payload filters ensure cross-namespace safety.
* Redis cache keys and Ray job metadata include namespace identifiers.
### Dual Identifier Model
| Identifier | Visible? | Purpose |
| ----------------- | -------- | ------------------------------------------ |
| `organization_id` | Yes | User-facing identifier in API responses |
| `internal_id` | No | Primary key for service-to-service lookups |
| `namespace_id` | Yes | Isolation boundary for data and compute |
Keep `internal_id` secret; it is intentionally absent from public APIs.
## Authorization & Rate Limits
* Routes declare required permission levels (`read`, `write`, `delete`, `admin`).
* Rate limits enforced via Redis middleware; set per-plan and per-route to protect backends.
* Tasks and retriever executions are metered in dollars; analytics endpoints expose usage metrics for billing reconciliation.
## Secrets & Credentials
Mixpeek provides an encrypted secrets vault for storing sensitive credentials like API keys. Secrets are encrypted at rest using Fernet symmetric encryption and are never exposed in API responses.
### Organization Secrets Vault
Store and manage secrets via the API:
<CodeGroup>
```bash Create a Secret theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "openai_api_key",
"secret_value": "sk-proj-abc123..."
}'
```
```bash List Secrets theme={null}
# Returns secret names only - values are never exposed
curl "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_API_KEY"
```
```bash Update a Secret theme={null}
curl -X PUT "https://api.mixpeek.com/v1/organizations/secrets/openai_api_key" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_value": "sk-proj-new-key..."
}'
```
```bash Delete a Secret theme={null}
curl -X DELETE "https://api.mixpeek.com/v1/organizations/secrets/openai_api_key" \
-H "Authorization: Bearer YOUR_API_KEY"
```
</CodeGroup>
### Bring Your Own Key (BYOK)
Use your own LLM API keys instead of Mixpeek's default keys. This gives you:
| Benefit | Description |
| ---------------- | --------------------------------------------- |
| **Cost Control** | Use your own LLM provider account and billing |
| **Rate Limits** | Use your own rate limits instead of shared |
| **Compliance** | Keep API calls under your own account |
| **Key Rotation** | Rotate keys without modifying retrievers |
#### Supported Providers
| Provider | Secret Name Example | Models |
| --------- | ------------------- | ---------------------------------------------- |
| OpenAI | `openai_api_key` | gpt-4o, gpt-4o-mini |
| Anthropic | `anthropic_api_key` | claude-3-haiku, claude-3-sonnet, claude-3-opus |
| Google | `google_api_key` | gemini-3.1-flash-lite, gemini-2.5-pro |
#### Apply a Key to All LLM Operations
The fastest way to use your own key is to set it as the **organization-wide default**. Once configured, it automatically applies to every LLM operation — extractors, retrievers, clustering, taxonomy inference, and manifest generation — with no per-stage configuration needed.
<Steps>
<Step title="Store your key in the secrets vault">
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "openai_api_key",
"secret_value": "sk-proj-abc123..."
}'
```
```python Python theme={null}
from mixpeek import Mixpeek
mx = Mixpeek(api_key="YOUR_API_KEY")
mx.organizations.secrets.create(
secret_name="openai_api_key",
secret_value="sk-proj-abc123..."
)
```
</CodeGroup>
</Step>
<Step title="Set it as the org-wide default for that provider">
<CodeGroup>
```bash cURL theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/organizations" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"default_llm_credentials": {
"openai": "openai_api_key"
}
}'
```
```python Python theme={null}
mx.organizations.update(
default_llm_credentials={
"openai": "openai_api_key"
}
)
```
</CodeGroup>
</Step>
</Steps>
You can configure defaults for multiple providers at once:
```json theme={null}
{
"default_llm_credentials": {
"openai": "my_openai_key",
"anthropic": "my_anthropic_key",
"google": "my_gemini_key"
}
}
```
<Tip>
You can also do this from Studio: when creating a secret, toggle **"Use as default LLM key"** and it will automatically be applied org-wide for the detected provider.
</Tip>
<Note>
Values in `default_llm_credentials` are **secret names** (not raw API keys). The actual keys are stored encrypted in your secrets vault and resolved at runtime.
</Note>
#### Override a Key on a Specific Stage
If you need a different key for a particular retriever stage, extractor, or cluster config, set `api_key` directly using the `{{secrets.name}}` template syntax. This overrides the org-wide default for that operation only.
```json theme={null}
{
"stages": [
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"provider": "openai",
"model": "gpt-4o-mini",
"prompt": "Summarize this document.",
"output_field": "summary",
"api_key": "{{secrets.openai_api_key}}"
}
}
}
]
}
```
#### Credential Resolution Order
Mixpeek resolves LLM credentials in this order (highest priority first):
1. **Per-resource** `api_key` / `{{secrets.name}}` — explicit key on a specific stage or extractor
2. **Organization default** — set once via `default_llm_credentials`, applied everywhere
3. **Mixpeek platform keys** — used when no custom key is configured (usage charged to your Mixpeek account)
#### Security
* **Encryption at rest**: All secrets are encrypted using Fernet symmetric encryption with a dedicated encryption key
* **Zero exposure**: Secret values are never returned in API responses — only secret names are visible
* **Per-request resolution**: Credentials are decrypted on-demand for each LLM call, not cached globally
* **No cross-tenant leakage**: Each organization's credentials are isolated; provider instances with custom keys are never shared between organizations
* **Audit trail**: Secret access and organization configuration changes are logged for compliance
### Security Best Practices
* **Rotate credentials regularly** – Update secrets via the API without changing retriever configurations
* **Use IAM roles** – For S3/GCS access, prefer IAM roles over long-lived access keys
* **Audit access logs** – Monitor secret access patterns for anomalies
* **Scope API keys** – Issue environment-specific Mixpeek API keys (dev, staging, prod)
## Data Protection
* **Storage**: rely on encryption at rest provided by MongoDB Atlas, [MVS](https://mixpeek.com/mvs), or your infrastructure.
* **Transit**: require TLS for API endpoints and Ray Serve; use mTLS or network policies for cross-service traffic when available.
* **Backups**: configure automated backups for MongoDB and MVS; version S3 buckets with lifecycle policies.
## Operational Safeguards
* Enable `/v1/health` probes in load balancers to route around unhealthy instances.
* Use webhooks to detect ingestion completion; failed webhook deliveries remain retriable in MongoDB.
* Monitor rate-limit counters and task failure rates to spot abusive or buggy clients.
* Log request IDs and namespace IDs to correlate incidents quickly.
## Hardening Checklist
1. **Network** – restrict API access to trusted origins, configure CORS, and use private networking for backend services.
2. **Auth** – issue scoped API keys, expire unused keys, enable audit logging.
3. **Secrets** – manage via Vault, AWS Secrets Manager, GCP Secret Manager, or Kubernetes secrets with rotation.
4. **Tenancy** – adopt one namespace per environment/tenant; enforce `X-Namespace` always.
5. **Monitoring** – alert on health endpoint status, rate-limit breaches, or repeated 401/403 responses.
## References
* [API Keys](/docs/operations/api-keys)
* [Namespaces](/docs/ingestion/namespaces)
* [Observability](/docs/operations/observability)
* [Webhooks](/docs/operations/webhooks)
* [Tasks](/docs/processing/tasks)
* [Health Check](/docs/api-reference/health/healthcheck)
# Webhooks
Source: https://docs.mixpeek.com/docs/operations/webhooks
Respond to Mixpeek events without polling
Webhooks notify your systems when ingestion, enrichment, or retrieval events occur. The Engine writes events to MongoDB, Celery Beat dispatches them, and Mixpeek delivers HTTP POST requests (or channel-specific messages) until acknowledged.
## Event Flow
```
Engine → MongoDB (webhook_events) → Celery Beat → Celery Worker → HTTP POST → Your endpoint
```
* Events persist until the worker receives a 2xx response.
* Retries use exponential backoff; failures remain in MongoDB for inspection.
* Delivery includes cache scopes so you can invalidate selectively.
## Common Event Types
| Event | Trigger | Payload Highlights |
| ------------------------------ | ------------------------------------------------------------------- | -------------------------------------------------------- |
| `collection.documents.written` | Engine finishes writing documents to [MVS](https://mixpeek.com/mvs) | `collection_id`, `document_ids`, `index_signature` |
| `object.created` | Object registered in a bucket | `bucket_id`, `object_id`, metadata snapshot |
| `batch.completed` | Batch processing succeeded | `batch_id`, `collection_ids`, counts |
| `cluster.completed` | Clustering run finished | `cluster_id`, `run_id`, artifact URIs |
| `taxonomy.materialized` | Taxonomy enrichment completed | `taxonomy_id`, `collection_id`, `updated_document_count` |
Use `/api-reference/webhooks/list-webhooks` to see the full catalog.
## Create a Webhook
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/organizations/webhooks" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"webhook_name": "prod-notifications",
"event_types": [
"collection.documents.written",
"cluster.completed"
],
"channels": [
{
"channel": "webhook",
"configs": {
"url": "https://hooks.example.com/mixpeek",
"headers": { "X-Mixpeek-Secret": "super-secret"},
"timeout": 10
}
}
]
}'
```
Mixpeek also supports Slack, email, and SMS channels via channel-specific configs.
## Payload Structure
```json theme={null}
{
"event_id": "evt_123",
"event_type": "collection.documents.written",
"occurred_at": "2025-10-28T10:03:22Z",
"namespace_id": "ns_prod",
"subject": {
"type": "collection",
"id": "col_products"
},
"metadata": {
"document_count": 100,
"collection_id": "col_products",
"index_signature": "sig_xyz789"
},
"cache_scope": {
"scope": "collection",
"collection_id": "col_products"
}
}
```
Use `event_id` for deduplication and store payloads for auditing.
## Security & Reliability
* Require HTTPS endpoints; reject plaintext URLs.
* Include a shared secret header (`X-Mixpeek-Secret`) and verify before processing.
* Respond quickly (\<10s). Offload heavy work to background jobs and return `200`.
* Use idempotent handlers; Mixpeek may retry on failure or timeout.
* Monitor webhook delivery with your logging pipeline; correlate by `event_id`.
## Operational Tips
1. Subscribe only to the events you need to reduce noise.
2. Combine webhook notifications with the Tasks API for full status context.
3. Use cache scopes to invalidate retriever caches efficiently (`collection`, `namespace`, or `key`).
4. Store webhook definitions in infrastructure-as-code so environments stay consistent.
5. Alert on sustained non-2xx responses—Mixpeek will keep retrying, but you should fix endpoint issues quickly.
## Inbound Webhooks (from upstream providers)
In addition to **outbound** webhooks (Mixpeek → your endpoint), Mixpeek can receive **inbound** webhooks from external systems to trigger ingestion and lifecycle events.
### Sync lifecycle webhooks
Some sync integrations support inbound webhooks for **lifecycle events** (delete, update) on previously synced assets:
| Provider | Endpoint | Events handled | Use case |
| -------- | --------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Mux** | `POST /v1/webhooks/mux/{connection_id}` | `video.asset.deleted`, `video.asset.updated` | Cascade-delete bucket objects when a Mux asset is deleted. Re-evaluate metadata filters when asset metadata changes. See [Mux integration → Webhooks](/docs/integrations/object-storage/mux#webhooks-delete-and-update). |
These webhooks share common properties:
* Scoped to a single connection by URL path (no Authorization header needed).
* Signatures verified with `hmac.compare_digest` and provider-specific HMAC schemes.
* Unhandled events acknowledged with `200` and `handled: false` so retries stop cleanly.
## References
* [Create Webhook](/docs/api-reference/webhooks/create-webhook)
* [List Webhooks](/docs/api-reference/webhooks/list-webhooks)
* [Delete Webhook](/docs/api-reference/webhooks/delete-webhook)
* [Mux cascade-delete webhook](/docs/integrations/object-storage/mux#cascade-delete-via-webhooks)
* [Tasks](/docs/processing/tasks) – track the jobs that trigger webhook events
* [Security](/docs/operations/security) – authentication, tenancy, and secret management
# Caching & Signatures
Source: https://docs.mixpeek.com/docs/overview/caching
Keep retrieval fast without serving stale data
Mixpeek layers several caches to deliver low-latency responses while guaranteeing consistency. Every layer relies on deterministic signatures so you never serve results from an outdated index.
## Cache Layers
| Layer | Scope | Backing Store | TTL | Purpose |
| ------------------ | ---------------------------------------------- | ------------------------------ | ------------------------------- | ------------------------------------------------------------- |
| Retriever response | Full execution output | Redis | 1 hour (configurable) | Return entire execution payload instantly on repeated queries |
| Stage output | Individual stages (`feature_search`, `rerank`) | Redis | 1 hour (configurable per stage) | Reuse expensive stages across similar queries |
| Inference | Embeddings & rerankers | Redis | \~1 hour | Avoid recomputing identical model inferences |
| Document features | Stored vectors/payloads | [MVS](https://mixpeek.com/mvs) | Permanent | Reuse ingestion-time features for future queries |
## How It Works
```mermaid theme={null}
graph LR
Q[Query] --> RC{Retriever Cache?}
RC -->|HIT| R1[Return Cached Response]
RC -->|MISS| P[Execute Pipeline]
P --> SC{Stage Cache?}
SC -->|HIT| SK[Skip Stage]
SC -->|MISS| EX[Run Stage]
EX --> SS[Store Stage Result]
SK --> N[Next Stage]
SS --> N
N --> ST[Store Full Response]
ST --> R2[Return Fresh Response]
```
On cache hit at the **retriever level**, the entire pipeline is skipped — response includes a `cached_at` timestamp so you can verify freshness. On cache hit at the **stage level**, only that stage is skipped and the rest of the pipeline continues.
## Index Signatures
Each collection stores an `index_signature` in MongoDB. The signature hashes:
* Collection configuration (feature extractor, passthrough fields)
* Document count and vector dimensions
* Timestamp of last ingestion event (with debounce logic)
Retriever cache keys include `index_signature`, so whenever ingestion updates the collection the signature changes and cached query responses automatically miss.
```text theme={null}
cache:retriever:quickstart-search:
hash(
inputs,
filters,
pagination,
collection_signature="xyz789"
)
```
## Response Cache Metadata
Retriever execution responses include cache information:
```json theme={null}
{
"execution_id": "exec_abc123",
"status": "completed",
"cached_at": 1714150000.5,
"documents": [...],
"stage_statistics": {
"stages": {
"text_search": {
"cache_hit": true,
"cached_at": 1714150000.2,
"duration_ms": 0.5
},
"rerank": {
"cache_hit": false,
"cached_at": null,
"duration_ms": 45.3
}
}
}
}
```
* **`cached_at`** (top-level) — Unix timestamp when the full response was cached. Present only on retriever-level cache hits. Compute freshness: `time.time() - cached_at`.
* **`cache_hit`** (per stage) — Whether this stage's result came from stage cache.
* **`cached_at`** (per stage) — Unix timestamp when this stage result was cached.
## Bypassing Cache
Force a fresh execution with `skip_cache`:
```bash theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/<id>/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H 'Content-Type: application/json' \
-d '{ "inputs": { "query": "smart speaker" }, "skip_cache": true }'
```
## Stage-Level Controls
Control caching per stage via `cache_behavior` and `cache_ttl_seconds`:
```json theme={null}
{
"stages": [
{
"stage_name": "text_search",
"config": {
"parameters": {
"cache_behavior": "auto",
"cache_ttl_seconds": 600
}
}
},
{
"stage_name": "rerank",
"config": {
"parameters": {
"cache_behavior": "disabled"
}
}
}
]
}
```
**`cache_behavior` options:**
* `auto` (default) — Cache deterministic operations automatically
* `disabled` — Skip caching entirely for this stage
* `aggressive` — Cache even non-deterministic operations (use with caution)
## Inference Cache
The Engine caches model calls using a hashed payload of `(model_name, inputs, parameters)`. Use it to:
* Reuse embeddings for identical prompts or documents
* Skip recomputing reranking scores for popular queries
* Short-circuit repeated LLM-based filters with static criteria
## Cache Invalidation
Caches are invalidated automatically on:
| Event | Scope |
| ---------------------------- | --------------------------------------------- |
| Document ingestion completes | Collection-level (via index signature change) |
| Retriever deleted | All keys for that retriever |
| Collection deleted/updated | All keys for that collection |
| Namespace deleted | All keys in namespace |
Manual invalidation is also available:
```bash theme={null}
DELETE /v1/retrievers/{retriever_id}/cache
```
## Monitoring Cache Performance
* Use **`GET /v1/analytics/retrievers/{id}/cache-performance`** for hit/miss ratios and latency deltas.
* `stage_statistics` inside retriever responses flag `cache_hit` per stage.
* Redis namespaces per feature (e.g., `cache:retriever:...`) make it easy to inspect keys if needed.
## Best Practices
* Caching is on by default with `cache_behavior: "auto"` — no setup needed.
* Use `skip_cache: true` for debugging or when you need guaranteed-fresh results.
* Disable stage caching for stages with time-sensitive inputs (`now()`, `random()`).
* Use stage caching when reranking or feature search is the bottleneck.
* Use inference caching for expensive LLM or GPU workloads — even small hit rates pay off.
# Billing & Pricing
Source: https://docs.mixpeek.com/docs/platform/billing
Three questions set your price: what kind of files, how much content, and what you want to search by — tiers include a monthly usage pool
## How pricing works
Your price is the answer to three questions, in this order:
1. **What kind of files?** Video, image, audio, PDF/document, text, or web pages — the modality.
2. **How much content?** Each file type counts in its honest natural unit: images per file, video and audio per minute, PDFs per page, text per token, web per crawled page.
3. **What do you want to search by?** Visual similarity, faces, on-screen text, document layout, transcripts, ... — these plain-language capabilities are **[features](/docs/processing/features)**.
For each file type: **price = amount × (base rate + the search-by features you enable)**. Each tier is a flat monthly fee that includes a usage pool; anything beyond the pool bills at the same rate card.
**Worked example** — 100 videos averaging 2 minutes each, searchable by what's on screen (base video search) plus who appears (faces):
1. What kind of files: video → bills per minute
2. How much: 100 videos × 2 min = **200 minutes**
3. Search by: video search (base, $0.05/min) + faces (+$0.10/min)
200 min × ($0.05 + $0.10) = \*\*$30**. On Scale, the $100 monthly pool covers it entirely; on Build, $10 comes out of the pool and $20 bills as overage.
Everything is in dollars — rates, usage, invoices, and quotes are dollar-denominated end to end. Quote any planned ingestion before you run it with the [estimate endpoint](#estimate-before-you-run); it uses the same rating engine that bills you.
For current numbers, see the [pricing page](https://mixpeek.com/pricing). Manage your plan and payment method in [Studio Billing](https://studio.mixpeek.com/billing).
<Note>
Under the hood, each feature maps to a feature extractor — that deeper layer (which extractor implements a feature, extractor options, bring-your-own extractors) is documented in [Features](/docs/processing/features#how-this-relates-to-extractors). You never need it to understand your bill.
</Note>
***
## Plans
<Note>
**Plan gating:** `GET /v1/organizations` returns `requires_plan`. When `true`,
the workspace must pick a plan before creating namespaces or running billable
work — gated endpoints return `403 PlanRequiredError` with a link to
[choose a plan](https://studio.mixpeek.com/signup/plan). There is no separate
"tier" endpoint: `account_type`, `credit_count`, and `requires_plan` on the
organization are the capability surface.
</Note>
Each tier is a **flat monthly fee that includes a usage pool** — a dollar allowance you can spend on any mix of modalities and features. Only usage beyond the pool bills as overage, at the same rate card.
| | Build | Scale | Enterprise |
| ----------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------- | ---------- |
| **Monthly price** | \$25 | \$250 | Custom |
| **Included usage pool** | \$10 | \$100 | Custom |
| **Pool buys, e.g.** | \~6,600 images **or** 200 video minutes **or** \~6,600 pages | \~66,000 images **or** 2,000 video minutes **or** \~66,000 pages | — |
| **Overage** | Rate card | Rate card | Contract |
| **Custom extractors** | Included | Included | Included |
| **Support** | Email | Priority + private Slack | Dedicated |
The pool is **fungible** — it isn't a per-modality cap. The natural-unit equivalents above are just the pool divided by each base rate, so you can compare tiers at a glance. Spend it on whatever mix you actually ingest.
<Note>
Enterprise runs on dedicated single-tenant infrastructure with custom contracts — see [Single Tenant](/docs/resources/single-tenant) or [contact sales](https://mixpeek.com/contact).
</Note>
***
## The rate card
**The API is the source of truth for rates.** `GET /v1/billing/pricing` is public (no auth) and returns the live rate card — the same payload that studio, the homepage, and the billing engine read, so displayed numbers can never disagree with billed numbers:
```bash theme={null}
curl "https://api.mixpeek.com/v1/billing/pricing"
```
```json theme={null}
{
"pricing_model": "v2",
"currency": "usd",
"placeholder_rates": false,
"modalities": [
{
"modality": "video",
"unit": "minute",
"unit_display": "per minute",
"base": { "key": "video_search", "name": "Video search (scene embeddings)", "rate_usd": 0.05, "per": 1 },
"addons": [
{ "key": "faces", "name": "Face detection + identity", "kind": "addon", "rate_usd": 0.10, "per": 1 },
{ "key": "onscreen_text", "name": "On-screen text (video OCR)", "kind": "addon", "rate_usd": 0.10, "per": 1 },
{ "key": "audio_fingerprint", "name": "Audio fingerprint (in video)", "kind": "addon", "rate_usd": 0.01, "per": 1 },
{ "key": "multimodal_understanding", "kind": "external", "pricing_note": "usage-based (external inference)" },
{ "key": "clustering", "kind": "included", "pricing_note": "included — no additional charge" },
{ "key": "taxonomy_enrichment", "kind": "included", "pricing_note": "included — no additional charge" }
]
}
],
"usage_pools": {
"build": {
"flat_monthly_usd": 25.0,
"usage_pool_usd": 10.0,
"pool_equivalents": { "images": 6667, "video_minutes": 200, "pages": 6667, "text_tokens": 5000000 }
}
},
"storage": { "vector_storage_usd_per_gb_month": 0.33 },
"reads": { "overage_usd_per_1m_queries": 2.0 },
"full_res_multiplier": 2.0
}
```
Response abbreviated — the live payload covers all six modalities and every tier. How to read it:
* **`modalities[].base`** — the per-unit rate for making that modality searchable. Creating any collection for a modality gets you its base feature.
* **`modalities[].addons`** — features you opt into, priced per the same unit. `kind: "external"` rows are backed by external LLM inference and priced as usage-based passthrough (never flat-rated). `kind: "included"` rows (clustering, taxonomy enrichment) cost nothing extra.
* **`usage_pools`** — each tier's flat fee and included dollar pool, with derived natural-unit equivalents.
* **`placeholder_rates`** — `false` means the rates are live and billable. (The nested `v2` object in the payload is a deprecated alias of these same top-level keys and will be removed.)
<Tip>
Don't hardcode rates into your own tooling — read them from `GET /v1/billing/pricing`, or quote concrete workloads with the [estimate endpoint](#estimate-before-you-run).
</Tip>
### How a batch is priced
When you submit a batch, each object bills as: **modality units x (base rate + enabled add-on rates)**.
Example at the rates shown above: a 10-minute video in a collection with `faces` and `onscreen_text` enabled bills 10 min x ($0.05 + $0.10 + $0.10) = **$2.50\*\*.
* **Measured units** — video/audio minutes come from measured duration (not file size), pages from actual page counts. Anything estimated at submit is trued up at batch completion, in both directions.
* **Batch minimum** — batches bill at least **\$0.01**.
* **Normalization included** — content is normalized once at ingest (video to 720p mezzanine, images capped, audio to 16kHz mono). Originals stay untouched in your bucket. Collections that opt out with `full_res: true` bill at a **2x multiplier**. See [full resolution opt-out](/docs/processing/features#full-resolution-opt-out).
* **Deduplication** — re-ingesting content Mixpeek has already processed (same content hash) skips extraction rather than re-billing full processing.
### Storage
Vector and index storage bills at a single all-in rate: **\$0.33 per GB-month** (`storage.vector_storage_usd_per_gb_month`). No separate per-vector charges.
### Reads (queries)
Tiers include a generous query allowance; beyond it, reads bill at **\$2 per 1M queries** (`reads.overage_usd_per_1m_queries`).
***
## Estimate before you run
`POST /v1/organizations/billing/estimate` quotes planned ingestion — same rating engine, same rate card, so the quote matches the eventual charge:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/billing/estimate" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"items": [{ "mime_type": "video/mp4", "minutes": 42 }],
"features": ["faces"]
}'
```
The response itemizes dollars per (modality, feature), reports how much falls inside your remaining usage pool, and what would bill as overage. Full request/response details in [Features → Estimate](/docs/processing/features#estimate-before-you-run).
***
## Tracking usage
Current-period usage is available in [Studio Billing](https://studio.mixpeek.com/billing) and via the [usage API](/docs/api-reference/organization-billing/get-current-usage). Usage reports break down dollars by modality and feature:
* units consumed per modality (minutes, images, pages, tokens) and per feature
* storage GB-month and reads
* pool consumption, overage, and the period's total in dollars
You can set spending caps to limit overage through [Studio Billing](https://studio.mixpeek.com/billing) or the [spending caps API](/docs/api-reference/organization-billing/get-spending-caps).
***
## How billing works
<Steps>
<Step title="Sign up and choose a plan">
Create an account at [studio.mixpeek.com](https://studio.mixpeek.com). Upgrade to Build or Scale from the billing page.
</Step>
<Step title="Add a payment method">
Paid plans require a card on file via Stripe. Add or update your card in [Studio Billing](https://studio.mixpeek.com/billing) or through the [payment method API](/docs/api-reference/organization-billing/setup-payment-method).
</Step>
<Step title="Use Mixpeek">
Your usage pool is available immediately and resets each cycle. Usage is metered continuously in dollars and visible in the dashboard.
</Step>
<Step title="Receive your invoice">
Stripe generates a monthly invoice: your flat plan fee plus any overage, itemized by modality and feature. Invoices are in [Studio Billing](https://studio.mixpeek.com/billing) and via the [invoices API](/docs/api-reference/organization-billing/list-invoices).
</Step>
</Steps>
***
## MVS Standalone
Using Mixpeek's vector store with your own embeddings (no extraction)? MVS bills separately: storage at \$0.33/GB-month all-in, with per-tier query and write allowances. See the live `mvs_plans` and `mvs_usage_rates` in `GET /v1/billing/pricing` and the [MVS overview](/docs/vector-store/overview).
***
## FAQ
<AccordionGroup>
<Accordion title="What happened to credits?">
Credits are gone as a customer-facing concept — every surface (rates, usage, invoices, quotes) is dollar-denominated. If you see a legacy `credits` field in an older API response, it's an internal ledger unit worth \$0.001 (a milli-dollar); the dollar fields alongside it are canonical.
</Accordion>
<Accordion title="Can the quote and the charge disagree?">
No. The estimate endpoint and the metering pipeline call the same rating engine against the same rate card. Estimated units (e.g. duration unknown at submit) are trued up from measured values at batch completion — in both directions.
</Accordion>
<Accordion title="Is the usage pool a per-modality cap?">
No. It's a fungible dollar pool — the natural-unit equivalents shown on tier cards are illustrations (pool ÷ base rate), not limits. Spend it on any mix of modalities and features.
</Accordion>
<Accordion title="Do unused pool dollars roll over?">
No. The usage pool resets at the start of each billing cycle.
</Accordion>
<Accordion title="Can I switch plans mid-cycle?">
Yes. Upgrades take effect immediately and are prorated. Downgrades take effect at the start of the next billing cycle.
</Accordion>
<Accordion title="How are LLM-backed features priced?">
Features marked `kind: "external"` (e.g. `multimodal_understanding`) are backed by external LLM inference and priced as usage-based passthrough — their upstream cost is unbounded, so they're never flat-rated. The pricing payload flags them with a `pricing_note`.
</Accordion>
<Accordion title="How are custom extractors priced?">
A `custom:<plugin>` feature derives its per-unit rate from the compute profile the plugin declares — the same machinery that prices native features. Quote it like anything else via the estimate endpoint.
</Accordion>
<Accordion title="What happens when I exceed caps?">
Requests that exceed hard resource caps return `403` with type `QuotaExceededError` and an `upgrade_plan` hint. Overage within your plan simply bills at the rate card — no interruption.
</Accordion>
<Accordion title="What payment methods does Mixpeek accept?">
All major credit and debit cards via Stripe. Enterprise customers can pay by invoice with net-30 terms.
</Accordion>
<Accordion title="Is there an annual billing option?">
Annual billing is available for Scale and Enterprise. Contact sales for details.
</Accordion>
</AccordionGroup>
# Classify Content
Source: https://docs.mixpeek.com/docs/platform/enrichment
Auto-label documents with taxonomies, retriever enrichments, and annotations
<Frame>
<img alt="Content classification: documents are enriched via taxonomies (flat or hierarchical similarity matching) or retriever enrichments (pipeline execution with field write-back)" />
</Frame>
<Info>
For full configuration details, parameters, and advanced options, see the [Taxonomies reference](/docs/enrichment/taxonomies).
</Info>
## Taxonomies
Auto-classify documents by matching them against reference collections. Two types:
**Flat** — match each document against a single reference collection. When similarity exceeds the threshold, enrichment fields (SKU, category, label) are attached.
**Hierarchical** — parent/child nodes with inheritance. Documents traverse levels of refinement (brand → category → subcategory) using different features at each level.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/taxonomies" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"taxonomy_name": "product-categories",
"type": "flat",
"reference_collection_id": "'$REF_COLLECTION_ID'",
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"similarity_threshold": 0.75,
"enrichment_fields": ["category", "subcategory", "brand"]
}'
```
### When to Run
| Mode | Runs | Use case |
| ------------- | ---------------------------------------- | ------------------------------------- |
| `on_demand` | At query time as a retriever stage | Dynamic classification, A/B testing |
| `materialize` | After extraction, persists to collection | Stable labels, fast queries |
| `retroactive` | Reapplies when taxonomy updates | Backfill when reference data improves |
[Taxonomy API →](/docs/api-reference/taxonomies/create-taxonomy)
## Retriever Enrichments
Attach a retriever pipeline to a collection so it runs on every new document. The retriever executes, and selected result fields are written back to the document.
```bash theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/collections/$COLLECTION_ID" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"retriever_enrichments": [{
"retriever_id": "'$RETRIEVER_ID'",
"input_mappings": { "query_text": { "source": "payload", "path": "description" } },
"write_back_fields": { "category": { "mode": "first", "path": "results[0].metadata.category" } }
}]
}'
```
Use cases: auto-classify via LLM, cross-collection joins, label propagation from seed documents.
[Collection update API →](/docs/api-reference/collections/update-collection)
## Annotations
Explicit human decisions with full provenance — the ground truth layer for compliance, review workflows, and improving retrieval quality over time.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/annotations" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"document_id": "doc_abc",
"collection_id": "col_xyz",
"retriever_id": "ret_123",
"execution_id": "exec_789",
"stage_name": "feature_search",
"label": "approved",
"confidence": 0.95,
"reasoning": "Matches reference product exactly",
"payload": { "sku": "SKU-001", "action": "keep" },
"actor_id": "user_456",
"actor_type": "human"
}'
```
### What Each Annotation Captures
| Field | Purpose |
| -------------------------------------------- | ------------------------------------------------------ |
| `document_id`, `collection_id` | What was reviewed |
| `retriever_id`, `execution_id`, `stage_name` | How it was surfaced |
| `label`, `confidence`, `reasoning` | The decision |
| `payload` | Structured workflow-specific data (SKU, action, notes) |
| `actor_id`, `actor_type` | Who decided (human or model) |
Annotations are stored independently from documents — they never modify the source data. Use them to build review queues, audit trails, and curated ground truth datasets.
### Bulk Operations
Process review queues at scale with the bulk API:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/annotations/bulk" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"create": [
{ "document_id": "doc_1", "collection_id": "col_xyz", "label": "approved" },
{ "document_id": "doc_2", "collection_id": "col_xyz", "label": "rejected", "reasoning": "Low quality match" }
],
"update": [],
"delete": []
}'
```
### The Feedback Loop
Annotations feed directly into the platform's learning cycle:
1. **Annotations** provide explicit ground truth for edge cases
2. **Learned fusion** uses annotations to auto-tune retriever stage weights
3. **Approved annotations** can be piped into reference collections, expanding your taxonomy's coverage
4. **Retroactive taxonomy application** reclassifies existing documents when annotations improve the reference set
[Annotation API →](/docs/api-reference/annotations/create-annotation) · [Bulk API →](/docs/api-reference/annotations/bulk-annotations)
## Choosing an Approach
| Goal | Use |
| ---------------------------------------------------- | ------------------------------------------------- |
| Auto-label with a reference catalog | Flat taxonomy (materialize mode) |
| Hierarchical classification (brand → category → SKU) | Hierarchical taxonomy |
| Auto-classify via LLM at ingest | Retriever enrichment with `llm_enrich` stage |
| Cross-collection joins (enrich from another dataset) | Retriever enrichment with `document_enrich` stage |
| Human review with audit trail | Annotations |
| Backfill when labels improve | Retroactive taxonomy application |
# Improve Relevance
Source: https://docs.mixpeek.com/docs/platform/improve-relevance
Make search better over time with interactions, fusion strategies, and evaluations
<Frame>
<img alt="Improve relevance: search results generate user interaction signals, which feed fusion strategies and evaluations in a feedback loop" />
</Frame>
## Interaction Signals
Capture implicit user behavior — clicks, views, dwell time, purchases — to feed into retrieval optimization.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers/interactions" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"execution_id": "exec_abc",
"feature_id": "doc_xyz",
"interaction_type": ["click"],
"position": 3,
"metadata": { "duration_ms": 4500 }
}'
```
`interaction_type` is always a JSON **array** (co-occurring signals of one action, e.g. `["click", "long_view"]`), and the document is `feature_id`. Track `position` for every signal — it is recorded for analytics and evaluation (NDCG and other rank-aware metrics).
### Signal Strength
| Signal | Weight | When to track |
| ------------------- | ------- | ----------------------------------------- |
| `click` | Medium | User clicked a result |
| `long_view` | High | Sustained engagement (pass `duration_ms`) |
| `add_to_cart` | High | Intent / funnel step |
| `purchase` | Highest | User completed a goal action |
| `negative_feedback` | Penalty | User disliked / hid the result |
See [Interaction Signals](/docs/retrieval/interactions#signal-strategy) for the full signal matrix and per-use-case patterns.
[Interaction API →](/docs/api-reference/retriever-interactions/create-interaction)
## Auto-Tune (Learned Fusion)
Auto-Tune automatically adapts fusion weights per user based on their interaction history. Instead of manually choosing weights, the system uses Thompson Sampling to learn the optimal blend of features for each user.
<CardGroup>
<Card title="How It Works" icon="brain" href="/retrieval/auto-tune">
Concept page — Thompson Sampling, context levels, reward signals
</Card>
<Card title="Reward Signals" icon="signal" href="/retrieval/reward-signals">
Configure which interactions drive learning and how much
</Card>
<Card title="Rollout Guide" icon="shield" href="/retrieval/auto-tune-rollout">
Traffic splitting, shadow mode, kill switch, per-user opt-out
</Card>
</CardGroup>
**Quick setup:**
<CodeGroup>
```python Python SDK theme={null}
# 1. Create a retriever with learned fusion
retriever = client.retrievers.create(
retriever_name="personalized-search",
stages=[{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"fusion": "learned",
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {"click": 1.0, "purchase": 3.0}
},
"final_top_k": 25
}
}
}]
)
# 2. Send interactions — learning happens automatically
client.retrievers.create_interaction(
feature_id="doc_123",
interaction_type=["click"],
position=2,
user_id="user_abc"
)
```
```bash cURL theme={null}
# 1. Create a retriever with learned fusion
curl -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "personalized-search",
"stages": [{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"fusion": "learned",
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {"click": 1.0, "purchase": 3.0}
},
"final_top_k": 25
}
}
}]
}'
# 2. Send interactions — learning happens automatically
curl -X POST "$MP_API_URL/v1/retrievers/interactions" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"feature_id": "doc_123",
"interaction_type": ["click"],
"position": 2,
"user_id": "user_abc"
}'
```
</CodeGroup>
For a step-by-step walkthrough, see the [Build a Feedback Loop](/docs/tutorials/feedback-loop) tutorial.
## Fusion Strategies
When a retriever has multiple search stages, fusion strategies determine how scores combine into the final ranking.
| Strategy | How it works | Best for |
| ------------------------------------------ | -------------------------------------------- | -------------------------------------- |
| **RRF** (Reciprocal Rank Fusion) | Combines ranks, not scores. `1/(k + rank)` | Default — works well with no tuning |
| **DBSF** (Distribution-Based Score Fusion) | Normalizes score distributions then averages | When scores have different scales |
| **Weighted** | Manual weights per stage | When you know which stage matters more |
| **Max** | Takes the highest score across stages | When any match is sufficient |
| **Learned** | Auto-tunes weights from interaction signals | When you have 500+ interactions |
Set `fusion` inside the `feature_search` stage parameters (alongside `searches` and `final_top_k`):
```json theme={null}
{
"fusion": "rrf"
}
```
For `"fusion": "learned"`, add a `learning_config` (see the Auto-Tune example above). Learned fusion uses Thompson Sampling to shift weight toward stages whose results users engage with; with zero interactions it behaves like `rrf` and transitions as signals accumulate.
## Evaluations
Measure retriever quality against ground truth datasets with standard IR metrics.
```bash theme={null}
# Create a ground truth dataset
curl -X POST "https://api.mixpeek.com/v1/retrievers/evaluations/datasets" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"dataset_name": "product-search-eval",
"queries": [
{
"query_id": "q1",
"query_input": {"query_text": "wireless headphones"},
"relevant_documents": ["doc_1", "doc_2"]
}
]
}'
# Run evaluation
curl -X POST "https://api.mixpeek.com/v1/retrievers/$RETRIEVER_ID/evaluations" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"dataset_name": "product-search-eval",
"evaluation_config": { "k_values": [1, 5, 10, 20] }
}'
```
The run is asynchronous. Poll `GET /v1/retrievers/{retriever_id}/evaluations/{evaluation_id}` for Precision, Recall, F1, NDCG, MAP, and MRR at your configured cutoffs. Use evaluations to compare retriever configurations before deploying changes.
[Evaluation API →](/docs/api-reference/retriever-evaluations/run-evaluation)
## Analytics
Monitor retriever performance in production:
* **Stage latency breakdown** — identify which stages are slow
* **Cache hit rates** — verify caching is effective
* **Score distributions** — detect relevance drift
* **Query patterns** — understand what users search for
See [Analytics & Performance](/docs/operations/analytics-overview) for the analytics endpoints to build dashboards or trigger alerts on degradation.
## The Feedback Loop
```
Search → Results → User interacts → Signal captured → Fusion learns → Better results
```
1. Users search via retrievers
2. Interaction signals capture what they engage with
3. Learned fusion adjusts stage weights automatically
4. Annotations provide explicit ground truth for edge cases
5. Evaluations measure improvement quantitatively
6. The cycle repeats — retrieval improves with usage
# Operate
Source: https://docs.mixpeek.com/docs/platform/operations
Run Mixpeek in production — security, webhooks, manifests, and infrastructure
<Frame>
<img alt="Operations: authentication with API keys and namespaces, document ACL for row-level security, webhooks for event subscriptions, manifests for config-as-code" />
</Frame>
## Authentication
Every request requires a Bearer token and namespace header:
```
Authorization: Bearer mxp_sk_...
X-Namespace: ns_production
```
API keys are scoped to an organization. Namespaces provide the authorization boundary — use separate namespaces for dev, staging, and production.
See [API Keys](/docs/operations/api-keys) to create scoped keys, set permissions, rotate and revoke them, and monitor per-key usage.
## Secrets & LLM Keys
Store third-party API keys in an encrypted secrets vault. Secrets are encrypted at rest using Fernet symmetric encryption and are never exposed in API responses.
### Manage Secrets
<CodeGroup>
```bash Create a Secret theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "openai_api_key",
"secret_value": "sk-proj-abc123..."
}'
```
```bash List Secrets theme={null}
# Returns secret names only — values are never exposed
curl "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_API_KEY"
```
```bash Update a Secret theme={null}
curl -X PUT "https://api.mixpeek.com/v1/organizations/secrets/openai_api_key" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "secret_value": "sk-proj-new-key..." }'
```
```bash Delete a Secret theme={null}
curl -X DELETE "https://api.mixpeek.com/v1/organizations/secrets/openai_api_key" \
-H "Authorization: Bearer YOUR_API_KEY"
```
</CodeGroup>
### Bring Your Own Key (BYOK)
Use your own LLM API keys instead of Mixpeek's default keys for cost control, higher rate limits, compliance, and independent key rotation.
**Supported providers:** OpenAI, Anthropic, Google
#### Apply a Key to All LLM Operations
Set a key as the **organization-wide default** and it automatically applies to every LLM operation — extractors, retrievers, clustering, taxonomy inference, and manifest generation — with no per-stage configuration.
<Steps>
<Step title="Store your key in the secrets vault">
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "openai_api_key",
"secret_value": "sk-proj-abc123..."
}'
```
```python Python theme={null}
from mixpeek import Mixpeek
mx = Mixpeek(api_key="YOUR_API_KEY")
mx.organizations.secrets.create(
secret_name="openai_api_key",
secret_value="sk-proj-abc123..."
)
```
</CodeGroup>
</Step>
<Step title="Set it as the org-wide default for that provider">
<CodeGroup>
```bash cURL theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/organizations" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"default_llm_credentials": {
"openai": "openai_api_key"
}
}'
```
```python Python theme={null}
mx.organizations.update(
default_llm_credentials={
"openai": "openai_api_key"
}
)
```
</CodeGroup>
</Step>
</Steps>
Configure defaults for multiple providers at once:
```json theme={null}
{
"default_llm_credentials": {
"openai": "my_openai_key",
"anthropic": "my_anthropic_key",
"google": "my_gemini_key"
}
}
```
<Tip>
In Studio, toggle **"Use as default LLM key"** when creating a secret to automatically apply it org-wide for the detected provider.
</Tip>
#### Override a Key on a Specific Stage
Set `api_key` directly on a retriever stage, extractor, or cluster config using `{{secrets.name}}` template syntax to override the org-wide default for that operation only:
```json theme={null}
{
"parameters": {
"provider": "openai",
"model": "gpt-4o-mini",
"api_key": "{{secrets.openai_api_key}}"
}
}
```
#### Credential Resolution Order
1. **Per-resource** `api_key` / `{{secrets.name}}` — explicit key on a specific stage or extractor
2. **Organization default** — set once via `default_llm_credentials`, applied everywhere
3. **Mixpeek platform keys** — used when no custom key is configured (usage charged to your Mixpeek account)
<Note>
Values in `default_llm_credentials` are **secret names**, not raw API keys. Keys are encrypted at rest, never returned in API responses, decrypted on-demand per LLM call, and isolated per organization with no cross-tenant leakage.
</Note>
## Document Access Control
Apply row-level security to documents with ACL rules. Filter results by user roles or attributes at query time without changing retriever logic. User-scoped keys (`usr_sk_`) transparently filter every read so each end-user sees only their documents — no app-side filtering.
For the full model (the `_acl` object, key types, public documents, sharing), see **[Document-Level ACL](/docs/operations/document-acl)**. For external policy engines, see [Permissions (OpenFGA)](/docs/platform/permissions). Broader auth/tenancy: [Security & Tenancy](/docs/operations/security).
## Webhooks
Subscribe to events like `batch.completed`, `document.created`, or `alert.triggered`:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/webhooks" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d '{
"webhook_name": "batch-notify",
"url": "https://example.com/webhook",
"events": ["batch.completed", "batch.failed"]
}'
```
[Webhook API →](/docs/api-reference/webhooks/create-webhook) · [Full webhooks guide →](/docs/operations/webhooks)
## Manifests
Declare your entire namespace configuration as code — buckets, collections, retrievers, taxonomies — and apply it in one request:
```bash theme={null}
# Export current state
curl "https://api.mixpeek.com/v1/manifest/export" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
# Apply a manifest
curl -X POST "https://api.mixpeek.com/v1/manifest/apply" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID" \
-H "Content-Type: application/json" \
-d @manifest.json
```
Use `POST /v1/manifest/diff` to preview changes before applying.
[Manifest API →](/docs/api-reference/manifest/apply-manifest)
## Lineage & Audit Traces
Every document links back to its source, and every retriever execution produces an auditable trace.
### Document Lineage
Track how a document was created — which object it came from, which collection processed it, and what features were extracted:
```bash theme={null}
curl "https://api.mixpeek.com/v1/documents/$DOCUMENT_ID/lineage" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
Returns the full decomposition tree: source object → batch → extracted documents. Use this to trace any search result back to the original file.
```bash theme={null}
# Get all documents derived from a source object
curl "https://api.mixpeek.com/v1/objects/$OBJECT_ID/derived" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
[Lineage API →](/docs/api-reference/document-lineage/get-document-lineage)
### Retriever Execution Traces
Every retriever execution captures a full trace — which stages ran, what scores were produced, which documents were dropped and why:
```bash theme={null}
curl "https://api.mixpeek.com/v1/retrievers/$RETRIEVER_ID/executions/$EXECUTION_ID" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $NAMESPACE_ID"
```
A trace includes:
* Which retriever version and config were used
* Each stage's input parameters, output set, scores, and latency
* Which feature URIs and collections were consulted
* Which filters matched and which documents were eliminated
* The final result set with per-document provenance
Traces are replayable — if a model version or taxonomy changes, you can compare results against historical executions.
[Execution API →](/docs/api-reference/retrievers/get-execution) · [Explain API →](/docs/api-reference/retrievers/explain-retriever-execution-plan)
## Environment Branching
Clone namespaces to create isolated environments for testing:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/$NS_ID/clone" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "namespace_name": "staging-branch" }'
```
Branched namespaces share no state with the source — safe for experimentation.
[Clone API →](/docs/api-reference/namespace-clone/clone-namespace)
# Permissions
Source: https://docs.mixpeek.com/docs/platform/permissions
Authorize retrieval per end-user — with Mixpeek's built-in document ACL or your own external authorization system (OpenFGA).
## Overview
Permissions control which documents an end-user can see when they search. Mixpeek
enforces them **server-side at retrieval time**, so a user's query only ever
returns documents they are authorized to access — your application never has to
add per-user filters, and there is no way for a client to bypass them.
There are two ways to drive permissions, and they share the same enforcement
path:
<CardGroup>
<Card title="Built-in document ACL" href="/platform/operations#document-access-control">
Mixpeek stores and enforces an `_acl` (owner / read / write / public) on each
document, keyed on user-scoped API keys. Best when Mixpeek is your source of
truth for who-can-see-what.
</Card>
<Card title="External authorization (OpenFGA)" href="#external-authorization-openfga">
Keep your existing permissions in your own [OpenFGA](https://openfga.dev)
deployment; Mixpeek queries it at retrieval time and filters results to match.
Best when an external system already owns authorization.
</Card>
</CardGroup>
Both are **opt-in** and **fail-closed**: a document with no resolvable grant is
excluded, never leaked. Namespaces that don't opt in behave exactly as before.
## Which should I use?
| | Built-in document ACL | External authorization (OpenFGA) |
| ------------------- | ---------------------------------------------- | -------------------------------------------------------------- |
| **Source of truth** | Mixpeek (`_acl` on each document) | Your OpenFGA store |
| **Grant model** | `read` list + `public` flag | Your full OpenFGA model (groups, roles, folder inheritance, …) |
| **Best for** | Apps where Mixpeek owns permissions | Apps where an external system already owns permissions |
| **Setup** | Create user-scoped keys; ACL set automatically | Run OpenFGA; opt the namespace in; map subjects |
| **Consistency** | Strong (enforced inline) | Strong (pull) or eventual (push) — you choose |
If you already manage authorization in OpenFGA (or a Zanzibar-style system), use
external authorization so you don't duplicate your permission model. Otherwise
the [built-in document ACL](/docs/platform/operations#document-access-control) is the simplest path.
## External authorization (OpenFGA)
When a namespace opts in, Mixpeek acts as a **relying party** on your OpenFGA: at
retrieval time it asks OpenFGA what the acting user can see and filters the
results accordingly. Your OpenFGA remains the single source of truth.
### How it works
1. **The acting subject** is the `principal_id` of the user-scoped API key
executing the query, mapped to an OpenFGA subject — `user:<principal_id>` by
default.
2. **Mixpeek queries OpenFGA** for that subject using one of the strategies below.
3. **Results are filtered** — unauthorized documents are removed before the
response is returned, for both ad-hoc and saved retrievers.
Your OpenFGA model must represent each Mixpeek document as an object whose id is
the Mixpeek `document_id`. By default Mixpeek checks the relation `viewer` on the
object type `document`:
```
document:doc_abc123 # viewer @ user:alice # alice can retrieve doc_abc123
document:doc_abc123 # viewer @ user:* # the document is public
```
Group, role, and parent-folder inheritance all work — OpenFGA resolves them
server-side; Mixpeek only relays the decision.
### Enforcement modes
Different strategies trade consistency against scale. Pick one with `mode`:
| `mode` | Strategy | Use when |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `auto` *(default)* | `ListObjects` → pre-filter when the accessible set is small (≤ `list_objects_max`), otherwise `BatchCheck` post-filter | You want sensible behavior at any size |
| `pull_list_objects` | Always `ListObjects` → `document_id` pre-filter | The accessible set per user is small (≤ \~1000) |
| `pull_batch_check` | Run the search, then `BatchCheck` the candidates and drop unauthorized | Users can access many documents |
| `push` | Sync OpenFGA grants into an indexed field and filter in-index | Lowest query latency; eventual consistency is acceptable |
<Note>
Pre-filtering via `ListObjects` only scales to a small accessible set
(\~1000 objects). For larger corpora, `BatchCheck` post-filtering **over-fetches
by `over_fetch_factor` (default 2×)** from the vector store so the page stays
full after unauthorized documents are dropped. `auto` switches between the two
for you.
</Note>
### Configuration
Opt a namespace in by setting `infrastructure.authorization`:
<CodeGroup>
```bash cURL theme={null}
curl -X PATCH https://api.mixpeek.com/v1/namespaces/ns_abc123 \
-H "Authorization: Bearer mxp_sk_your-org-key" \
-H "Content-Type: application/json" \
-d '{
"infrastructure": {
"authorization": {
"enabled": true,
"provider": "openfga",
"api_url": "https://openfga.your-company.com",
"store_id": "01J0XEXAMPLESTORE",
"relation": "viewer",
"mode": "auto"
}
}
}'
```
</CodeGroup>
<ParamField type="boolean">
Master switch. `false` (default) means the namespace is unchanged — no
enforcement and no calls to OpenFGA.
</ParamField>
<ParamField type="string">
Authorization backend. `openfga` is the supported provider.
</ParamField>
<ParamField type="string">
Base URL of your OpenFGA HTTP API, reachable from Mixpeek.
</ParamField>
<ParamField type="string">
The OpenFGA store id holding your relationship tuples.
</ParamField>
<ParamField type="string">
Authorization-model id. When omitted, the store's latest model is used.
</ParamField>
<ParamField type="string">
Name of an [organization secret](/docs/operations/security) holding the bearer
token for your OpenFGA API. The token is resolved from the encrypted secrets
vault at request time and is never stored in your namespace config. This is
the required way to supply a token for any hosted (`https://`) OpenFGA
deployment — store the token once via the organization secrets API, then
reference it here by name.
</ParamField>
<ParamField type="string">
Deprecated plaintext bearer token, accepted only for local development
(`http://` or `localhost`) where OpenFGA runs without auth. Rejected for a
hosted OpenFGA deployment — use `api_token_secret_ref` instead so the token
is never sent or stored in plaintext.
</ParamField>
<ParamField type="string">
The OpenFGA object type representing a Mixpeek document.
</ParamField>
<ParamField type="string">
The relation that grants read/retrieve access.
</ParamField>
<ParamField type="string">
Enforcement strategy: `auto`, `pull_list_objects`, `pull_batch_check`, or `push`.
</ParamField>
<ParamField type="integer">
Post-filter over-fetch multiplier (≥ 2 recommended) so a page stays full after
unauthorized documents are dropped.
</ParamField>
### Setup
<Steps>
<Step title="Run OpenFGA and write your tuples">
Stand up OpenFGA (or use your existing deployment) and write `viewer` tuples
whose object ids are Mixpeek `document_id`s, e.g.
`document:doc_abc123 # viewer @ user:alice`. Use `user:*` for public documents.
</Step>
<Step title="Create user-scoped keys per end-user">
Generate a `usr_sk_` key whose `principal_id` matches the OpenFGA user id
(e.g. `alice`). Create it with your org-scoped key via
[`POST /v1/organizations/users/{user_email}/api-keys`](/docs/api-reference/organization-api-keys/create-api-key).
There is no `key_type` field — setting `principal_id` on the request is what
makes the key user-scoped (the response key carries the `usr_sk_` prefix).
</Step>
<Step title="Opt the namespace in">
PATCH the namespace with the `infrastructure.authorization` block above.
</Step>
<Step title="Query normally">
Execute retrievers with each user's `usr_sk_` key — results are filtered to
what that user can see in OpenFGA. No query changes needed.
</Step>
</Steps>
### Keeping permissions current (push mode)
In `push` mode, Mixpeek subscribes to your OpenFGA changelog and projects grants
onto an indexed field on each document, then filters in-index for the lowest
query latency. Grant and revocation changes propagate within the sync interval
(eventually consistent). For strict point-in-time consistency, use `auto` or
`pull_batch_check`, which evaluate against OpenFGA live on every query.
## Behavior and guarantees
* **Fail-closed.** If OpenFGA is unreachable, queries return the safe subset
(fewer results), never unauthorized documents.
* **No caching leaks.** When authorization is active, the per-retriever result
cache is bypassed so one user's authorized page can never be served to another.
* **Both retrieval paths.** Ad-hoc and saved retrievers enforce identically.
* **Opt-out is unchanged.** A namespace with `enabled: false` (or no
`authorization` block) behaves exactly as before.
<Warning>
Permissions are enforced at the Mixpeek API layer. If you have direct access to
the underlying vector store or database, filters are not applied. Always route
end-user traffic through the Mixpeek API.
</Warning>
## Related
* [Document Access Control](/docs/platform/operations#document-access-control) — Mixpeek's built-in `_acl` permissions
* [Operate](/docs/platform/operations) — authentication, secrets, and namespace isolation
* [Filters](/docs/retrieval/filters) — manual query filters (permission filters are injected on top)
* [OpenFGA documentation](https://openfga.dev/docs) — modeling, tuples, and the Check / ListObjects / BatchCheck APIs
# Triggers
Source: https://docs.mixpeek.com/docs/platform/triggers
Schedule clustering, taxonomy enrichment, and batch reruns on cron, intervals, or events
Triggers run platform actions automatically — re-cluster a collection nightly, re-enrich documents as new data lands, or rerun a batch on a schedule. Instead of calling an endpoint by hand, you define *what* to run and *when*, and Mixpeek executes it.
## Anatomy of a trigger
Every trigger combines an **action** (what to run) with a **schedule** (when to run it):
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/triggers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"action_type": "cluster",
"action_config": { "cluster_id": "clust_abc123" },
"trigger_type": "cron",
"schedule_config": { "cron_expression": "0 2 * * *", "timezone": "UTC" },
"description": "Daily clustering at 2am"
}'
```
| Field | Description |
| ----------------- | ----------------------------------------------------------------------------------- |
| `action_type` | What to run: `cluster`, `taxonomy_enrichment`, `batch_rerun`, `collection_trigger`. |
| `action_config` | Action-specific config (e.g. `{ "cluster_id": "..." }`). |
| `trigger_type` | When to run: `cron`, `interval`, `event`, `conditional`. |
| `schedule_config` | Schedule-specific config (cron expression, interval seconds, etc.). |
| `description` | Human-readable label. |
| `status` | `active` (default) or `paused`. |
## Schedule types
<Tabs>
<Tab title="Cron">
Run on a cron expression in a given timezone — best for fixed times ("every night at 2am").
```json theme={null}
{
"trigger_type": "cron",
"schedule_config": { "cron_expression": "0 2 * * *", "timezone": "UTC" }
}
```
The expression is standard 5-field cron (`minute hour day-of-month month day-of-week`).
</Tab>
<Tab title="Interval">
Run every N seconds — best for "keep it fresh" loops. Set `start_immediately` to run once on creation.
```json theme={null}
{
"trigger_type": "interval",
"schedule_config": { "interval_seconds": 3600, "start_immediately": false }
}
```
</Tab>
<Tab title="Event / Conditional">
`event` triggers fire in response to platform events (e.g. new documents in a collection); `conditional` triggers fire when a condition is met. Provide the event/condition details in `schedule_config`. Use cron or interval for time-based schedules.
</Tab>
</Tabs>
## Actions
### Cluster
Re-run a clustering definition on a schedule so groupings stay current as data grows.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/triggers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"action_type": "cluster",
"action_config": {
"cluster_id": "clust_abc123",
"filters": { "status": "active" },
"output_collection_ids": ["col_daily_clusters"]
},
"trigger_type": "cron",
"schedule_config": { "cron_expression": "0 2 * * *", "timezone": "UTC" },
"description": "Daily clustering of active items"
}'
```
### Taxonomy enrichment
Re-classify a collection against a taxonomy. Use `incremental: true` to enrich only new documents — far cheaper than re-running the whole collection.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/triggers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"action_type": "taxonomy_enrichment",
"action_config": {
"collection_id": "col_inventory",
"taxonomy_id": "tax_products",
"incremental": true
},
"trigger_type": "interval",
"schedule_config": { "interval_seconds": 3600 },
"description": "Hourly incremental enrichment (only new docs)"
}'
```
Enrich into a separate target collection on a nightly cron:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/triggers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"action_type": "taxonomy_enrichment",
"action_config": {
"collection_id": "col_inventory",
"taxonomy_id": "tax_products",
"target_collection_id": "col_enriched",
"incremental": true,
"batch_size": 500,
"parallelism": 8
},
"trigger_type": "cron",
"schedule_config": { "cron_expression": "0 3 * * *", "timezone": "UTC" },
"description": "Nightly incremental enrichment to target collection"
}'
```
<Tip>
`incremental: true` only processes documents added since the last run, so it won't re-pay extraction/enrichment cost on documents already processed. Prefer it for recurring enrichment.
</Tip>
### Batch rerun & collection trigger
`batch_rerun` re-runs processing for a collection's documents; `collection_trigger` fires a collection's configured processing. Both take a `collection_id` in `action_config`.
## Manage triggers
| Operation | Endpoint |
| -------------- | ---------------------------------------------------------- |
| List | `POST /v1/triggers/list` |
| Get | `GET /v1/triggers/{trigger_id}` |
| Update | `PATCH /v1/triggers/{trigger_id}` |
| Run now | `POST /v1/triggers/{trigger_id}/execute` |
| Pause / resume | `POST /v1/triggers/{trigger_id}/pause` · `POST .../resume` |
| Run history | `GET /v1/triggers/{trigger_id}/history` |
| Delete | `DELETE /v1/triggers/{trigger_id}` |
Run a trigger on demand (outside its schedule) to test it:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/triggers/{trigger_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
Check recent runs and their outcomes:
```bash theme={null}
curl -sS "$MP_API_URL/v1/triggers/{trigger_id}/history" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
## Related
* [Clusters](/docs/enrichment/clusters) — define the clustering a `cluster` trigger reruns
* [Taxonomies](/docs/enrichment/taxonomies) — define the taxonomy a `taxonomy_enrichment` trigger applies
* [Alerts](/docs/enrichment/alerts) — get notified when content matches a condition
* [Storage syncs](/docs/tutorials/ingest-video-from-s3) — continuously ingest new files from object storage
# Tasks
Source: https://docs.mixpeek.com/docs/processing/tasks
Track asynchronous jobs across Mixpeek
Tasks provide a uniform way to monitor long-running operations (batch processing, clustering, taxonomy materialization, namespace migrations, etc.). Every task exposes a status from the shared `TaskStatusEnum`.
## TaskStatusEnum
```
PENDING → PROCESSING → COMPLETED
↘ ↘
FAILED COMPLETED_WITH_ERRORS
```
Additional values include `IN_PROGRESS`, `CANCELED`, `SKIPPED`, `UNKNOWN`, `DRAFT`, `ACTIVE`, `ARCHIVED`, and `SUSPENDED`. All async resources in Mixpeek adopt this enum, so your polling logic works everywhere.
<Warning>
**Terminal statuses are `COMPLETED`, `COMPLETED_WITH_ERRORS`, `FAILED`, and `CANCELED`.** A poller that waits only for `COMPLETED` will hang forever on a batch that finished with some failed items (`COMPLETED_WITH_ERRORS` — partial success). Always stop on any terminal status.
</Warning>
## Anatomy of a Task
```json theme={null}
{
"task_id": "tsk_processing_123",
"task_type": "api_buckets_batches_process",
"status": "PROCESSING",
"inputs": ["batch_xyz789"],
"outputs": null,
"additional_data": {
"batch_id": "batch_xyz789",
"bucket_id": "bkt_products",
"job_id": "ray_job_123"
},
"error_message": null
}
```
* Cached in Redis for \~24 hours (fast lookup).
* Persisted in MongoDB for historical auditing.
* `additional_data` stores resource-specific details (e.g., Ray job IDs).
## Polling Strategy
<Steps>
<Step title="Poll the task">
Query `/v1/tasks/{task_id}` with exponential backoff (start at 1s, cap at 30s).
</Step>
<Step title="Handle 404 gracefully">
After Redis TTL expires you may receive `404`; fall back to the underlying resource (batch, cluster, etc.).
</Step>
<Step title="Switch to resource polling">
Use `/v1/buckets/{bucket_id}/batches/{batch_id}`, `/v1/clusters/{cluster_id}`, etc., for long-running operations.
</Step>
</Steps>
Example hybrid poller:
```python theme={null}
while True:
try:
task = get_task(task_id)
except NotFound:
task = get_batch(bucket_id, batch_id)
if task.status in ("COMPLETED", "COMPLETED_WITH_ERRORS"):
break # both are terminal; COMPLETED_WITH_ERRORS = partial success
if task.status in ("FAILED", "CANCELED"):
raise RuntimeError(task.error_message)
time.sleep(delay)
delay = min(delay * 1.5, 30)
```
## Webhooks & Notifications
* Engine emits webhook events (e.g., `collection.documents.written`) when tasks complete relevant work.
* Celery Beat dispatches those events to invalidate caches, update schemas, and notify external systems.
* Prefer webhooks for near-real-time updates instead of aggressive polling.
## Managing Tasks
* `GET /v1/tasks/{task_id}` – fetch the latest status.
* `POST /v1/tasks/list` – filter by type, status, namespace, or creation time.
* `POST /v1/tasks/{task_id}/kill` – request cancellation (supported for batches and clustering jobs using Celery’s `AbortableAsyncResult`).
## Best Practices
1. **Store task IDs** returned by submit endpoints.
2. **Use exponential backoff** to avoid hammering the API.
3. **Respect terminal states** (`COMPLETED`, `COMPLETED_WITH_ERRORS`, `FAILED`, `CANCELED`) and surface errors to operators.
4. **Use webhooks** for side-effects like cache invalidation or notifications.
5. **Instrument monitoring**—task history in MongoDB plus webhook logs provide a full audit trail.
Tasks keep the asynchronous parts of Mixpeek manageable—treat them as durable receipts for every long-running job.
# Analytics
Source: https://docs.mixpeek.com/docs/relevance/analytics
Monitor retriever performance, identify slow queries, and get AI-powered tuning recommendations
Analytics gives you visibility into how your retrievers perform in production. Track latency, stage-level bottlenecks, cache efficiency, and get automated recommendations for improvement.
## Endpoints Overview
| Endpoint | Method | Returns |
| ---------------------------------------------- | ------ | ----------------------------------------------------------------- |
| `/analytics/retrievers/{id}/performance` | GET | Latency percentiles (P50/P95/P99), query counts, trends |
| `/analytics/retrievers/{id}/stages` | GET | Per-stage execution times and document flow |
| `/analytics/retrievers/{id}/signals` | GET | Operational signals (cache hits, rerank scores, filter reduction) |
| `/analytics/retrievers/{id}/cache-performance` | GET | Cache hit/miss rates, latency savings |
| `/analytics/retrievers/{id}/slow-queries` | GET | Slowest queries with stage-level breakdown |
| `/analytics/retrievers/{id}/analyze-tuning` | POST | AI-powered parameter tuning recommendations |
## Performance Metrics
Get latency percentiles and query volume over time:
<CodeGroup>
```bash cURL theme={null}
curl "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/performance?group_by=hour&start_date=2025-01-01T00:00:00Z" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
performance = client.analytics.retrievers.performance(
retriever_id="ret_abc123",
group_by="hour",
start_date="2025-01-01T00:00:00Z"
)
```
</CodeGroup>
**Query parameters:**
| Parameter | Type | Default | Description |
| ------------ | -------- | -------- | ------------------------------------ |
| `start_date` | datetime | — | Start of time range (UTC) |
| `end_date` | datetime | — | End of time range (UTC) |
| `group_by` | string | `"hour"` | Time grouping: `hour`, `day`, `week` |
**Response includes:** P50, P95, and P99 latency, query counts, result counts, and trends over the time range.
## Stage Breakdown
Understand which stages consume the most time:
<CodeGroup>
```bash cURL theme={null}
curl "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/stages?hours=24" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python theme={null}
stages = client.analytics.retrievers.stages(
retriever_id="ret_abc123",
hours=24
)
```
</CodeGroup>
**Query parameters:**
| Parameter | Type | Default | Description |
| --------- | ------- | ------- | ------------------------ |
| `hours` | integer | `24` | Hours of history (1–720) |
**Response includes:** Per-stage execution time, document count entering/exiting each stage, and stage-level latency distributions. Use this to identify bottlenecks — a rerank stage processing 500 documents is slower than one processing 50.
## Slow Queries
Find the queries that take the longest to execute:
<CodeGroup>
```bash cURL theme={null}
curl "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/slow-queries?limit=10&hours=24" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python theme={null}
slow = client.analytics.retrievers.slow_queries(
retriever_id="ret_abc123",
limit=10,
hours=24
)
```
</CodeGroup>
**Query parameters:**
| Parameter | Type | Default | Description |
| --------- | ------- | ------- | -------------------------------- |
| `limit` | integer | `10` | Number of slow queries to return |
| `hours` | integer | `24` | Hours of history (1–720) |
**Response includes:** Query text, total execution time, result count, and stage-by-stage breakdown for each slow query. Use this to find pathological queries that need optimization.
## Cache Performance
Monitor how effectively caching reduces latency:
<CodeGroup>
```bash cURL theme={null}
curl "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/cache-performance?hours=24" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python theme={null}
cache = client.analytics.retrievers.cache_performance(
retriever_id="ret_abc123",
hours=24
)
```
</CodeGroup>
**Response includes:** Hit/miss rates, average latency for cache hits vs full execution, and hourly trends. A low hit rate may indicate your queries are too diverse for caching, or that cache TTL needs adjustment.
## Retriever Signals
Get raw operational signals for debugging:
<CodeGroup>
```bash cURL theme={null}
curl "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/signals?signal_type=rerank_scores&limit=100&hours=24" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python theme={null}
signals = client.analytics.retrievers.signals(
retriever_id="ret_abc123",
signal_type="rerank_scores",
limit=100,
hours=24
)
```
</CodeGroup>
**Signal types:**
| Signal | Description |
| ------------------- | --------------------------------------------------- |
| `cache_hit` | Query served from cache |
| `cache_miss` | Cache miss, full execution |
| `rerank_scores` | Score distribution from rerank stage |
| `filter_reduction` | How much the filter stage reduced the candidate set |
| `expansion_results` | Query expansion output |
## AI-Powered Tuning
Get automated recommendations for improving your retriever:
<CodeGroup>
```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/analyze-tuning?days=7" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python theme={null}
tuning = client.analytics.retrievers.analyze_tuning(
retriever_id="ret_abc123",
days=7
)
```
</CodeGroup>
**Query parameters:**
| Parameter | Type | Default | Description |
| --------- | ------- | ------- | --------------------------------- |
| `days` | integer | `7` | Days of history to analyze (1–90) |
**Response includes:** Parameter suggestions (e.g., "reduce top\_k from 500 to 200"), cache optimization tips, and performance improvement estimates based on observed patterns.
## Identifying Relevance Issues
Use analytics to spot relevance problems:
| Symptom | Analytics Signal | Likely Cause | Action |
| ---------------------------- | ----------------------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------- |
| High latency, normal results | Stage breakdown shows slow rerank | Too many candidates entering rerank | Reduce `top_k` or add a limit stage before rerank |
| Low click-through rate | Interaction signals show high skip rate | Poor ranking or irrelevant results | Check fusion strategy, consider learned fusion |
| Cache hit rate dropping | Cache performance shows increasing misses | Query diversity increasing or TTL too short | Adjust cache strategy, review in [caching best practices](/docs/best-practices/caching-strategies) |
| Inconsistent latency | Slow queries show specific patterns | Certain query types trigger expensive paths | Add pre-filters or query-specific optimization |
## Monitoring Cadence
| Frequency | Check | Tools |
| ----------- | ----------------------------------------------------- | -------------------------------------------------------- |
| **Daily** | Slow queries, P95 latency | `/slow-queries`, `/performance` |
| **Weekly** | Stage breakdown, cache efficiency, interaction trends | `/stages`, `/cache-performance`, `/signals` |
| **Monthly** | AI tuning analysis, full evaluation run | `/analyze-tuning`, [Evaluations](/docs/retrieval/evaluations) |
## Related
* [Evaluations](/docs/retrieval/evaluations) — offline quality measurement
* [Benchmarks](/docs/retrieval/benchmarks) — historical session replay
* [Caching Strategies](/docs/best-practices/caching-strategies) — optimizing cache performance
* [Interaction Signals](/docs/retrieval/interactions) — capturing user behavior data
# Annotations
Source: https://docs.mixpeek.com/docs/relevance/annotations
Record human decisions on documents for review workflows, compliance, and model improvement
Annotations capture explicit human judgments on documents — approve, reject, defer, or any domain-specific label. Unlike [interaction signals](/docs/retrieval/interactions) which track implicit behavior (clicks, views, dwell time), annotations record deliberate decisions with optional confidence scores, reasoning, and structured payloads. They are the foundation for human-in-the-loop workflows where retriever results need expert review before action.
## When to Use Annotations
Annotations solve the problem of turning retriever output into verified decisions. Any workflow where a person reviews documents and records a judgment benefits from annotations:
| Use Case | Labels | Payload Example |
| ------------------------- | -------------------------------------- | --------------------------------------------------------- |
| Medical coding review | `approved`, `rejected`, `deferred` | `{"codes_approved": ["E11.40"], "raf_impact": 0.302}` |
| Brand infringement triage | `infringement`, `safe`, `needs_review` | `{"confidence_model": 0.91, "match_type": "logo"}` |
| Duplicate detection | `confirmed_dupe`, `false_positive` | `{"canonical_id": "doc_abc", "similarity": 0.97}` |
| Content moderation | `approved`, `flagged`, `removed` | `{"policy_violation": "copyright", "severity": "high"}` |
| Document classification | `correct`, `incorrect`, `ambiguous` | `{"predicted_class": "invoice", "true_class": "receipt"}` |
Annotations are domain-agnostic — labels are free-form strings, and the `payload` field accepts any structured JSON your workflow needs.
## Annotation Lifecycle
```
Retriever executes → Results returned → Human reviews → Annotation recorded → Audit trail preserved
```
Each annotation links back to a document and optionally to the retriever execution that surfaced it:
* **`document_id`** and **`collection_id`** — what was reviewed
* **`retriever_id`**, **`execution_id`**, **`stage_name`** — how the document was found (provenance)
* **`label`**, **`confidence`**, **`reasoning`** — the human decision
* **`payload`** — structured data specific to the workflow
* **`actor_id`** and **`actor_type`** — who made the decision (user, API key, or system)
All mutations emit [webhooks](/docs/operations/webhooks) (`annotation.created`, `annotation.updated`, `annotation.deleted`) and log to the [audit trail](/docs/api-reference/organization-audit/list-audit-logs).
## Create an Annotation
Record a decision after reviewing a document:
<CodeGroup>
```bash cURL theme={null}
curl -sS -X POST "$MP_API_URL/v1/annotations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"document_id": "doc_2e7650fa254b",
"collection_id": "col_clinical_notes",
"label": "approved",
"confidence": 0.95,
"reasoning": "Note clearly documents peripheral neuropathy with supporting lab values.",
"payload": {
"codes_approved": ["E11.40", "E11.65"],
"raf_impact": 0.420,
"annual_revenue": 2522
},
"retriever_id": "ret_hcc_review",
"execution_id": "exec_abc123"
}'
```
```python Python theme={null}
from mixpeek import Mixpeek
mp = Mixpeek(api_key="API_KEY")
annotation = mp.annotations.create(
document_id="doc_2e7650fa254b",
collection_id="col_clinical_notes",
label="approved",
confidence=0.95,
reasoning="Note clearly documents peripheral neuropathy with supporting lab values.",
payload={
"codes_approved": ["E11.40", "E11.65"],
"raf_impact": 0.420,
"annual_revenue": 2522,
},
retriever_id="ret_hcc_review",
execution_id="exec_abc123",
namespace="ns_vitae",
)
```
```javascript JavaScript theme={null}
import Mixpeek from "mixpeek";
const mp = new Mixpeek({ apiKey: "API_KEY" });
const annotation = await mp.annotations.create({
documentId: "doc_2e7650fa254b",
collectionId: "col_clinical_notes",
label: "approved",
confidence: 0.95,
reasoning:
"Note clearly documents peripheral neuropathy with supporting lab values.",
payload: {
codes_approved: ["E11.40", "E11.65"],
raf_impact: 0.42,
annual_revenue: 2522,
},
retrieverId: "ret_hcc_review",
executionId: "exec_abc123",
namespace: "ns_vitae",
});
```
</CodeGroup>
## Query Annotations
List annotations with filters to build review queues or dashboards:
<CodeGroup>
```bash cURL theme={null}
# All rejected annotations for a collection
curl -sS -X POST "$MP_API_URL/v1/annotations/list" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_id": "col_clinical_notes",
"label": "rejected"
}'
# All annotations on a specific document
curl -sS -X POST "$MP_API_URL/v1/annotations/list" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"document_id": "doc_2e7650fa254b"
}'
```
```python Python theme={null}
# All rejected annotations
rejected = mp.annotations.list(
collection_id="col_clinical_notes",
label="rejected",
namespace="ns_vitae",
)
# All annotations on a specific document
doc_annotations = mp.annotations.list(
document_id="doc_2e7650fa254b",
namespace="ns_vitae",
)
```
```javascript JavaScript theme={null}
// All rejected annotations
const rejected = await mp.annotations.list({
collectionId: "col_clinical_notes",
label: "rejected",
namespace: "ns_vitae",
});
// All annotations on a specific document
const docAnnotations = await mp.annotations.list({
documentId: "doc_2e7650fa254b",
namespace: "ns_vitae",
});
```
</CodeGroup>
Available filters: `document_id`, `collection_id`, `label`, `actor_id`, `retriever_id`. All filters are optional and can be combined.
## Aggregate Stats
Get label distribution counts for dashboards and progress tracking:
<CodeGroup>
```bash cURL theme={null}
# Stats across all annotations
curl -sS "$MP_API_URL/v1/annotations/stats" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
# Stats for a specific collection
curl -sS "$MP_API_URL/v1/annotations/stats?collection_id=col_clinical_notes" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python theme={null}
stats = mp.annotations.stats(namespace="ns_vitae")
# {"total": 142, "by_label": {"approved": 89, "rejected": 31, "deferred": 22}}
```
```javascript JavaScript theme={null}
const stats = await mp.annotations.stats({ namespace: "ns_vitae" });
// {total: 142, byLabel: {approved: 89, rejected: 31, deferred: 22}}
```
</CodeGroup>
## Update a Decision
When a review is revisited — for example, a deferred case gets a clinical consult and can now be approved:
<CodeGroup>
```bash cURL theme={null}
curl -sS -X PATCH "$MP_API_URL/v1/annotations/ann_3cefcdaf7536a19a" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"label": "approved",
"confidence": 0.88,
"reasoning": "Clinical review completed — peripheral neuropathy confirmed."
}'
```
```python Python theme={null}
updated = mp.annotations.update(
annotation_id="ann_3cefcdaf7536a19a",
label="approved",
confidence=0.88,
reasoning="Clinical review completed — peripheral neuropathy confirmed.",
namespace="ns_vitae",
)
```
```javascript JavaScript theme={null}
const updated = await mp.annotations.update({
annotationId: "ann_3cefcdaf7536a19a",
label: "approved",
confidence: 0.88,
reasoning: "Clinical review completed — peripheral neuropathy confirmed.",
namespace: "ns_vitae",
});
```
</CodeGroup>
The audit trail records both the original and updated values, preserving the full decision history.
## Bulk Operations
Process up to 1000 creates, updates, and deletes in a single call. Each operation is independent — a failure in one does not roll back the others.
<CodeGroup>
```bash cURL theme={null}
curl -sS -X POST "$MP_API_URL/v1/annotations/bulk" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"create": [
{"document_id": "doc_001", "collection_id": "col_notes", "label": "approved", "confidence": 0.95},
{"document_id": "doc_002", "collection_id": "col_notes", "label": "approved", "confidence": 0.91},
{"document_id": "doc_003", "collection_id": "col_notes", "label": "rejected", "reasoning": "Insufficient documentation"}
],
"update": [
{"annotation_id": "ann_abc123", "label": "approved", "confidence": 0.88}
],
"delete": ["ann_def456", "ann_ghi789"]
}'
```
```python Python theme={null}
result = mp.annotations.bulk(
create=[
{"document_id": "doc_001", "collection_id": "col_notes", "label": "approved", "confidence": 0.95},
{"document_id": "doc_002", "collection_id": "col_notes", "label": "approved", "confidence": 0.91},
{"document_id": "doc_003", "collection_id": "col_notes", "label": "rejected", "reasoning": "Insufficient documentation"},
],
update=[
{"annotation_id": "ann_abc123", "label": "approved", "confidence": 0.88},
],
delete=["ann_def456", "ann_ghi789"],
namespace="ns_vitae",
)
# result.created_count=3, result.updated_count=1, result.deleted_count=2
```
```javascript JavaScript theme={null}
const result = await mp.annotations.bulk({
create: [
{ documentId: "doc_001", collectionId: "col_notes", label: "approved", confidence: 0.95 },
{ documentId: "doc_002", collectionId: "col_notes", label: "approved", confidence: 0.91 },
{ documentId: "doc_003", collectionId: "col_notes", label: "rejected", reasoning: "Insufficient documentation" },
],
update: [
{ annotationId: "ann_abc123", label: "approved", confidence: 0.88 },
],
delete: ["ann_def456", "ann_ghi789"],
namespace: "ns_vitae",
});
// result.createdCount=3, result.updatedCount=1, result.deletedCount=2
```
</CodeGroup>
The response includes per-operation results so you can identify and retry individual failures.
## Example: Medical Coding Review Workflow
A healthcare organization uses Mixpeek to surface HCC suspect conditions from clinical notes. Coders review each result and record their decision:
1. **Retriever** runs `agent_search` across clinical notes, returning suspect HCC conditions with supporting evidence.
2. **Review queue** — the application calls `POST /v1/annotations/list?label=deferred` to show unresolved cases.
3. **Coder annotates** — for each document, the coder selects a label and the app calls `POST /v1/annotations` with the decision, ICD-10 codes, and RAF impact.
4. **Dashboard** — `GET /v1/annotations/stats?collection_id=col_notes` powers a progress bar showing approved/rejected/deferred counts.
5. **Audit** — compliance officers query annotations by `actor_id` to review individual coder decisions. The `reasoning` field provides the justification trail required for CMS audits.
<Note>
Annotations are stored independently from documents — they don't modify the underlying document data. This separation ensures that the original clinical record remains untouched while the review layer captures all human decisions.
</Note>
## Best Practices
* **Use consistent labels** within a workflow. Pick a label vocabulary (e.g., `approved`, `rejected`, `deferred`) and stick with it — the stats endpoint groups by exact string match.
* **Include reasoning** for audit-sensitive workflows. The `reasoning` field is indexed and retrievable, making it valuable for compliance reviews and dispute resolution.
* **Link provenance** when annotating retriever results. Setting `retriever_id`, `execution_id`, and `stage_name` lets you trace exactly how the document was surfaced, which is critical for evaluating retriever quality.
* **Use payload for structured data** rather than encoding it in the label. Labels should be human-readable categories; domain-specific fields (codes, scores, amounts) belong in `payload`.
* **Listen to webhooks** for real-time updates. Subscribe to `annotation.created` and `annotation.updated` events to trigger downstream workflows (e.g., auto-submit approved records, escalate rejected ones).
## References
* [Create Annotation](/docs/api-reference/annotations/create-annotation)
* [List Annotations](/docs/api-reference/annotations/list-annotations)
* [Annotation Stats](/docs/api-reference/annotations/annotation-stats)
* [Update Annotation](/docs/api-reference/annotations/update-annotation)
* [Delete Annotation](/docs/api-reference/annotations/delete-annotation)
* [Bulk Annotations](/docs/api-reference/annotations/bulk-annotations)
* [Interaction Signals](/docs/retrieval/interactions) — implicit behavioral signals (complementary to annotations)
* [Webhooks](/docs/operations/webhooks) — subscribe to annotation lifecycle events
# Architecture
Source: https://docs.mixpeek.com/docs/relevance/architecture
How Mixpeek's two-tower design decouples ingestion from retrieval and learns from usage
Mixpeek follows a **two-tower architecture** — a well-known pattern in recommendation systems adapted for multimodal search.
<Frame>
<img alt="Two towers: document tower writes N representation spaces at ingest, query tower reads and fuses them at search time, interaction signals close the feedback loop" />
</Frame>
## Document Tower (Ingestion)
Source files enter through a [bucket](/docs/platform/data-model#buckets), trigger one or more [collections](/docs/platform/data-model#collections), and pass through the Ray engine for feature extraction. Each collection produces a different representation — text embeddings, multimodal embeddings, metadata, taxonomy labels — all stored as named vectors on a single point in [MVS](/docs/vector-store/overview), the Mixpeek Vector Store.
Documents are encoded **once at ingest time**. Adding a new extractor or updating a taxonomy triggers a re-process on the bucket — documents get new representations without changing the ingestion path.
## Query Tower (Retrieval)
A query arrives, gets encoded, and passes through a [multi-stage retriever](/docs/retrieval/retrievers). The key stage is **feature search**, which runs a separate vector query per embedding space and fuses the results.
The fusion strategy determines how per-feature scores combine into a final ranking:
| Strategy | Behavior |
| ---------- | ----------------------------------------------------------------- |
| `rrf` | Rank-based, no tuning needed |
| `weighted` | Manual weights you set |
| `learned` | Weights sampled from Beta distributions, updated by user behavior |
See [Fusion Strategies](/docs/relevance/fusion-strategies) for the full comparison.
## Closing the Loop
With [learned fusion](/docs/relevance/learned-fusion), the two towers aren't static — they're connected by a feedback loop:
1. **Results** are shown to users
2. **Interactions** (clicks, purchases, skips) are captured and stored in ClickHouse
3. **Thompson Sampling** aggregates interactions into Beta(α, β) distributions per feature — α counts positive signals, β counts non-engagement
4. **Sampled weights** are drawn from those distributions on each query, naturally balancing exploration and exploitation
5. Weights converge toward the optimal blend as interactions accumulate
The system handles cold start through [hierarchical fallback](/docs/relevance/learned-fusion#hierarchical-fallback): personal weights → demographic segment → global → uniform prior. With zero interactions, learned fusion behaves identically to RRF.
## What Makes This Different
Standard two-tower systems learn a single embedding space. Mixpeek's document tower fans out into **N representation spaces** (visual, audio, text, multimodal, metadata), and the query tower traverses them in sequence through multi-stage retrieval. Learning happens at the **fusion layer** — which spaces to weight — not inside the embeddings themselves.
This means you can add a new extractor, re-process your data, and the bandit will automatically discover whether the new feature improves results — without retraining any model.
## Related
* [Feedback Loop Tutorial](/docs/tutorials/feedback-loop) — step-by-step setup guide
* [Learned Fusion](/docs/relevance/learned-fusion) — Thompson Sampling algorithm details
* [Interaction Signals](/docs/retrieval/interactions) — which signals to capture and when
* [Fusion Strategies](/docs/relevance/fusion-strategies) — all 5 strategies compared
# Fusion Strategies
Source: https://docs.mixpeek.com/docs/relevance/fusion-strategies
How multiple search results are combined into a single ranked list using RRF, DBSF, Weighted, Max, or Learned fusion
When a [feature search](/docs/retrieval/stages/feature-search) stage queries multiple embedding indexes (e.g., text + image), it produces separate ranked lists that need to be merged. Fusion strategies determine how those lists become one.
<Frame>
<img alt="Comparison of RRF, Weighted, and Learned fusion strategies" />
</Frame>
## Strategy Reference
| Strategy | Formula | Configuration | Best For |
| ---------- | -------------------------------- | ------------------- | --------------------------------------------- |
| `rrf` | `1 / (k + rank)` | None (k=60 default) | General purpose, no tuning needed |
| `dbsf` | Distribution-based normalization | None | Different score distributions across features |
| `weighted` | `w₁·score₁ + w₂·score₂` | `weight` per search | Known feature importance |
| `max` | `max(score₁, score₂)` | None | Any single match is sufficient |
| `learned` | Thompson Sampling | Interaction data | Personalized, adaptive weights |
## Reciprocal Rank Fusion (RRF)
The default strategy. RRF ignores raw similarity scores and uses only rank position. This makes it robust when different features produce scores on different scales.
**Formula:**
```
score(doc) = Σ 1 / (k + rank_i(doc))
```
Where `k = 60` (constant that prevents top-ranked items from dominating) and `rank_i` is the document's position in the i-th feature's result list.
**Why it works:** A document ranked #1 by text search and #3 by image search gets a higher fused score than a document ranked #2 by both. The rank-based approach means you don't need to calibrate score ranges across features.
<Warning>
**RRF scores are small on purpose — read order, not magnitude.** Because the score
is `1 / (k + rank)` with `k = 60`, the best possible result scores about
`1 / (60 + 1) ≈ 0.0164` per search (a little higher when several searches rank the
same document). A top score of `~0.017` is **expected and healthy** — it is a fused
rank position, **not** a similarity or a quality percentage, so a `0.0167` top hit
does not mean the search matched poorly. Rank documents by their **relative** order.
If you need an absolute similarity value (e.g. `0.0`–`1.0`), use a single search
without fusion, or a `weighted`/`max` strategy that preserves raw scores.
</Warning>
<CodeGroup>
```json Configuration theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"fusion": "rrf",
"final_top_k": 25
}
}
}
```
</CodeGroup>
<Tip>
RRF is the best default. Use it unless you have a specific reason to choose another strategy.
</Tip>
## Distribution-Based Score Fusion (DBSF)
DBSF normalizes scores from each feature into a common distribution before combining them. This handles cases where one feature produces scores in \[0.8, 0.99] and another in \[0.1, 0.6].
**How it works:**
1. For each feature, compute the mean (μ) and standard deviation (σ) of scores
2. Normalize each score: `normalized = (score - μ) / σ`
3. Sum normalized scores across features
DBSF is useful when features have different score distributions, but the raw scores themselves carry meaningful information (unlike RRF which ignores scores entirely).
<CodeGroup>
```json Configuration theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"fusion": "dbsf",
"final_top_k": 25
}
}
}
```
</CodeGroup>
## Weighted Fusion
You manually assign a weight to each search feature. Scores are multiplied by their weight and summed. Use this when you know from domain expertise that one feature is more important than another.
**Formula:**
```
score(doc) = w₁ · score₁(doc) + w₂ · score₂(doc) + ...
```
Weights don't need to sum to 1 — they're relative importance indicators.
<CodeGroup>
```json Configuration theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100,
"weight": 0.7
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.image_url}}",
"top_k": 100,
"weight": 0.3
}
],
"fusion": "weighted",
"final_top_k": 25
}
}
}
```
</CodeGroup>
<Warning>
Weighted fusion is sensitive to score scale differences. If text scores are in \[0.8, 1.0] and image scores are in \[0.1, 0.5], a 50/50 weight split will still favor text. Consider DBSF or RRF if score ranges differ significantly.
</Warning>
## Max Fusion
Takes the maximum score across all features for each document. A document only needs to be a strong match on **one** feature to rank highly.
**Formula:**
```
score(doc) = max(score₁(doc), score₂(doc), ...)
```
Use this when any single strong match is sufficient — for example, a product that matches either the text description or the visual similarity should rank high.
<CodeGroup>
```json Configuration theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": "{{INPUT.image_url}}",
"top_k": 100
}
],
"fusion": "max",
"final_top_k": 25
}
}
}
```
</CodeGroup>
## Learned Fusion
Learned fusion replaces static weights with weights that adapt automatically from user [interaction](/docs/retrieval/interactions) data. Under the hood, it uses Thompson Sampling with Beta distributions to balance exploration (trying different weight combinations) with exploitation (using what's known to work).
See [Learned Fusion](/docs/relevance/learned-fusion) for the full deep dive.
<CodeGroup>
```json Configuration theme={null}
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.image_url}}",
"top_k": 100
}
],
"fusion": "learned",
"final_top_k": 25
}
}
}
```
</CodeGroup>
## Choosing a Strategy
```
Do you have multiple search features?
├── No → No fusion needed (single feature search)
└── Yes
├── Do you have interaction data (100+ signals)?
│ ├── Yes → Use "learned" (adapts automatically)
│ └── No
│ ├── Do you know which features matter more?
│ │ ├── Yes → Use "weighted"
│ │ └── No
│ │ ├── Do features have different score scales?
│ │ │ ├── Yes → Use "dbsf"
│ │ │ └── No → Use "rrf" (default)
│ └── Is any single match sufficient?
│ └── Yes → Use "max"
```
## Performance Comparison
| Strategy | Latency Overhead | Configuration Effort | Adapts Over Time | Best Quality Ceiling |
| -------- | :--------------: | :------------------: | :--------------: | :------------------: |
| RRF | \< 5ms | None | No | Good |
| DBSF | \< 5ms | None | No | Good |
| Weighted | \< 5ms | Manual tuning | No | Good (if well-tuned) |
| Max | \< 5ms | None | No | Moderate |
| Learned | \< 10ms | Interaction tracking | Yes | Best |
All strategies add minimal latency. The real difference is in quality: learned fusion converges toward optimal weights for your specific users and content, while static strategies require manual tuning or accept a one-size-fits-all approach.
## Related
* [Feature Search stage](/docs/retrieval/stages/feature-search) — where fusion is configured
* [Learned Fusion deep dive](/docs/relevance/learned-fusion) — Thompson Sampling explained
* [Interaction Signals](/docs/retrieval/interactions) — capturing the data that powers learned fusion
* [Evaluations](/docs/retrieval/evaluations) — measuring the impact of different fusion strategies
# Learned Fusion
Source: https://docs.mixpeek.com/docs/relevance/learned-fusion
How Thompson Sampling adapts fusion weights from user interactions to personalize search results
Learned fusion automatically discovers the optimal blend of embedding features for your users. Instead of manually setting weights (`text: 0.7, image: 0.3`), the system learns from interaction data which features produce results users engage with.
<Frame>
<img alt="Thompson Sampling: Beta distributions evolve from uniform to peaked as interactions accumulate" />
</Frame>
## How It Works
Learned fusion uses **Thompson Sampling**, a well-studied algorithm for the multi-armed bandit problem. Here's how it applies to search fusion:
<Steps>
<Step title="Initialize with uniform priors">
Each search feature (e.g., text embeddings, image embeddings) starts with a Beta(1, 1) distribution — a flat line that assigns equal probability to all weight values. This means zero assumptions about which feature is better.
</Step>
<Step title="Sample weights at query time">
When a query arrives, the system draws a random weight from each feature's Beta distribution and normalizes them to sum to 1. Early on, samples are highly variable (exploration). As data accumulates, they stabilize (exploitation).
</Step>
<Step title="Execute search with sampled weights">
The feature search stage runs each embedding search and fuses results using the sampled weights — functionally identical to weighted fusion, but with dynamically chosen weights.
</Step>
<Step title="Capture user interactions">
Users interact with results: clicks, purchases, skips. Each interaction is recorded with the document ID, position, and the context key that identifies which weight sample was used.
</Step>
<Step title="Update Beta distributions">
Positive interactions (clicks, purchases) increment the `alpha` parameter: `alpha = 1 + clicks`. Non-engagement increments `beta`: `beta = 1 + (impressions - clicks)`. This shifts the distribution toward weights that produce engaging results.
</Step>
<Step title="Repeat with better weights">
Next query: the updated distributions produce weight samples closer to what works. After hundreds of interactions, the system converges on near-optimal weights while still occasionally exploring alternatives.
</Step>
</Steps>
## Thompson Sampling Explained
Think of it like flipping weighted coins. Each feature has its own coin:
* **At the start**, both coins are fair — you have no idea which feature is better, so you flip both and take whatever comes up.
* **After 50 interactions**, the text feature's coin lands "heads" 65% of the time (users click on text-matched results more). You naturally start weighting text higher, but still try image sometimes.
* **After 1000 interactions**, the text coin lands heads 72% of the time with very little variance. You're confident in the weights and rarely deviate.
The mathematical version: each "coin" is a Beta(alpha, beta) distribution where alpha counts successes (clicks) and beta counts non-successes (impressions without clicks). Sampling from this distribution gives you a weight that naturally balances exploration and exploitation.
## Hierarchical Fallback
Not every user has enough interaction history for personalized weights. The system uses a four-level fallback:
| Level | Context | Min Interactions | When Used |
| --------------- | --------------- | :--------------: | ------------------------------------------------------ |
| **Personal** | Individual user | 5 | User has clicked/purchased enough for reliable weights |
| **Demographic** | User segment | 1 | User is new, but their segment has data |
| **Global** | All users | 1 | No segment data; uses aggregate behavior |
| **Prior** | Uniform | 0 | No interactions at all; falls back to equal weights |
The `user_id` in your interaction signals enables personal-level learning. The `demographic_features` config (e.g., `["INPUT.user_segment"]`) enables demographic-level learning.
## End-to-End Walkthrough
### 1. Create a retriever with learned fusion
<CodeGroup>
```json Retriever Configuration theme={null}
{
"retriever_name": "product-search-learned",
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"fusion": "learned",
"final_top_k": 25
}
}
}
]
}
```
</CodeGroup>
### 2. Execute a search
```bash theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"inputs": {
"query": "wireless earbuds noise canceling",
"user_id": "user_456"
}
}'
```
With zero interactions, this behaves like RRF (uniform weights). The response includes an `execution_id` you'll use for interaction tracking.
### 3. Capture interactions
```bash theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/interactions" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"feature_id": "doc_product_789",
"interaction_type": ["click", "purchase"],
"position": 2,
"metadata": {
"query": "wireless earbuds noise canceling"
},
"user_id": "user_456",
"session_id": "sess_abc"
}'
```
### 4. Improved results over time
After 100+ interactions, the same search for `user_456` returns results with personalized fusion weights. If this user consistently engages with text-matched results over image-matched ones, the text feature weight increases for their queries.
### 5. Verify convergence
Use [analytics](/docs/relevance/analytics) to check how weights are evolving:
```bash theme={null}
curl "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/signals?signal_type=learned_weights&hours=168" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
## Response Metadata
When learned fusion is active, the execution response includes a `__learned_fusion__` object in each result's metadata. Use it to verify the system is working and debug weight evolution:
```json theme={null}
{
"__learned_fusion__": {
"context_level": "personal",
"context_key": "a3f8b2c1d4e5f607",
"sampled_weights": {
"mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1": 0.72,
"mixpeek://image_extractor@v1/google_siglip_base_v1": 0.28
},
"feature_uris": [
"mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"mixpeek://image_extractor@v1/google_siglip_base_v1"
],
"effective_exploration": 0.37,
"circuit_breaker_triggered": false,
"weight_resolution_ms": 12.5
}
}
```
| Field | Description |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `context_level` | Which fallback level was used: `personal`, `demographic`, `global`, or `none` (circuit breaker triggered, using uniform weights) |
| `context_key` | A truncated SHA-256 hash of the context values used to look up interaction history (e.g., `a3f8b2c1d4e5f607`) |
| `sampled_weights` | The actual weights used for this query, keyed by feature URI |
| `effective_exploration` | Current exploration multiplier after decay — lower means more exploitation |
| `circuit_breaker_triggered` | `true` if the weight lookup timed out and fell back to uniform weights |
| `weight_resolution_ms` | How long the weight lookup took in milliseconds |
<Tip>
Check `context_level` to verify personalization is active. If you see `"none"` consistently, the user may not have enough interactions (check `min_interactions`) or the circuit breaker may be triggering due to ClickHouse latency.
</Tip>
## Configuration Reference
<ParamField type="string">
Set to `"learned"` to enable Thompson Sampling fusion.
</ParamField>
<ParamField type="string">
Each feature URI defines an "arm" in the bandit. The system learns a separate weight for each.
</ParamField>
<ParamField type="string">
Passed at execution time. Enables personal-level weight learning. Without this, the system uses global weights only.
</ParamField>
The Thompson Sampler uses these parameters, configurable in `learning_config`:
| Parameter | Default | Range | Description |
| ------------------- | ------- | ---------- | ----------------------------------------------------------------------------------- |
| `prior_alpha` | `1.0` | `>= 0.1` | Beta distribution alpha prior. Higher = initial belief that features are effective |
| `prior_beta` | `1.0` | `>= 0.1` | Beta distribution beta prior. Higher = initial belief that features are ineffective |
| `exploration_bonus` | `1.0` | `0.1–10.0` | Multiplier for distribution variance; >1 increases exploration |
| `min_interactions` | `5` | — | Minimum interactions before using personal context |
## When to Use Learned vs Static
| Scenario | Recommendation | Why |
| -------------------------------------------- | ----------------- | ---------------------------------------------------------- |
| New product, no interaction data | `rrf` | No data to learn from; RRF is a strong default |
| Domain expert knows feature importance | `weighted` | Manual weights capture expert knowledge immediately |
| Diverse user base with different preferences | `learned` | Different users may benefit from different feature weights |
| A/B testing fusion approaches | `rrf` → `learned` | Start with baseline, measure improvement with evaluations |
| Single search feature | None needed | Fusion only applies when combining multiple features |
## Session-Level Adaptation
Learned fusion persists weight state in ClickHouse, which has write-then-read latency (seconds to minutes). For within-session adaptation -- where a user's first few clicks should influence their next search immediately -- the system uses a Redis session cache.
When a user interacts with a result, the interaction is written to both ClickHouse (durable) and a Redis session cache (ephemeral, 1-hour TTL). On the next search in the same session, the bandit merges the session cache entries into the ClickHouse-backed Beta distributions before sampling:
```
1. Read base α/β from ClickHouse (persistent history)
2. Read session interactions from Redis (current session)
3. Merge: α += session_successes, β += session_failures
4. Sample weights from the merged posterior
```
This gives sub-50ms feedback within a session while ClickHouse handles long-term persistence. Pass `session_id` on both search and interaction requests to enable this:
```python theme={null}
results = client.retrievers.execute(
retriever_id,
inputs={
"query": "running shoes",
"user_id": "user_456",
"session_id": "sess_abc",
},
)
```
<Note>
Without `session_id`, the system still learns from interactions -- it just won't reflect them until ClickHouse ingests them (typically a few seconds). Session-level adaptation is optional but recommended for real-time UX.
</Note>
## Temporal Decay
User preferences change over time. The system applies exponential decay to older interactions so recent behavior matters more:
```
decayed_reward = reward * (decay_factor ^ days_ago)
```
With the default `decay_factor: 0.995`, the decay curve looks like:
| Age | Retained Weight | Effect |
| -------- | :---------------: | ------------------------- |
| 1 day | 99.5% | Essentially full strength |
| 30 days | 86% (`0.995^30`) | Still strong |
| 90 days | 64% (`0.995^90`) | Noticeably faded |
| 180 days | 41% (`0.995^180`) | Weak influence |
| 365 days | 16% (`0.995^365`) | Nearly gone |
Configure decay in the `learning_config`:
```json theme={null}
{
"learning_config": {
"decay_factor": 0.995,
"decay_window_days": 365
}
}
```
* **`decay_factor`** (default `0.995`) -- per-day multiplier. Set to `1.0` to disable decay entirely.
* **`decay_window_days`** (default `365`) -- interactions older than this are ignored completely, reducing query cost.
<Warning>
Setting `decay_factor` too low (e.g., `0.95`) causes rapid forgetting -- a week-old interaction retains only 70% of its weight. Use values between `0.99` and `0.999` for most use cases.
</Warning>
## Weight Clamping
Thompson Sampling can produce extreme weights that effectively silence a feature (e.g., `text: 0.99, image: 0.01`). Weight clamping prevents this by enforcing minimum and maximum bounds:
```json theme={null}
{
"learning_config": {
"min_weight": 0.05,
"max_weight": 0.95
}
}
```
After sampling from the Beta posteriors and normalizing, each weight is clamped to `[min_weight, max_weight]` and then re-normalized. This guarantees that every feature contributes at least `min_weight` to the final fusion, even for users with heavily skewed interaction histories.
**Why this matters:** Without clamping, a user who clicks only text results could end up with `image: 0.01` -- effectively removing image search from their experience. If their preferences shift later, recovery is slow because the silenced feature produces almost no impressions to learn from.
## Exploration Decay
The `exploration_bonus` parameter controls how much the bandit explores (tries different weight combinations) vs. exploits (uses what it has learned). With a static bonus, the bandit never fully settles on the best weights.
Exploration decay reduces the bonus as interactions accumulate:
```
effective_exploration = max(exploration_floor, exploration_bonus * exploration_decay ^ total_interactions)
```
Configure it in `learning_config`:
```json theme={null}
{
"learning_config": {
"exploration_bonus": 1.0,
"exploration_decay": 0.99,
"exploration_floor": 0.1
}
}
```
* **`exploration_bonus`** (default `1.0`) -- initial exploration multiplier. Higher values mean more random early sampling.
* **`exploration_decay`** (default `0.99`) -- per-interaction decay rate.
* **`exploration_floor`** (default `0.1`) -- minimum exploration. The bandit never fully stops exploring -- this prevents it from getting permanently stuck on suboptimal weights if preferences change.
After 100 interactions: `1.0 * 0.99^100 = 0.37`. After 500: `1.0 * 0.99^500 = 0.007` (floored to `0.1`). The system converges toward exploitation while maintaining a baseline level of exploration.
## Multi-Signal Rewards
By default, learned fusion treats `click` as the only learning signal. The `reward_map` lets you assign different reward magnitudes to different interaction types:
```json theme={null}
{
"learning_config": {
"reward_map": {
"click": 1.0,
"purchase": 3.0,
"add_to_cart": 2.0,
"bookmark": 1.5,
"positive_feedback": 2.0,
"negative_feedback": -2.0,
"skip": -1.0
}
}
}
```
Positive values increase the `alpha` parameter for the associated feature (making it more likely to be weighted higher). Negative values increase the `beta` parameter (penalizing the feature). A purchase at `3.0` shifts weights three times as much as a click at `1.0`.
Per-interaction rewards are also capped at `max_reward_per_interaction` (default `5.0`) to prevent a single buggy or malicious interaction batch from dominating the learned weights.
See the [Reward Signals reference](/docs/retrieval/reward-signals) for all 17 supported interaction types and guidance on choosing reward values.
## Related
* [Auto-Tune overview](/docs/retrieval/auto-tune) -- the top-level guide to the full feedback loop
* [Reward Signals](/docs/retrieval/reward-signals) -- configuring which interactions drive learning
* [Rollout Guide](/docs/retrieval/auto-tune-rollout) -- traffic splitting, shadow mode, kill switch
* [Fusion Strategies](/docs/relevance/fusion-strategies) -- comparison of all 5 strategies
* [Interaction Signals](/docs/retrieval/interactions) -- capturing the data that powers learning
* [Evaluations](/docs/retrieval/evaluations) -- measuring learned fusion quality
* [Feature Search stage](/docs/retrieval/stages/feature-search) -- where fusion is configured
# Best Practices
Source: https://docs.mixpeek.com/docs/resources/best-practices
Schema design, feature selection, caching, and cost optimization
## Schema Design
* **One bucket per data domain** (products, support tickets, footage). Keep schemas coarse — collections slice data differently downstream.
* **Separate mutable from immutable fields.** Put stable data (file URL, created date) in the bucket schema. Put changing data (status, tags) in metadata that can be patched.
* **Use `key_prefix`** in objects for organization (e.g., `/2025/04/`).
* **Enable document schema validation** on collections (`validation_mode: "strict"`) to catch malformed data early.
## Feature Selection
| Data Type | Recommended Extractor | Output |
| ------------- | ------------------------- | ------------------------------------------ |
| Text search | `text_extractor` | E5-Large 1024D embeddings |
| Image search | `multimodal_extractor` | Vertex AI 1408D embeddings |
| Face matching | `face_identity_extractor` | ArcFace 512D embeddings |
| Video scenes | `multimodal_extractor` | Scene embeddings + transcripts + keyframes |
| Documents/PDF | `universal_extractor` | Text chunks + OCR + embeddings |
| Audio | `multimodal_extractor` | Transcripts + audio embeddings |
* **Start with one extractor per collection.** Add more collections for additional features rather than overloading one.
* **Match extractors to your query patterns.** If users search by text, prioritize text embeddings. If they search by image, prioritize visual embeddings.
* **Test with a small batch first** before processing your full corpus.
## Caching
Mixpeek caches at two levels:
* **Retriever-level** — cache the full pipeline result. Set `cache_config.ttl_seconds` on the retriever.
* **Stage-level** — cache expensive stages (KNN search, reranking) independently. Useful when early stages are stable but later stages change.
```json theme={null}
{
"cache_config": {
"enabled": true,
"ttl_seconds": 300
}
}
```
* Collection index signatures auto-invalidate caches when documents change.
* Use shorter TTLs (60-300s) for frequently updated collections, longer (3600s+) for stable corpora.
* Monitor cache hit rates via the [analytics API](/docs/api-reference/retriever-evaluations/list-evaluations).
## Cost Optimization
Mixpeek prices in dollars: each feature you enable is billed per modality unit (images, video minutes, document pages, text tokens, crawled web pages). Each tier includes a monthly dollar usage pool, and overage bills at the rate card — see [Billing](/docs/platform/billing). Quote a workload before running it with `POST /v1/organizations/billing/estimate`. Key cost drivers:
| Operation | Cost Level | Optimization |
| --------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Feature extraction | High | Choose the right extractor — don't over-extract |
| LLM enrichment stages | High | Set `max_tokens` limits, cache results |
| Vector search | Low | Reads are included with each tier (\$2 per 1M queries beyond included) — cheap; the heavy cost is enrichment/LLM stages. See [Rate Limits & Quotas](/docs/operations/rate-limits-quotas) |
| Storage | Low | Auto-tiered — hot/warm/cold based on access patterns |
**Top optimizations:**
1. **Deduplicate before ingesting** — skip objects already processed
2. **Use field passthrough** for metadata that doesn't need extraction
3. **Batch process** rather than single-object ingestion
4. **Cache retriever results** for repeated query patterns
5. **Set reranking limits** (`top_k`) to avoid scoring too many candidates
# Single Tenant
Source: https://docs.mixpeek.com/docs/resources/single-tenant
Dedicated infrastructure with isolated compute, storage, and data — deploy on any cloud, in any region
Mixpeek's single-tenant deployment gives enterprise customers a fully isolated data plane: dedicated database, compute cluster, cache, object storage, and job queues. The shared control plane (API gateway, auth, billing) routes requests to your data plane transparently — your API keys and SDKs work the same way.
## Architecture
<Frame>
<img alt="Single-tenant architecture: shared control plane routing to isolated enterprise data plane and shared standard data plane" />
</Frame>
### What's isolated
| Resource | Isolation Level | Details |
| ---------------- | --------------------- | --------------------------------------------------------------------------- |
| **Database** | Dedicated database | Separate MongoDB instance or database per tenant |
| **Compute** | Dedicated Ray cluster | Head node + autoscaling worker pools (CPU, batch, GPU) |
| **Cache** | Dedicated Redis | Separate instance with independent memory and connection pools |
| **Storage** | Dedicated bucket | Separate GCS/S3 bucket per tenant |
| **Job queues** | Dedicated queues | Celery queues prefixed per tenant — jobs never compete with other customers |
| **Vector store** | Dedicated shard | Isolated MVS shard with tenant-specific GCS-backed snapshots |
### What's shared
The control plane is stateless — it routes requests but holds no customer data:
* **API gateway** — resolves your API key to your data plane endpoints
* **Studio UI** — connects to the API, holds no data
* **Auth and API key management**
* **Billing and usage metering**
* **Container image registry** — same code, separate compute
## Cloud & Region Deployment
Mixpeek's single-tenant architecture supports deployment across cloud providers and regions. Each tenant's data plane is self-contained — all customer data stays in the region you choose.
### In-region co-location
Your data plane runs **within your cloud provider and region**. All traffic between your application and Mixpeek stays in-region — no cross-region or cross-cloud networking overhead. The only out-of-region hop is the initial API request through the control plane for auth and routing (\~1 RTT, no customer data persisted).
### Supported clouds and regions
| Cloud | Region | Location | Status |
| ------- | ------------ | -------------- | --------- |
| **GCP** | `us-east1` | South Carolina | Available |
| **GCP** | `eu-west1` | Belgium | On-demand |
| **AWS** | `us-east-1` | N. Virginia | On-demand |
| **AWS** | `eu-west-1` | Ireland | On-demand |
| **AWS** | `eu-west-3` | Paris | On-demand |
| **AWS** | `ap-south-1` | Mumbai | On-demand |
"On-demand" regions are provisioned when a customer commits. Lead time is approximately one week for the first tenant in a new region. Additional tenants in the same region deploy in hours.
Need a region not listed? Contact us — we can deploy to any GCP or AWS region.
### How it works
The control plane runs centrally and routes requests to your data plane via URL-based tenant configuration. Your `engine_url`, `mongo_uri`, `redis_url`, and `storage_bucket` all point to infrastructure in your chosen cloud and region.
```
┌─────────────────────────────┐
│ Control Plane (shared) │
│ api.mixpeek.com │
│ Auth · Routing · Billing │
└──────┬──────────┬────────────┘
│ │
┌────▼───┐ ┌──▼──────┐
│ GCP │ │ AWS │
│ GKE │ │ EKS │
│ GCS │ │ S3 │
└────────┘ └─────────┘
```
When you onboard, you select a cloud provider and region. Mixpeek provisions your isolated data plane there — dedicated compute, storage, database, and cache. The control plane reaches your data plane over private networking (VPC peering or internal load balancers), never over the public internet.
<Note>
Switching between cloud providers or regions after initial deployment requires a data migration. Choose your target cloud and region during onboarding.
</Note>
### Node & resource selection
Your tenant's workloads can run on dedicated node pools with tenant-specific taints and labels. This gives you:
* **Hardware selection** — choose machine types per worker group (CPU-optimized, memory-optimized, GPU)
* **Spot/preemptible nodes** — reduce cost for batch-tolerant workloads
* **GPU acceleration** — dedicated GPU nodes (NVIDIA L4, A100) for video processing and large model inference
* **Isolation guarantees** — tenant taints ensure no other workloads land on your nodes
Node pool configuration is defined in your tenant overrides file:
```yaml theme={null}
node_pools:
cpu-workers:
machine_type: n2-highmem-8 # or r6i.2xlarge on AWS
min_nodes: 1
max_nodes: 5
spot: true
gpu-workers:
machine_type: g2-standard-8 # or g5.2xlarge on AWS
min_nodes: 0
max_nodes: 4
accelerator:
type: nvidia-l4
count: 1
```
## Tenant Routing
Every API request goes through tenant resolution:
1. Your API key authenticates against the shared auth layer
2. The API resolves your organization to a tenant configuration
3. The tenant config specifies your data plane endpoints (database, cache, compute, storage)
4. The request executes entirely within your isolated infrastructure
<Note>
Tenant routing is transparent. Your API keys, SDKs, and integrations work identically to the shared platform — no code changes required.
</Note>
## Compute Cluster
Your Ray cluster runs in a dedicated Kubernetes namespace with independent scaling.
### Worker groups
| Group | Default Range | Use Case |
| ----------------- | --------------------- | ------------------------------------------------------------ |
| **CPU workers** | 1–4 nodes | Text embeddings, reranking, classification, image embeddings |
| **Batch workers** | 0–30 nodes | Large ingestion jobs (scale from zero on demand) |
| **GPU workers** | 0–8 nodes (NVIDIA L4) | Video processing, large model inference |
Each group autoscales independently. Batch and GPU workers can default to zero replicas and scale up when jobs arrive — you only pay for compute when it's active.
### Extractor scaling
Individual extractors (embedding models, classifiers, etc.) scale independently within your cluster:
* **`min_replicas`** — minimum always-running instances (0 = scale to zero when idle)
* **`max_replicas`** — maximum instances under load
* **`target_ongoing_requests`** — requests per replica before scaling up
* **`downscale_delay_s`** — cooldown before scaling down (prevents flapping)
<Tip>
Set `min_replicas: 1` for latency-sensitive extractors (e.g., your primary embedding model for search). Use `min_replicas: 0` for batch-only extractors to save cost.
</Tip>
### Disabling extractors
If you don't use certain capabilities (e.g., audio embeddings, face recognition, web scraping), disable the corresponding extractors. This frees compute resources for the extractors you do use and reduces your always-on footprint.
## Self-Service Configuration
Enterprise tenants manage their cluster configuration via a YAML overrides file. On each platform deploy, Mixpeek merges your overrides with the latest extractor registry — new extractors appear automatically, disabled extractors stay disabled.
### What you can configure
| Section | Controls |
| ----------------- | ----------------------------------------------------------------- |
| **`auto_deploy`** | When `true`, platform updates on main auto-deploy to your cluster |
| **`disabled`** | List of extractors to exclude from your cluster |
| **`overrides`** | Per-extractor scaling (min/max replicas, resources, concurrency) |
| **`cluster`** | Worker group sizing (replicas, min/max nodes) |
| **`head`** | Head node resources (CPU, memory) |
| **`celery`** | Batch and general worker pool sizing, concurrency, queue bindings |
| **`redis`** | Cache memory limits, persistence policy |
| **`mvs`** | Vector store shard config (WAL, snapshots, index parameters) |
| **`node_pools`** | Dedicated node pool machine types, autoscaling ranges, GPU config |
| **`env`** | Environment variable overrides |
| **`spec`** | Health check thresholds |
### Example overrides
```yaml theme={null}
# Auto-deploy platform updates (set false if running a fork)
auto_deploy: true
# Disable extractors you don't need
disabled:
- mixpeek__playwright # no web scraping
- laion__clap_htsat_tiny # no audio embeddings
- insightface__arcface # no face recognition
# Scale extractors for your workload
overrides:
intfloat__multilingual_e5_large_instruct:
autoscaling_config:
min_replicas: 2 # always warm for search
max_replicas: 6 # burst for batch ingestion
# Size your worker groups
cluster:
cpu-workers:
minReplicas: 1
maxReplicas: 4
batch-workers:
minReplicas: 0
maxReplicas: 5 # more batch capacity
gpu-workers:
minReplicas: 0
maxReplicas: 3
# Celery worker pools
celery:
batch:
replicas: 3
concurrency: 2
autoscaling:
minReplicas: 3
maxReplicas: 6
general:
replicas: 1
concurrency: 4
```
Changes take effect on the next deploy. When `auto_deploy: true`, every push to main automatically rebuilds and deploys to your cluster.
## Kubernetes Access
Enterprise tenants get operator-level access to their namespace:
| Action | Access |
| ----------------------------- | ------ |
| View pods, logs, events | Yes |
| Scale worker groups | Yes |
| Restart stuck pods | Yes |
| Port-forward to Ray dashboard | Yes |
| View Ray cluster status | Yes |
| Access secrets or RBAC | No |
| Modify other namespaces | No |
### Quick scaling
For immediate scaling (e.g., before a large batch job):
```bash theme={null}
# Scale batch workers to 3
kubectl -n <your-namespace> scale raycluster <your-cluster> \
--replicas=3 --resource-name=batch-workers
# Scale GPU workers to 1 for video processing
kubectl -n <your-namespace> scale raycluster <your-cluster> \
--replicas=1 --resource-name=gpu-workers
# Scale back down after the job
kubectl -n <your-namespace> scale raycluster <your-cluster> \
--replicas=0 --resource-name=batch-workers
```
<Note>
Manual scaling is temporary. The next deploy resets to the values in your overrides file.
</Note>
## Monitoring
### Ray Dashboard
Port-forward to access your Ray dashboard locally:
```bash theme={null}
kubectl -n <your-namespace> port-forward svc/<your-head-svc> 8265:8265
# Open http://localhost:8265
```
The dashboard shows active deployments, replica counts, request queues, worker resource usage, and cluster utilization.
### Grafana
Each tenant gets scoped Grafana dashboards:
* **Queue Health** — depth and age of your job queues
* **Batch Status** — ingestion progress, success/failure rates
* **API Latency** — p50/p95/p99 for your requests
* **Cost & Usage** — compute hours, resource utilization
* **Error Rate** — 5xx errors scoped to your tenant
### kubectl
```bash theme={null}
# Pod resource usage
kubectl -n <your-namespace> top pods
# Recent events (scheduling failures, OOM, probe failures)
kubectl -n <your-namespace> get events --sort-by=.lastTimestamp | tail -20
# Ray Serve status (which extractors are running)
kubectl -n <your-namespace> exec -it deploy/<your-head> -- serve status
```
## Billing
Single-tenant billing has two components:
1. **Platform fee** — fixed monthly fee for access to the Mixpeek platform, API, Studio, and support
2. **Compute passthrough** — actual cloud infrastructure cost (nodes, storage, networking) passed through at cost plus a management markup
There are no per-operation usage charges on the single-tenant plan. You pay for the underlying cloud resources your cluster consumes, and Mixpeek handles provisioning, monitoring, upgrades, and support.
<Tip>
Keep batch and GPU workers at `minReplicas: 0` — they scale from zero on demand. You only pay for compute when it's active. Disable unused extractors to reduce your always-on footprint.
</Tip>
## Troubleshooting
### Pods stuck in Pending
Check events for the pending pod:
```bash theme={null}
kubectl -n <your-namespace> describe pod <pod-name>
```
Common causes:
* **Insufficient resources** — cluster autoscaler is provisioning a new node (2-3 minutes)
* **GPU unavailable** — GPUs may be temporarily exhausted in the region
* **Resource limits** — reduce `maxReplicas` on other worker groups to free capacity
### OOMKilled pods
A pod exceeded its memory limit. Increase memory for the affected worker group in your overrides file:
```yaml theme={null}
cluster:
cpu-workers:
resources:
limits:
memory: "48Gi"
```
### Extractor returning 503
The extractor has no running replicas (scaled to zero) or all replicas are saturated:
* First request after idle takes 5-10 seconds for cold start
* Set `min_replicas: 1` for latency-sensitive extractors
* Increase `max_replicas` if you're seeing sustained 503s under load
### Batch jobs stuck
1. Check if batch workers are running: `kubectl -n <your-namespace> get pods -l ray.io/group=batch-workers`
2. If no batch workers, manually scale up: `kubectl -n <your-namespace> scale raycluster <your-cluster> --replicas=1 --resource-name=batch-workers`
3. Check queue depth in Grafana — a backlog is normal for large batches
## Custom Code (Fork Deploys)
Single-tenant customers can fork the Mixpeek codebase and deploy custom code to their tenant:
* **Custom extractors** — add domain-specific feature extraction logic
* **Modified inference** — tune model parameters, swap models, add pre/post-processing
* **Engine changes** — adjust batch processing, add custom endpoints
Your fork builds into a tenant-specific container image and deploys only to your namespace. The shared platform is unaffected.
### Workflow
1. Fork the Mixpeek repo
2. Make your changes (extractors, inference, engine code)
3. Trigger a tenant deploy via GitHub Actions — builds from your fork, deploys to your namespace
4. Rebase on upstream periodically to pick up platform updates
<Note>
Config-only changes (disabling extractors, adjusting scaling) don't require a fork or image build — edit your overrides file and trigger a deploy.
</Note>
## Getting Started
To provision a single-tenant data plane:
1. Contact your Mixpeek account manager or email [sales@mixpeek.com](mailto:sales@mixpeek.com)
2. Choose your cloud provider (GCP or AWS) and target region
3. We provision your isolated infrastructure (database, compute, cache, storage)
4. You receive kubectl access to your namespace and Grafana dashboards
5. Your existing API keys are routed to your dedicated data plane — no code changes
<Info>
Migration from the shared platform to single-tenant needs no change on your side. Your data is copied to the isolated database, the tenant config is updated, and routing switches instantly. Rollback is equally fast.
</Info>
# Troubleshooting
Source: https://docs.mixpeek.com/docs/resources/troubleshooting
Common errors, rate limits, and debugging tips
## API Error Format
All non-validation errors return a consistent envelope. The machine-readable
field is `type` (stable PascalCase), not a SCREAMING\_SNAKE `code`:
```json theme={null}
{
"success": false,
"status": 404,
"error": {
"message": "bucket not found",
"type": "NotFoundError",
"code": "optional_string",
"details": {}
}
}
```
Request-validation errors (`422`) use FastAPI's shape instead:
```json theme={null}
{
"detail": [
{ "loc": ["body", "bucket_schema"], "msg": "field required", "type": "value_error.missing" }
]
}
```
## Common Errors
| Type | Status | Cause | Fix |
| -------------------------------------- | ------ | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ValidationError` | 422 | Missing or invalid fields | Check required fields in the [API reference](https://api.mixpeek.com/docs/swagger) |
| `AuthenticationError` | 401 | Invalid, missing, or revoked API key | Verify the `Authorization: Bearer <key>` header |
| `UnauthorizedError` / `ForbiddenError` | 403 | Missing/wrong `X-Namespace`, or insufficient permissions | Check the `X-Namespace` header matches the resource you're calling |
| `PlanRequiredError` | 403 | Workspace has not picked a plan yet, so billable work is gated | [Choose a plan](https://studio.mixpeek.com/signup/plan). `GET /v1/organizations` returns `requires_plan: true` while gated — see [Billing](/docs/platform/billing) |
| `QuotaExceededError` | 403 | Plan usage/quota exceeded | See [Rate limits & quotas](/docs/operations/rate-limits-quotas) |
| `NotFoundError` | 404 | Resource doesn't exist | Verify the ID and namespace |
| `TooManyRequestsError` | 429 | Too many requests | Back off and retry with exponential delay (respect `Retry-After`) |
| `ProcessingError` | 500 | Processing error in engine | Check task details for the specific failure reason |
<Note>
**401 vs 403 — they mean different things.** A **401 `AuthenticationError`** means the API key itself is bad (missing, malformed, or revoked). A **403** means the key is valid but the request isn't allowed: either the `X-Namespace` header is missing/doesn't match the resource (`UnauthorizedError`/`ForbiddenError`), you've exceeded a plan quota (`QuotaExceededError`), or the workspace has not chosen a plan yet (`PlanRequiredError`). A new workspace hits `PlanRequiredError` first: it blocks namespace creation and any billable work until you pick a plan. Match the `type` field in the response body — not just the status code — to the fix.
</Note>
## Rate Limits
| Tier | Requests/min | Concurrent tasks |
| ---------- | ------------ | ---------------- |
| Free | 60 | 5 |
| Pro | 600 | 50 |
| Enterprise | Custom | Custom |
When you hit a 429, the response includes `Retry-After` header with seconds to wait.
## Debugging Checklist
### Objects not processing
1. Check batch status: `GET /v1/buckets/{bucket_id}/batches/{batch_id}`
2. Check task status: `GET /v1/tasks/{task_id}`
3. Verify the collection's `feature_extractor` matches the bucket schema's blob types
4. Check for failed documents: `GET /v1/buckets/{bucket_id}/batches/{batch_id}/failed-documents`
### Retriever returning zero results
Zero results almost always trace to one of these — check in order:
| Likely cause | How to confirm | Fix |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Documents aren't indexed yet** | `GET /v1/tasks/{task_id}` — did the batch reach `COMPLETED`? `POST /v1/collections/{id}/documents/list` — are there any docs? | Wait for the batch to reach `COMPLETED` / `COMPLETED_WITH_ERRORS` before querying |
| **Wrong `feature_uri`** (most common) | A mismatched URI matches nothing and returns 0 **silently** — feature\_search attaches a note to the response `warnings` array instead of erroring | `GET /v1/collections/{id}` → copy `vector_indexes[].feature_uri` exactly; see [Find your feature\_uri](/docs/processing/features#find-your-feature_uri-to-search) |
| **Filter or threshold too strict** | Temporarily remove `attribute_filter` stages and any `score_threshold` / `min_score`, then re-run | Loosen the filter or lower the threshold, then re-tighten |
| **Wrong `collection_identifiers`** | The retriever points at a collection that has no matching documents | Set `collection_identifiers` to the collection you actually ingested into |
| **Empty query input** | An empty `inputs` value yields an empty query embedding → empty results | Pass a non-empty query in `inputs` |
Then use the [explain endpoint](/docs/api-reference/retrievers/explain-retriever-execution-plan) to see the execution plan and per-stage candidate counts.
### Poor retrieval quality
1. Check if the right extractor is being used for your query type (text query → text embedding, image query → visual embedding)
2. Add a reranking stage to improve precision
3. Review the execution trace for score distributions
4. Consider adding more retriever stages (filters, MMR for diversity)
### Slow processing
1. Video processing time scales with duration — 1 min video ≈ 1-2 min processing
2. Use batch processing for bulk imports instead of single-object ingestion
3. Check for resource contention: `GET /v1/tasks?status=PROCESSING`
## FAQ
**Can I use multiple feature extractors on the same data?**
Yes — create multiple collections pointing to the same bucket, each with a different extractor.
**How do I re-process documents after changing a collection's extractor?**
Create a new batch with the same objects and submit it. New documents replace old ones.
**What file formats are supported?**
Video (MP4, MOV, AVI, WebM), Images (JPG, PNG, WebP, GIF), Audio (MP3, WAV, M4A, FLAC), Documents (PDF, DOCX, TXT, HTML).
**How do I delete all data in a namespace?**
Delete the namespace: `DELETE /v1/namespaces/{namespace_id}`. This removes all buckets, collections, retrievers, and documents.
**Is there a size limit for uploads?**
Default: 500MB per file. Enterprise plans support larger files. Use URL references for files already in cloud storage.
[Contact support →](https://mixpeek.com/contact)
# Auto-Tune
Source: https://docs.mixpeek.com/docs/retrieval/auto-tune
Self-improving relevance that adapts per user
Retriever fusion weights automatically adapt based on how each user interacts with search results. Instead of manually tuning `text: 0.7, image: 0.3`, the system learns from clicks, purchases, and feedback to discover the optimal blend for every user — using Thompson Sampling (a multi-armed bandit algorithm) with hierarchical fallback from personal to demographic to global priors.
Auto-Tune closes the gap between "search works" and "search works *for this user*" without building a separate recommendation system.
## How It Works
<Steps>
<Step title="User searches">
A query arrives with a `user_id`. The system looks up that user's learned fusion weights — or falls back to segment-level or global weights if the user is new.
</Step>
<Step title="Results ranked by fusion weights">
The feature search stage runs each embedding search and fuses results using the personalized weights. Early on, weights are exploratory (high variance). As data accumulates, they stabilize around what works for this user.
</Step>
<Step title="User interacts">
The user clicks, purchases, skips, or provides feedback. Each interaction is recorded with the document ID, position, and the feature URI that produced the match.
</Step>
<Step title="Weights updated">
Positive interactions (clicks, purchases) increase the weight of the feature that surfaced the result. Negative signals (skips, negative feedback) decrease it. Different interaction types carry different reward magnitudes — a purchase is a stronger signal than a click.
</Step>
<Step title="Next search reflects preferences">
The next query from this user samples from the updated weight distributions. After dozens of interactions, the system converges on near-optimal weights while still occasionally exploring alternatives.
</Step>
</Steps>
<Frame>
<img alt="Thompson Sampling: Beta distributions evolve from uniform to peaked as interactions accumulate" />
</Frame>
## Quick Start
### 1. Create a retriever with learned fusion
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "product-search-personalized",
"collection_identifiers": ["col_products"],
"input_schema": {
"query": { "type": "text", "description": "Search query", "required": true },
"user_id": { "type": "text", "description": "User identifier", "required": true }
},
"stages": [
{
"stage_name": "Personalized Search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
}
],
"fusion": "learned",
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {
"click": 1.0,
"purchase": 3.0,
"add_to_cart": 2.0,
"negative_feedback": -2.0
},
"min_interactions": 5,
"exploration_bonus": 1.0,
"decay_factor": 0.995
},
"final_top_k": 25
}
}
}
]
}'
```
```python Python SDK theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
retriever = client.retrievers.create(
retriever_name="product-search-personalized",
collection_identifiers=["col_products"],
input_schema={
"query": {"type": "text", "description": "Search query", "required": True},
"user_id": {"type": "text", "description": "User identifier", "required": True},
},
stages=[{
"stage_name": "Personalized Search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": {"input_mode": "text", "value": "{{INPUT.query}}"},
"top_k": 100,
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": {"input_mode": "text", "value": "{{INPUT.query}}"},
"top_k": 100,
},
],
"fusion": "learned",
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {
"click": 1.0,
"purchase": 3.0,
"add_to_cart": 2.0,
"negative_feedback": -2.0,
},
"min_interactions": 5,
"exploration_bonus": 1.0,
"decay_factor": 0.995,
},
"final_top_k": 25,
},
},
}],
)
retriever_id = retriever["retriever_id"]
```
</CodeGroup>
### 2. Execute with a user ID
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers/$RETRIEVER_ID/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"inputs": {
"query": "wireless earbuds noise canceling",
"user_id": "user_456"
}
}'
```
With zero interactions, this behaves like RRF (uniform weights). The response includes an `execution_id` and each result contains a `feature_id` and `feature_uri` — you'll use these in Step 3.
### 3. Post interactions — results automatically improve
When a user clicks, purchases, or otherwise engages with a result, post an interaction using the `feature_id` and `execution_id` from the search response:
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers/interactions" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"retriever_id": "'$RETRIEVER_ID'",
"feature_id": "doc_product_789",
"interaction_type": ["click", "purchase"],
"position": 2,
"user_id": "user_456",
"session_id": "sess_abc",
"execution_id": "exec_from_step_2",
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
}'
```
| Field | Required | Description |
| ------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `feature_id` | Yes | The `feature_id` from the search result the user interacted with. |
| `interaction_type` | Yes | Array of interaction types for this event (e.g. `["click"]` or `["click", "purchase"]`). Multiple types in one call record a compound action. |
| `position` | Yes | 0-indexed position of the result in the ranked list. Recorded for analytics and rank-aware metrics (NDCG). |
| `execution_id` | No | The `execution_id` from the search response. Links the interaction to a specific search. |
| `session_id` | No | Client-generated session identifier. Groups interactions within a browsing session. |
| `feature_uri` | No | The `feature_uri` that produced the result. When provided, the system learns which embedding features the user prefers. |
**Python SDK shortcut** — `create_interaction_from_result()` extracts all the IDs for you:
```python theme={null}
results = client.retrievers.execute(retriever_id, inputs={"query": "earbuds", "user_id": "user_456"})
client.retrievers.create_interaction_from_result(results, position=2, interaction_type=["purchase"], user_id="user_456")
```
After enough interactions, the same search for `user_456` returns results personalized to their feature preferences. If this user consistently engages with text-matched results over image-matched ones, the text feature weight increases for their queries.
## Key Concepts
<CardGroup>
<Card title="Reward Signals" icon="signal" href="/retrieval/reward-signals">
Configure how different interaction types (clicks, purchases, feedback) influence learned fusion weights. Customize the reward map, handle negative signals, and tune temporal decay.
</Card>
<Card title="Rollout & Safety" icon="shield-check" href="/retrieval/auto-tune-rollout">
Safely deploy learned fusion with traffic splitting, shadow mode, kill switches, per-user opt-out, and weight bounds. Includes a recommended rollout plan.
</Card>
<Card title="Interactions" icon="hand-pointer" href="/retrieval/interactions">
How to capture clicks, purchases, and feedback that power the learning loop.
</Card>
<Card title="Evaluations" icon="chart-line" href="/retrieval/evaluations">
Measure whether learned fusion actually improves retrieval quality. Compare NDCG, precision, and recall against static fusion baselines.
</Card>
</CardGroup>
## Configuration Reference
The `learning_config` object is set inside the feature search stage parameters alongside `fusion: "learned"`:
| Field | Type | Default | Description |
| ---------------------------- | ---------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `context_features` | `string[]` | `["INPUT.user_id"]` | Input fields used for personal-level weight learning. Each value references an `INPUT.*` field from the retriever's `input_schema`. |
| `demographic_features` | `string[]` | `[]` | Input fields for segment-level fallback (e.g., `"INPUT.user_segment"`). Used when a user has insufficient personal history. |
| `reward_signal` | `string` | `"click"` | *Deprecated.* Single interaction type used as the learning signal. Use `reward_map` instead. If both are provided, `reward_map` takes precedence and `reward_signal` is ignored. |
| `reward_map` | `object` | See [Reward Signals](/docs/retrieval/reward-signals) | Maps interaction types to reward magnitudes. Positive values reinforce; negative values penalize. |
| `min_interactions` | `integer` | `5` | Minimum interactions before using personal-level weights. Below this threshold, the system falls back to demographic or global weights. |
| `exploration_bonus` | `float` | `1.0` | Initial multiplier for distribution variance. Higher values increase exploration (more weight variability). |
| `exploration_decay` | `float` | `0.99` | Per-interaction decay applied to the exploration bonus. Gradually shifts from exploration to exploitation. |
| `exploration_floor` | `float` | `0.1` | Minimum exploration bonus. Prevents the system from fully exploiting — there is always some chance of trying alternative weights. |
| `decay_factor` | `float` | `0.995` | Per-day exponential decay applied to older interactions. `1.0` disables decay (interactions never fade). |
| `decay_window_days` | `integer` | `365` | Interactions older than this are ignored entirely. |
| `min_weight` | `float` | `0.05` | Minimum weight any feature can receive after sampling. Prevents a feature from being silenced. |
| `max_weight` | `float` | `0.95` | Maximum weight any feature can receive after sampling. Prevents one feature from completely dominating. |
| `rollout_pct` | `float` | `100.0` | Percentage of requests (0-100) that use learned weights. The rest use static fusion. Uses deterministic bucketing so a user does not flip-flop between treatments. |
| `shadow_mode` | `boolean` | `false` | When `true`, learned weights are computed and logged but static fusion results are served. Use this to evaluate learned fusion before going live. |
| `algorithm` | `string` | `"thompson_sampling"` | Learning algorithm. Currently only `"thompson_sampling"` is supported. |
| `prior_alpha` | `float` | `1.0` | Alpha parameter for the Beta prior distribution. Higher values bias toward exploitation. |
| `prior_beta` | `float` | `1.0` | Beta parameter for the Beta prior distribution. `alpha=1, beta=1` is a uniform prior. |
| `max_reward_per_interaction` | `float` | `5.0` | Maximum absolute reward value per interaction. Rewards beyond this are clamped. |
| `circuit_breaker_timeout_ms` | `integer` | `1000` | Timeout in milliseconds for learned weight resolution. If exceeded, falls back to static fusion. Range: 1–30000. |
| `fallback_strategy` | `string` | `"hierarchical"` | How to resolve weights when personal data is insufficient. `"hierarchical"` uses the four-level fallback below; `"global"` skips demographic and goes straight to global. |
<Note>
**Every `learning_config` field is optional** — each has the default shown above. The only field you almost always set is `context_features` (so the system knows which `INPUT.*` field identifies a user). The minimal config in the [quickstart](#1-create-a-retriever-with-learned-fusion) (`context_features` + `reward_map` + `min_interactions` + `exploration_bonus`) is enough to get started; the rest are tuning knobs.
</Note>
### Complete `learning_config` example
Every documented field, set to its default. Drop this into the feature search stage's `parameters` (alongside `fusion: "learned"`) and adjust only what you need — omitted fields fall back to these defaults:
```json theme={null}
"learning_config": {
"context_features": ["INPUT.user_id"],
"demographic_features": ["INPUT.user_segment"],
"reward_map": {
"click": 1.0,
"add_to_cart": 2.0,
"purchase": 3.0,
"negative_feedback": -2.0
},
"min_interactions": 5,
"exploration_bonus": 1.0,
"exploration_decay": 0.99,
"exploration_floor": 0.1,
"decay_factor": 0.995,
"decay_window_days": 365,
"min_weight": 0.05,
"max_weight": 0.95,
"rollout_pct": 100.0,
"shadow_mode": false,
"algorithm": "thompson_sampling",
"prior_alpha": 1.0,
"prior_beta": 1.0,
"max_reward_per_interaction": 5.0,
"circuit_breaker_timeout_ms": 1000,
"fallback_strategy": "hierarchical"
}
```
<Warning>
`min_weight` must be strictly less than `max_weight` (the API rejects the retriever otherwise), and `reward_map` keys must be valid [interaction types](/docs/retrieval/interactions#signal-types).
</Warning>
## Hierarchical Fallback
Not every user has enough interaction history for personalized weights. The system uses a four-level fallback:
| Level | Context | Min Interactions | When Used |
| --------------- | --------------- | :----------------------: | ------------------------------------------------------------------------ |
| **Personal** | Individual user | Configurable (default 5) | User has enough interactions for reliable personal weights |
| **Demographic** | User segment | 1 | User is new, but their segment (e.g., "enterprise", "consumer") has data |
| **Global** | All users | 1 | No segment data available; uses aggregate behavior across all users |
| **Prior** | Uniform | 0 | No interactions at all; falls back to equal weights (equivalent to RRF) |
The `context_features` field controls personal-level resolution (typically `["INPUT.user_id"]`). The `demographic_features` field enables segment-level fallback (e.g., `["INPUT.plan_tier"]`).
<Tip>
A new user starts at the Global or Prior level and automatically graduates to Personal as they interact with results — no configuration changes needed. The transition happens transparently at query time.
</Tip>
## Related
* [Interactions](/docs/retrieval/interactions) — capturing the user behavior that powers learning
* [Feature Search stage](/docs/retrieval/stages/feature-search) — where fusion and `learning_config` are configured
* [Fusion Strategies](/docs/relevance/fusion-strategies) — comparison of all 5 fusion strategies
# Rollout & Safety
Source: https://docs.mixpeek.com/docs/retrieval/auto-tune-rollout
Safely deploy learned fusion with traffic splitting, shadow mode, and kill switches
Learned fusion changes how search results are ranked — per user, per query. That power requires operational controls. This guide covers how to roll out Auto-Tune safely: start with shadow mode, ramp traffic gradually, monitor, and know how to revert instantly if something goes wrong.
## Traffic Splitting
The `rollout_pct` field controls what percentage of requests use learned fusion weights. The rest fall back to static fusion (RRF by default).
```json theme={null}
{
"learning_config": {
"rollout_pct": 10.0
}
}
```
Bucketing is **deterministic by user ID** — the same user always gets the same treatment (learned or static) on consecutive requests. This prevents a user from seeing different ranking behavior on every search.
```
rollout_pct: 0 → all requests use static fusion
rollout_pct: 10 → ~10% of users get learned fusion
rollout_pct: 50 → ~50% of users get learned fusion
rollout_pct: 100 → all requests use learned fusion (default)
```
<Info>
Bucketing uses a hash of the `user_id`. If no `user_id` is provided in the query, the request always uses static fusion regardless of `rollout_pct`.
</Info>
### Live rollout override
`rollout_pct` in `learning_config` is the *durable* setting. To **ramp rollout up or down without editing the retriever config** — for a gradual canary, or to dial back instantly during an incident — set a live override:
```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/rollout
{ "rollout_pct": 25 }
```
The override is stored in Redis and takes effect on the next request, just like the kill switch — no config mutation, no redeploy. Setting `rollout_pct: 100` clears the override so the configured value applies again. In the Studio, the **Learning tab → Rollout Controls** slider drives this endpoint. The `learned-fusion/stats` response reports the `rollout_pct` actually in effect plus a `rollout_override` flag and the underlying `config_rollout_pct`.
## Shadow Mode
Shadow mode computes learned fusion weights and logs what the results *would* have been, but serves static fusion results to the user. This lets you evaluate the quality of learned fusion before it affects real users.
```json theme={null}
{
"learning_config": {
"shadow_mode": true,
"rollout_pct": 100.0
}
}
```
In shadow mode:
1. Both fusion strategies execute in parallel
2. Static fusion results are served to the user
3. Learned fusion weights and re-ranked results are logged as `shadow_execution` metadata
4. The [evaluation system](/docs/retrieval/evaluations) can compare shadow vs. served results offline
The execution response includes a `learned_fusion_context` field (when shadow mode is active) so you can inspect what weights *would* have been used:
```json theme={null}
{
"results": [ ... ],
"learned_fusion_context": {
"shadow_mode": true,
"served_fusion": "rrf",
"sampled_weights": {
"mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1": 0.68,
"mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding": 0.32
},
"context_level": "personal",
"effective_exploration": 0.42,
"feature_uris": [
"mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding"
]
}
}
```
<Tip>
Run shadow mode for at least one week before enabling live traffic. Compare NDCG and click-through rates between shadow and served results using [evaluations](/docs/retrieval/evaluations).
</Tip>
## Kill Switch
Instantly disable learned fusion for all users on a retriever:
```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/disable
```
This sets a separate kill-switch key in Redis — it does not modify `rollout_pct`. All requests fall back to static fusion within the next request cycle. Re-enabling removes the kill-switch key, restoring the previous rollout percentage.
To re-enable:
```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/enable
```
The kill switch does not modify the retriever configuration permanently. It is an operational toggle — the `learning_config` and accumulated user weights remain intact, ready to resume when re-enabled.
<Warning>
The kill switch is an emergency control. For planned rollbacks, use `rollout_pct: 0` in the retriever configuration instead — that is the durable setting.
</Warning>
## Per-User Opt-Out
Exclude a specific user from learned fusion. Their requests will always use static fusion regardless of `rollout_pct`.
```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/opt-out/{user_id}
```
Use cases:
* Internal test accounts that would skew weights
* Users who reported unexpected result changes
* Debugging — isolate a user to static fusion while investigating
To re-include:
```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/opt-in/{user_id}
```
## Preference Reset
Delete all learned personalization for a specific user, returning them to global-level weights:
```bash theme={null}
DELETE /v1/retrievers/{retriever_id}/learned-fusion/user/{user_id}
```
This:
1. Flushes their session cache entries
2. Marks pre-reset interactions so the aggregation query ignores them
3. Returns the user to global fallback weights immediately
The user can begin building new personal weights from scratch through future interactions.
## Weight Bounds
Even with strong personalization data, individual feature weights are clamped to prevent degenerate rankings:
```json theme={null}
{
"learning_config": {
"min_weight": 0.05,
"max_weight": 0.95
}
}
```
| Setting | Default | Effect |
| ------------ | :-----: | ----------------------------------------------------------------------------- |
| `min_weight` | `0.05` | No feature drops below 5% weight — it always contributes something to results |
| `max_weight` | `0.95` | No feature exceeds 95% weight — a single feature cannot completely dominate |
After Thompson Sampling produces raw weights, they are clamped to `[min_weight, max_weight]` and re-normalized to sum to 1.0. This means even a user with extreme interaction patterns will always see results influenced by all configured features.
## Circuit Breaker
If learned fusion weight resolution takes too long (due to ClickHouse latency, Redis issues, or high load), the system automatically falls back to static fusion for that individual request:
```
learned fusion resolution > circuit_breaker_timeout_ms (default 1000ms) → fall back to static fusion
```
The timeout is configurable via the `circuit_breaker_timeout_ms` field in `learning_config` (range 1–30000ms).
The circuit breaker is per-request, not per-retriever. A single slow request does not disable learned fusion for everyone. The response's `learned_fusion_context` metadata indicates when a fallback occurred:
```json theme={null}
{
"learned_fusion_context": {
"context_level": "none",
"sampled_weights": { ... },
"effective_exploration": 1.0,
"circuit_breaker_triggered": true,
"weight_resolution_ms": 1023.4
}
}
```
A `context_level` of `"none"` with `circuit_breaker_triggered: true` indicates the circuit breaker fired and uniform priors were used. The `weight_resolution_ms` field shows the actual resolution time. Monitor the `mxp_learned_fusion_circuit_breaker_total` Prometheus metric to detect systemic latency issues.
## Recommended Rollout Plan
<Steps>
<Step title="Shadow mode (1-2 weeks)">
Enable `shadow_mode: true` with `rollout_pct: 100`. All users get static results, but learned fusion runs in parallel and logs what it would have returned.
**Check:** Run evaluations comparing shadow vs. served results. Learned fusion NDCG should be equal to or better than static.
</Step>
<Step title="1% canary (1 week)">
Set `shadow_mode: false`, `rollout_pct: 1.0`. A small fraction of users get learned fusion.
**Check:** Monitor click-through rates and circuit breaker triggers. No degradation vs. control group.
</Step>
<Step title="10% ramp (1 week)">
Set `rollout_pct: 10.0`. Enough traffic to produce statistically significant comparisons.
**Check:** Run a [benchmark](/docs/retrieval/evaluations) comparing learned vs. static. Expect learned fusion to show improvement for users with sufficient interaction history.
</Step>
<Step title="50% ramp (1 week)">
Set `rollout_pct: 50.0`. Half of users get personalized results.
**Check:** Monitor weight distributions across users. Look for convergence patterns (weights stabilizing) rather than oscillation.
</Step>
<Step title="100% GA">
Set `rollout_pct: 100.0`. All users get personalized fusion weights, with hierarchical fallback for new users.
**Check:** Keep evaluations running on a schedule to catch regressions. Set up alerts for high global fallback rates and circuit breaker storms.
</Step>
</Steps>
At any step, if you see regression:
1. Use the **kill switch** for immediate revert
2. Investigate using the `/learned-fusion/weights/{user_id}` endpoint to inspect specific users
3. Check whether the issue is global (bad reward map) or user-specific (outlier behavior)
4. Adjust the `learning_config` and restart from the previous step
## Python SDK
All rollout controls are available via the Python SDK:
<CodeGroup>
```python Kill Switch theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="sk_...")
# Disable learned fusion (emergency kill switch)
client.retrievers.disable_learned_fusion("ret_abc123")
# Re-enable
client.retrievers.enable_learned_fusion("ret_abc123")
```
```python Rollout & Opt-Out theme={null}
# Set rollout percentage (live override, no config change)
client.retrievers.set_rollout("ret_abc123", rollout_pct=25.0)
# Opt out a specific user
client.retrievers.opt_out_user("ret_abc123", user_id="user_456")
# Opt them back in
client.retrievers.opt_in_user("ret_abc123", user_id="user_456")
# Reset all personalization for a user
client.retrievers.reset_user("ret_abc123", user_id="user_456")
```
```python Monitoring theme={null}
# Global weight distribution
weights = client.retrievers.get_weights("ret_abc123")
print(weights["feature_weights"])
# Per-user weights and context level
user_weights = client.retrievers.get_user_weights("ret_abc123", user_id="user_456")
print(user_weights["context_level"]) # "personal", "demographic", or "global"
# Aggregate stats (learner count, interaction count, rollout info)
stats = client.retrievers.get_stats("ret_abc123")
# Recent activity feed
activity = client.retrievers.get_activity("ret_abc123")
```
</CodeGroup>
## Related
* [Auto-Tune](/docs/retrieval/auto-tune) — overview of the full feedback loop
* [Reward Signals](/docs/retrieval/reward-signals) — tuning how interactions influence weights
* [Evaluations](/docs/retrieval/evaluations) — measuring retrieval quality
# Evaluations
Source: https://docs.mixpeek.com/docs/retrieval/evaluations
Measure and compare retriever quality with ground truth datasets and standard IR metrics
Evaluations run your retriever against a curated set of queries with known-relevant documents, then compute standard information retrieval metrics at multiple cutoff points. Use them to quantify retriever quality, compare configurations, and catch regressions before they reach production.
## Quickstart
<Steps>
<Step title="Create a ground truth dataset">
Each query pairs an input with the document IDs that should be returned.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/evaluations/datasets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"dataset_name": "product-search-golden",
"queries": [
{
"query_id": "q1",
"query_input": {"query": "wireless earbuds"},
"relevant_documents": ["doc_a1", "doc_a2", "doc_a3"],
"relevance_scores": {"doc_a1": 5, "doc_a2": 3, "doc_a3": 2}
},
{
"query_id": "q2",
"query_input": {"query": "noise canceling headphones"},
"relevant_documents": ["doc_b1", "doc_b4", "doc_b7"]
}
]
}'
```
```python Python theme={null}
dataset = client.retrievers.evaluations.create_dataset(
dataset_name="product-search-golden",
queries=[
{
"query_id": "q1",
"query_input": {"query": "wireless earbuds"},
"relevant_documents": ["doc_a1", "doc_a2", "doc_a3"],
"relevance_scores": {"doc_a1": 5, "doc_a2": 3, "doc_a3": 2},
},
{
"query_id": "q2",
"query_input": {"query": "noise canceling headphones"},
"relevant_documents": ["doc_b1", "doc_b4", "doc_b7"],
},
],
)
```
```javascript JavaScript theme={null}
const dataset = await client.retrievers.evaluations.createDataset({
datasetName: "product-search-golden",
queries: [
{
queryId: "q1",
queryInput: { query: "wireless earbuds" },
relevantDocuments: ["doc_a1", "doc_a2", "doc_a3"],
relevanceScores: { doc_a1: 5, doc_a2: 3, doc_a3: 2 },
},
{
queryId: "q2",
queryInput: { query: "noise canceling headphones" },
relevantDocuments: ["doc_b1", "doc_b4", "doc_b7"],
},
],
});
```
</CodeGroup>
</Step>
<Step title="Run the evaluation">
Execute your retriever against every query in the dataset. The evaluation runs asynchronously and returns a `task_id` for progress tracking.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/evaluations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"dataset_name": "product-search-golden",
"evaluation_config": {
"k_values": [1, 5, 10, 20],
"metrics": ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"]
}
}'
```
```python Python theme={null}
evaluation = client.retrievers.evaluations.run(
retriever_id="ret_abc123",
dataset_name="product-search-golden",
evaluation_config={
"k_values": [1, 5, 10, 20],
"metrics": ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"],
},
)
# evaluation["task_id"], evaluation["evaluation_id"]
```
```javascript JavaScript theme={null}
const evaluation = await client.retrievers.evaluations.run({
retrieverId: "ret_abc123",
datasetName: "product-search-golden",
evaluationConfig: {
kValues: [1, 5, 10, 20],
metrics: ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"],
},
});
// evaluation.taskId, evaluation.evaluationId
```
</CodeGroup>
</Step>
<Step title="Get results">
Poll the evaluation until `status` is `completed`.
```bash theme={null}
curl "$MP_API_URL/v1/retrievers/{retriever_id}/evaluations/{evaluation_id}" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```json Example response theme={null}
{
"evaluation_id": "eval_abc123",
"status": "completed",
"query_count": 50,
"overall_metrics": {
"precision_at_5": 0.85,
"recall_at_5": 0.72,
"f1_at_5": 0.78,
"f2_at_5": 0.74,
"ndcg_at_5": 0.81,
"map": 0.71,
"mrr": 0.93
},
"metrics_by_k": {
"1": {"precision": 0.90, "recall": 0.18, "f1": 0.30, "f2": 0.21, "ndcg": 0.90},
"5": {"precision": 0.85, "recall": 0.72, "f1": 0.78, "f2": 0.74, "ndcg": 0.81},
"10": {"precision": 0.75, "recall": 0.88, "f1": 0.81, "f2": 0.85, "ndcg": 0.89},
"20": {"precision": 0.62, "recall": 0.95, "f1": 0.75, "f2": 0.86, "ndcg": 0.91}
}
}
```
</Step>
</Steps>
## Metrics
Every metric is computed at each K value you specify. The defaults cover most use cases:
| Metric | What It Measures | Formula |
| ---------------- | ---------------------------------------- | -------------------------------------------------------------------------------- |
| **Precision\@K** | Accuracy of top results | relevant in top K ÷ K |
| **Recall\@K** | Coverage of relevant documents | relevant in top K ÷ total relevant |
| **F1\@K** | Balanced precision/recall | harmonic mean of P and R |
| **F2\@K** | Recall-weighted balance | weighted harmonic mean (β=2), penalizes missed docs 4× more than false positives |
| **MAP** | Ranking quality across all queries | average of precision at each relevant doc's position |
| **MRR** | How quickly users find a relevant result | 1 ÷ rank of first relevant document |
| **NDCG\@K** | Ranking quality with graded relevance | normalized discounted cumulative gain |
<Note>
**F2 vs F1:** Use F2 when missing a relevant document is worse than showing an irrelevant one — the common case in search, recommendations, and discovery. F1 treats both errors equally.
</Note>
### Reading Your Scores
| Score | What It Tells You |
| ----------------------- | ------------------------------------------------------------------------------------------ |
| **NDCG\@10 = 0.89** | Your top-10 ranking captures 89% of the ideal ordering. Relevant docs appear near the top. |
| **Precision\@5 = 0.85** | 4–5 of every 5 results are relevant. Users see high-quality results. |
| **Recall\@20 = 0.95** | You surface 95% of all relevant documents within the top 20. Strong coverage. |
| **F2\@10 = 0.85** | Recall-weighted balance is strong — few relevant documents are being missed. |
| **MRR = 0.93** | The first relevant result typically appears at position 1 or 2. |
| **MAP = 0.71** | Overall ranking quality is solid but there's room to improve ordering. |
### Graded Relevance
When you provide `relevance_scores` in your dataset, NDCG uses graded relevance instead of binary. This distinguishes "exactly right" from "somewhat relevant":
```json theme={null}
{
"query_id": "q1",
"query_input": {"query": "wireless earbuds"},
"relevant_documents": ["doc_a1", "doc_a2", "doc_a3"],
"relevance_scores": {
"doc_a1": 5,
"doc_a2": 3,
"doc_a3": 1
}
}
```
| Score | Meaning |
| ----- | ------------------- |
| 5 | Perfect match |
| 3–4 | Highly relevant |
| 1–2 | Marginally relevant |
| 0 | Not relevant |
Without `relevance_scores`, all metrics use binary relevance (relevant or not).
## Comparing Retrievers
Run the same dataset against different retriever configurations to find the best pipeline:
```bash theme={null}
# Evaluate baseline (vector search only)
curl -X POST "$MP_API_URL/v1/retrievers/ret_baseline/evaluations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{"dataset_name": "product-search-golden"}'
# Evaluate candidate (vector search + reranker)
curl -X POST "$MP_API_URL/v1/retrievers/ret_reranked/evaluations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{"dataset_name": "product-search-golden"}'
```
Compare the `metrics_by_k` side by side:
| Metric | Baseline | + Reranker | Delta |
| ------------ | -------- | ---------- | ----- |
| NDCG\@10 | 0.78 | 0.89 | +14% |
| Precision\@5 | 0.72 | 0.85 | +18% |
| F2\@10 | 0.76 | 0.85 | +12% |
| MRR | 0.81 | 0.93 | +15% |
<Tip>
Run the same dataset after every pipeline change — adding stages, swapping models, adjusting fusion weights — to quantify the impact before deploying.
</Tip>
## Ground Truth Datasets
### Dataset Requirements
* At least 1 query (aim for 50+ for statistically meaningful results)
* Each query must have at least 1 relevant document
* `query_input` must match your retriever's input schema
* `relevance_scores`, if provided, must cover all `relevant_documents`
### Managing Datasets
```bash theme={null}
# List all datasets
curl "$MP_API_URL/v1/retrievers/evaluations/datasets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
# Get a specific dataset
curl "$MP_API_URL/v1/retrievers/evaluations/datasets/product-search-golden" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
### Building Good Datasets
<CardGroup>
<Card title="Include query diversity" icon="shuffle">
Cover head queries (popular), torso (moderate), and tail (rare/specific). Don't just test easy cases.
</Card>
<Card title="Use graded relevance" icon="ranking-star">
Binary relevant/not-relevant misses nuance. Score documents 0–5 so NDCG can distinguish good rankings from great ones.
</Card>
<Card title="Match real traffic" icon="chart-line">
Sample queries from production logs. Synthetic queries test what you think users ask, not what they actually ask.
</Card>
<Card title="Version your datasets" icon="code-branch">
Keep datasets stable across evaluations so you can track metric trends over time. Create new versions for schema changes.
</Card>
</CardGroup>
## Configuration Reference
<ParamField type="string">
Name of the ground truth dataset to evaluate against.
</ParamField>
<ParamField type="integer[]">
Cutoff positions for @K metrics. Include the K values that match your UI — if you show 10 results per page, include `10`.
</ParamField>
<ParamField type="string[]">
Metrics to compute. Available: `precision`, `recall`, `f1`, `f2`, `map`, `ndcg`, `mrr`.
</ParamField>
## Related
* [Retriever Benchmarks](/docs/api-reference/retriever-benchmarks/create-benchmark) — replay live sessions against candidate retrievers
* [Improve Relevance](/docs/platform/improve-relevance) — interaction signals, fusion strategies, and the feedback loop
* [API Reference: Run Evaluation](/docs/api-reference/retriever-evaluations/run-evaluation) — full endpoint specification
* [API Reference: Create Dataset](/docs/api-reference/retriever-evaluations/create-evaluation-dataset) — dataset creation endpoint
# Filters
Source: https://docs.mixpeek.com/docs/retrieval/filters
Compose filter conditions with logical operators
<Frame>
<img alt="Filter composition: AND, OR, and NOT operators nest to build complex filter logic on document payloads" />
</Frame>
Filters narrow results using logical operators to combine conditions. They operate on document payloads (metadata, enrichments, passthrough fields) and can be applied in retriever execution or as dedicated `filter@v1` stages.
## Payload Indexes
<Warning>
Filters require **payload indexes** on the fields you filter by. Without an index, the vector store performs a full scan — which is slow on large collections and may return incomplete results.
</Warning>
Create indexes on your namespace before using filters:
```bash theme={null}
curl -X PATCH https://api.mixpeek.com/v1/namespaces/{namespace_id} \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"payload_indexes": [
{"field_name": "metadata.category", "type": "keyword"},
{"field_name": "metadata.price", "type": "integer"},
{"field_name": "metadata.status", "type": "keyword"}
]
}'
```
Supported index types:
| Type | Use for |
| ---------- | ------------------------------------------- |
| `keyword` | Exact-match strings (categories, IDs, tags) |
| `integer` | Whole numbers |
| `float` | Decimal numbers |
| `bool` | Boolean fields |
| `datetime` | Timestamps |
| `text` | Full-text search |
| `geo` | Geospatial queries |
If you filter on an unindexed field, the response includes a `warnings` array telling you which fields need indexes:
```json theme={null}
{
"warnings": [
"Filter field 'brand' has no payload index — filtering may be slow or unreliable on large collections. Create an index via PATCH /v1/namespaces/{namespace_id} with payload_indexes: [{\"field_name\": \"brand\", \"type\": \"keyword\"}]"
]
}
```
<Note>
System fields (`collection_id`, `bucket_id`, `object_id`, `batch_id`) and `_internal.*` fields are indexed automatically — you only need to create indexes for your own fields.
</Note>
## Logical Operators
Mixpeek filters support three logical operators for composing conditions:
| Operator | Description | Usage |
| -------- | ----------------------------------- | ------------------------------------- |
| `AND` | All conditions must be true | Combine multiple required constraints |
| `OR` | At least one condition must be true | Match any of several alternatives |
| `NOT` | Inverts the condition | Exclude matching documents |
### AND Operator
Requires all nested conditions to match:
```json theme={null}
{
"AND": [
{ "field": "metadata.status", "operator": "eq", "value": "published" },
{ "field": "metadata.price", "operator": "lte", "value": 100 }
]
}
```
### OR Operator
Matches if any nested condition is true:
```json theme={null}
{
"OR": [
{ "field": "metadata.category", "operator": "eq", "value": "video" },
{ "field": "metadata.category", "operator": "eq", "value": "audio" }
]
}
```
### NOT Operator
Excludes documents matching the condition:
```json theme={null}
{
"NOT": {
"field": "metadata.status", "operator": "eq", "value": "draft"
}
}
```
## Nesting Operators
Logical operators can be nested to create complex filter logic:
```json theme={null}
{
"AND": [
{ "field": "metadata.status", "operator": "eq", "value": "published" },
{
"OR": [
{ "field": "metadata.category", "operator": "eq", "value": "video" },
{ "field": "metadata.category", "operator": "eq", "value": "audio" }
]
},
{
"NOT": {
"field": "metadata.restricted", "operator": "eq", "value": true
}
}
]
}
```
This filter matches documents that are:
* Published **AND**
* Either video or audio **AND**
* Not restricted
## Comparison Operators
Use these operators within conditions:
| Operator | Description |
| ------------- | ----------------------------------------------------------------- |
| `eq` | Equals |
| `ne` | Not equals |
| `gt` | Greater than |
| `gte` | Greater than or equal |
| `lt` | Less than |
| `lte` | Less than or equal |
| `in` | Value in list |
| `nin` | Value not in list |
| `exists` | Field exists |
| `is_null` | Field is null |
| `contains` | String contains substring |
| `starts_with` | String starts with prefix |
| `ends_with` | String ends with suffix |
| `regex` | Matches regular expression |
| `text` | Full-text search (token-based; word order **not** preserved) |
| `phrase` | Exact phrase — matches the words **in order**, word-boundary safe |
<Note>
Use `text` to match any of the query tokens (BM25); use `phrase` when word
order matters — e.g. find a transcript where someone says an exact quote.
`{ "field": "transcription", "operator": "phrase", "value": "make america great again" }`
matches "...make america great again..." but not "america will be great again".
</Note>
## Geospatial Operators
Three operators filter by geographic location. Their `value` is an **object**, not
a scalar. See [Geospatial filtering](/docs/retrieval/stages/attribute-filter#geospatial-filtering)
for field formats, exact shapes, worked examples, and the `[lon, lat]` GeoJSON caveat.
| Operator | Description |
| ------------------ | ---------------------------------------- |
| `geo_radius` | Within N meters of a center point |
| `geo_bounding_box` | Inside a lat/lon box (antimeridian-safe) |
| `geo_polygon` | Inside an arbitrary polygon (≥ 3 points) |
```json theme={null}
{
"field": "location",
"operator": "geo_radius",
"value": { "center": { "lat": 40.758, "lon": -73.9855 }, "radius": 15000 }
}
```
## Geospatial Operators
Geospatial operators filter documents by a **location field** — a payload value
holding a geographic point. A point may be either an object `{ "lat": <num>, "lon": <num> }`
or a GeoJSON-style `[lon, lat]` array. A field holding a **list** of points matches
if **any** point satisfies the predicate.
| Operator | Matches documents whose location… |
| ------------------ | ----------------------------------------------------------------------- |
| `geo_radius` | falls within `radius` **meters** of a center point (haversine distance) |
| `geo_bounding_box` | falls inside an axis-aligned box defined by two corners |
| `geo_polygon` | falls inside an arbitrary polygon (ray-casting, point-in-polygon) |
Each operator takes a structured `value`:
```json theme={null}
// geo_radius — within 5 km of the Eiffel Tower
{ "field": "location", "operator": "geo_radius",
"value": { "center": { "lat": 48.8584, "lon": 2.2945 }, "radius": 5000 } }
// geo_bounding_box — top_left (NW) and bottom_right (SE) corners
{ "field": "location", "operator": "geo_bounding_box",
"value": { "top_left": { "lat": 49.0, "lon": 2.0 },
"bottom_right": { "lat": 48.0, "lon": 3.0 } } }
// geo_polygon — exterior ring of >= 3 points (auto-closed)
{ "field": "location", "operator": "geo_polygon",
"value": { "exterior": { "points": [
{ "lat": 48.0, "lon": 2.0 }, { "lat": 49.0, "lon": 2.0 },
{ "lat": 49.0, "lon": 3.0 }, { "lat": 48.0, "lon": 3.0 } ] } } }
```
<Note>
Distances use the haversine formula on a spherical earth (R = 6,371,000 m).
Bounding boxes handle the **antimeridian**: when `top_left.lon > bottom_right.lon`
the box is treated as wrapping across ±180°. Malformed geometry (out-of-range
`lat`/`lon`, a missing corner, or fewer than 3 polygon points) is **rejected at
request time** with a descriptive error; a document whose location field is
missing or unparseable is a non-match (it is never an error).
</Note>
## Lineage Shortcuts
Every Mixpeek document carries a `_internal.lineage` block recording where it
came from. To filter by lineage you don't have to use the underscore-prefixed
paths — use the friendly aliases below in any `field` position.
| Alias | Resolves to | Use for |
| ----------------- | ---------------------------------------- | ------------------------------------------------------- |
| `from_object` | `_internal.lineage.root_object_id` | "Everything derived from this bucket object" |
| `from_bucket` | `_internal.lineage.root_bucket_id` | "Everything derived from this bucket" |
| `from_document` | `_internal.lineage.source_document_id` | Direct children of one upstream document |
| `from_collection` | `_internal.lineage.source_collection_id` | Documents whose immediate parent was in this collection |
```json theme={null}
{
"AND": [
{ "field": "from_object", "operator": "eq", "value": "obj_video_123" }
]
}
```
You can mix lineage aliases with regular fields and templates:
```json theme={null}
{
"AND": [
{ "field": "from_object", "operator": "eq", "value": "{{INPUT.object_id}}" },
{ "field": "metadata.scene_score", "operator": "gte", "value": 0.8 }
]
}
```
The aliases are also accepted by document list endpoints and retriever filter
stages — the same vocabulary works everywhere `field` is used.
## Using Templates
Reference request inputs or stage outputs in filter values:
```json theme={null}
{
"AND": [
{ "field": "metadata.category", "operator": "eq", "value": "{{INPUT.category}}" },
{ "field": "metadata.price", "operator": "lte", "value": "{{INPUT.max_price}}" }
]
}
```
## Filter Stage Example
```json theme={null}
{
"stage_name": "filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"strategy": "structured",
"structured_filter": {
"AND": [
{ "field": "metadata.category", "operator": "eq", "value": "audio" },
{ "field": "metadata.price", "operator": "lte", "value": "{{INPUT.max_price}}" }
]
}
}
}
}
```
## Stage Pre-Filters and Post-Filters
Every stage accepts optional `pre_filters` and `post_filters` as siblings of
`parameters`. Use `pre_filters`. They narrow the candidate set **before** the
stage runs, pushed down into the vector store as native filters. Both take the
same logical-operator shape as any other filter.
<Warning>
**`post_filters` is accepted and never applied.** The field is declared on
every stage and passes validation, and no stage applies it. A `post_filters`
predicate returns the same documents as sending no filter at all, with HTTP
200 and no warning.
Measured on `feature_search`: a nonsense value returns the full unfiltered
set, identical to an unfiltered run. The identical predicate in `pre_filters`
filters correctly.
Put any predicate you rely on in `pre_filters`. Never use `post_filters` to
restrict scope, because it does not restrict anything.
</Warning>
**Canonical shape** — wrap conditions in an explicit logical operator:
```json theme={null}
{
"pre_filters": {
"AND": [
{ "field": "archive_status", "operator": "ne", "value": "ARCHIVED" },
{ "field": "Keywords", "operator": "contains", "value": "skincare" }
]
}
}
```
Always prefer the explicit `{ "AND": [ ... ] }` form — it is unambiguous and
nests cleanly with `OR`/`NOT`.
<Note>
For convenience, two shorthand forms are coerced to an `AND` group:
* a **single bare condition** — `{ "field": "...", "operator": "...", "value": "..." }` becomes `{ "AND": [ <condition> ] }`
* a **list of conditions** — `[ { ... }, { ... } ]` becomes `{ "AND": [ ... ] }`
Each condition must carry all three of `field`, `operator`, and `value`. An
**incomplete** condition (for example, a missing `operator`) is rejected with a
clear error rather than silently ignored. That guarantee covers the shape of a
condition, not where you place it. A well-formed condition placed in
`post_filters` still degrades into an unfiltered result, as described above.
</Note>
## Options
| Option | Default | Description |
| ---------------- | ------- | ---------------------------------------- |
| `case_sensitive` | `false` | Enable case-sensitive string comparisons |
```json theme={null}
{
"field": "metadata.title",
"operator": "contains",
"value": "AI",
"case_sensitive": true
}
```
# Interactions
Source: https://docs.mixpeek.com/docs/retrieval/interactions
Turn user behavior into better retrieval through automated feedback loops
Interactions capture how users engage with search results—clicks, purchases, skips, dwell time. This feedback automatically improves your retrievers through fine-tuning, personalization, and analytics.
<Frame>
<img alt="Mixpeek Interactions Flow" />
</Frame>
## Why Track Interactions
**Automatic Quality Improvement**: User behavior becomes training data for fine-tuning embedding models and ranking algorithms—no manual labeling required.
**Measure Real Performance**: Traditional IR metrics (precision, recall) use test sets. Interactions reveal actual user satisfaction through click-through rates, conversion, and engagement.
**Personalization at Scale**: Build user preference profiles from interaction history to deliver personalized results without managing separate recommendation systems.
**Identify Blind Spots**: Find queries with low engagement, results that users skip despite high rankings, and content gaps causing zero-result queries.
## What You Get
<CardGroup>
<Card title="Training Data Generation" icon="brain">
Interactions format into contrastive pairs (query, clicked\_doc) for embedding fine-tuning. Position is recorded for future bias correction.
</Card>
<Card title="Relevance Analytics" icon="chart-line">
CTR by position, documents with high skip rates, queries needing tuning, time-to-first-click—all without custom infrastructure.
</Card>
<Card title="Result Deduplication" icon="filter">
Exclude previously purchased, viewed, or consumed content automatically by filtering on past interactions.
</Card>
<Card title="Popularity Boosting" icon="arrow-trend-up">
Rerank results based on recent interaction signals (clicks, conversions) to surface trending content.
</Card>
</CardGroup>
## Capture Interactions
Record user behavior with a single API call:
```bash theme={null}
POST /v1/retrievers/interactions
{
"feature_id": "doc_xyz789", # Document ID from retriever results
"interaction_type": ["click"], # One or more signal types
"position": 3, # 0-based position in results (critical for bias correction)
"metadata": {
"duration_ms": 4500, # Optional: dwell time, device, query text
"query": "wireless earbuds"
},
"user_id": "user_123", # For personalization and session tracking
"session_id": "sess_456"
}
```
<Note>
**`interaction_type` is always a JSON array — even for a single signal** (`["click"]`, not `"click"`). The array captures the **co-occurring signals of one user action**, not several separate events: a single result that was shown, clicked, and dwelt on is `["view", "click", "long_view"]` in **one** request. Send a separate request per discrete user action (each click, each purchase) — don't batch unrelated events into one array.
</Note>
### Signal Types
| Type | Strength | Use Case |
| ------------------- | ----------------- | ---------------------------------------------- |
| `impression` | Neutral | Result was rendered on screen |
| `view` | Weak positive | User viewed a result |
| `click` | Moderate positive | User clicked result |
| `dwell` | Weak positive | User lingered on a result |
| `long_view` | Strong positive | Sustained engagement (track via `duration_ms`) |
| `purchase` | Strong positive | Conversion event |
| `add_to_cart` | Positive | Intent to purchase |
| `wishlist` | Positive | User added to wishlist |
| `positive_feedback` | Explicit positive | Thumbs up vote |
| `negative_feedback` | Explicit negative | Thumbs down vote |
| `share` | Strong positive | User shared the result |
| `bookmark` | Moderate positive | User saved for later |
| `query_refinement` | Neutral | User modified their search |
| `zero_results` | Neutral | Query yielded no results |
| `filter_toggle` | Neutral | User modified filters |
| `skip` | Weak negative | User ignored result |
| `return_to_results` | Moderate negative | User bounced back quickly |
<Tip>
Combine multiple types per event: `["click", "long_view"]` when a user clicks *and* stays engaged.
</Tip>
## Signal Strategy
Which signals to capture, when, and why. Not all signals carry equal weight.
### Signal Strength Matrix
This table maps each interaction type to how it's used across the platform:
| Signal Type | Strength | Fusion Learning | Personalization | Analytics | Fine-Tuning |
| ------------------- | ----------------- | :-----------------------: | :------------------: | :-----------------: | :-------------: |
| `click` | Moderate | Primary positive signal | Session history | CTR metrics | Positive pairs |
| `long_view` | Strong | High-weight positive | Preference indicator | Engagement depth | Strong positive |
| `purchase` | Strongest | Conversion signal | Purchase exclusion | Revenue attribution | Gold standard |
| `add_to_cart` | Strong | Intent signal | Cart-based recs | Funnel metrics | Near-positive |
| `positive_feedback` | Explicit | Direct reward | Profile building | Quality score | Ground truth |
| `negative_feedback` | Explicit | Direct penalty | Anti-preference | Issue detection | Hard negative |
| `skip` | Weak negative | Implicit penalty | — | Skip rate | Soft negative |
| `return_to_results` | Moderate negative | Bounce signal | — | Bounce rate | Negative pair |
| `share` | Strong | Social proof | Sharing patterns | Virality | Strong positive |
| `bookmark` | Moderate | Save intent | Saved items | Save rate | Positive |
| `dwell` | Variable | Duration-weighted | — | Engagement depth | Weighted |
| `query_refinement` | Neutral (0.0) | Not in default reward map | — | Query analysis | — |
### Use Case Patterns
<Tabs>
<Tab title="E-commerce">
**Primary signals:** `purchase`, `add_to_cart`, `click` — conversion is the strongest relevance indicator.
**Key patterns:**
* Track `add_to_cart` separately from `purchase` to measure funnel drop-off
* Use `position` for learning-to-rank bias correction
* Store `order_value` in metadata for revenue-weighted metrics
</Tab>
<Tab title="Media & Entertainment">
**Primary signals:** `long_view`, `click`, `share` — engagement depth matters more than the click.
**Key patterns:**
* Combine `click` and `long_view` in one event when engagement is sustained
* Use `duration_ms` and `completion_pct` to distinguish genuine views from bounces
* Track `share` as a strong positive signal for content quality
</Tab>
<Tab title="Enterprise Search">
**Primary signals:** `click`, `positive_feedback`, `bookmark` — clicks alone may indicate obligation, not satisfaction.
**Key patterns:**
* Add thumbs-up/down UI to capture explicit feedback
* Store `department` or `role` in metadata for demographic-level learned fusion
* Track `query_refinement` to find queries where results fall short
</Tab>
<Tab title="Recommendations">
**Primary signals:** `click`, `purchase`, `negative_feedback` — you need negative signals to avoid recommending disliked content.
**Key patterns:**
* Capture "not interested" / "hide" as `negative_feedback` to suppress similar content
* Use purchase history to filter already-bought items from results
* Track position carefully — recommendation position bias is stronger than search
</Tab>
</Tabs>
### Position Tracking
<Warning>
**Always capture `position`.** Although position-based reward weighting is not yet implemented, `position` is stored for analytics and will be used for future position-aware modeling. Recording it now avoids a costly backfill later.
</Warning>
Position bias is a well-known problem — users tend to click higher-ranked results regardless of relevance. The system records `position` on every interaction for analytics and audit purposes. However, position-based reward weighting is **not yet implemented** — a click at position 8 currently receives the same reward as a click at position 0. Record position now so you won't need to backfill when position-aware modeling is added.
### Cold Start
When you first deploy you have zero interactions. The system adapts automatically through the [hierarchical fallback](/docs/relevance/learned-fusion#hierarchical-fallback) in Thompson Sampling (personal → demographic → global → uniform prior):
| Interactions | Fusion Behavior | What To Do |
| ------------ | ---------------------------------------- | ------------------------------------------ |
| 0 | Falls back to `rrf` (uniform weights) | Ship with `rrf`, start collecting signals |
| 1–50 | Global-level learned weights only | Monitor analytics for obvious issues |
| 50–500 | Demographic-level personalization begins | Verify evaluation metrics are trending up |
| 500+ | Per-user personalization kicks in | Run benchmarks to compare against baseline |
### Client-Side Capture
Track clicks and dwell time directly from the browser, using `sendBeacon` so dwell events survive page unload:
```javascript theme={null}
// Track clicks with position
document.querySelectorAll('.search-result').forEach((el, index) => {
el.addEventListener('click', () => {
fetch('/api/interactions', {
method: 'POST',
body: JSON.stringify({
feature_id: el.dataset.documentId,
interaction_type: ['click'],
position: index,
metadata: { query: currentQuery },
user_id: userId, session_id: sessionId
})
});
});
});
// Track dwell time on result detail pages
let startTime = Date.now();
window.addEventListener('beforeunload', () => {
const dwellMs = Date.now() - startTime;
navigator.sendBeacon('/api/interactions', JSON.stringify({
feature_id: currentDocId,
interaction_type: dwellMs > 5000 ? ['click', 'long_view'] : ['click'],
position: resultPosition,
metadata: { duration_ms: dwellMs, query: originalQuery },
user_id: userId
}));
});
```
## Outcomes & Use Cases
### 1. Fine-Tune Embedding Models
```python theme={null}
# Collect 30 days of interaction data as training pairs
training_data = analytics.get_query_document_pairs(
date_range="last_30_days",
min_interactions=2 # Only docs with real engagement
)
# System formats as contrastive pairs:
# Positive: (query, clicked_doc)
# Hard negative: (query, high_ranked_but_skipped_doc)
# Position bias automatically corrected
fine_tuned_model = training.fine_tune(
base_model="text-embedding-3-small",
training_data=training_data
)
```
**Outcome**: 10-30% improvement in relevance metrics from domain-specific fine-tuning using real user preferences.
### 2. Identify & Fix Ranking Issues
```python theme={null}
# Find queries where top results are skipped
signals = analytics.get_retriever_signals(retriever_id="ret_123")
# Returns:
# - Documents ranked high but skipped (ranking issues)
# - Queries with <5% CTR (need tuning)
# - Average position of first click (relevance proxy)
```
**Outcome**: Data-driven decisions on which retrievers need adjustment, which taxonomy mappings to update, or which filters to refine.
### 3. Personalize Without Recommendation Infrastructure
```python theme={null}
# Exclude content user already consumed
previous_purchases = interactions.list(
user_id="user_123",
interaction_type=["purchase"],
days=90
)
retriever.execute(
query="new arrivals",
filters={
"document_id": {"$nin": [i.feature_id for i in previous_purchases]}
}
)
```
**Outcome**: Improved user experience (no duplicate suggestions) without building separate collaborative filtering or recommendation systems.
### 4. Continuously Tune Ranking from Interactions
Set `fusion: "learned"` and click/purchase signals automatically re-weight your search over time — no manual reranking rules. Weight higher-intent signals above clicks with the `reward_map`:
```json theme={null}
{
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 },
{ "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
],
"fusion": "learned",
"learning_config": {
"reward_map": { "click": 1.0, "add_to_cart": 2.0, "purchase": 3.0 },
"decay_window_days": 30
},
"final_top_k": 25
}
}
}
]
}
```
**Outcome**: click/purchase signals continuously re-tune feature weights via [auto-tune](/docs/retrieval/auto-tune) — higher-intent interactions carry more weight — with no manual reranking rules. See [Reward Signals](/docs/retrieval/reward-signals) for the full `reward_map`.
## Query & Export
Retrieve interactions for analysis or external ML pipelines:
```bash theme={null}
# Get all interactions for a document
POST /v1/retrievers/interactions/list
{
"feature_id": "doc_xyz789"
}
# Get user interaction history
POST /v1/retrievers/interactions/list
{
"user_id": "user_123",
"limit": 50
}
# Export for training (bulk pagination)
POST /v1/retrievers/interactions/list
{
"page": 1,
"page_size": 1000
}
```
## Reward Values and Auto-Tune
When a retriever uses [Auto-Tune](/docs/retrieval/auto-tune) (`fusion: "learned"`), every interaction automatically receives a computed `reward_value` stored in its metadata. This value is derived from the retriever's `reward_map` configuration at write time:
```json theme={null}
{
"interaction_type": ["click", "purchase"],
"metadata": {
"reward_value": 3.0,
"query": "wireless earbuds"
}
}
```
In this example, the reward is determined by the interaction type with the largest absolute value: `purchase (3.0)` wins over `click (1.0)`, so `reward_value` is `3.0` (not summed). The reward value determines how much this interaction shifts the fusion weights for the associated feature.
**How it feeds into Auto-Tune:**
1. Each interaction records which feature URI surfaced the clicked result
2. The `reward_value` adjusts the Thompson Sampling Beta distribution for that feature
3. Positive rewards increase the feature's weight; negative rewards decrease it
4. The next search for this user reflects the updated weights
See [Reward Signals](/docs/retrieval/reward-signals) for the full default reward map, customization options, and examples for different use cases (e-commerce, content, internal search).
### Feature URI
The `feature_uri` field identifies which embedding model/feature produced the result the user interacted with. This is **required for Auto-Tune** — without it, the interaction cannot contribute to weight learning because the system doesn't know which feature to reward.
```json theme={null}
{
"feature_id": "doc_product_789",
"interaction_type": ["click"],
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"user_id": "user_456"
}
```
You can find the `feature_uri` for each result in the execution response's `learned_fusion_context.feature_uris` array, or extract it from the document's metadata. If `feature_uri` is set as a top-level field, it is automatically promoted into `metadata.feature_uri` for backward compatibility.
<Tip>
The `session_id` field enables within-session adaptation. When provided, clicks in the current session influence the next search immediately via a session cache — without waiting for the ClickHouse write-then-read cycle.
</Tip>
## Privacy & Compliance
**GDPR Deletion**: Remove all interactions for a user to honor right-to-deletion requests.
```bash theme={null}
DELETE /v1/retrievers/interactions/<interaction_id>
```
**Anonymization**: Hash `user_id` before sending if you need consistent tracking without PII.
## Python SDK
<CodeGroup>
```python Create Interaction theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="sk_...")
# After executing a retriever search
results = client.retrievers.execute(
"ret_abc123",
inputs={
"query": "wireless headphones",
"user_id": "user_456",
"session_id": "sess_789",
},
)
# Record a click on the 3rd result
client.retrievers.create_interaction(
feature_id=results["documents"][2]["feature_id"],
interaction_type=["click"],
position=2,
user_id="user_456",
session_id="sess_789",
retriever_id="ret_abc123"
)
```
```python Manage Interactions theme={null}
# List interactions for a retriever
interactions = client.retrievers.list_interactions("ret_abc123", page=1, page_size=50)
# Get a specific interaction
interaction = client.retrievers.get_interaction("int_abc123")
# Delete an interaction (GDPR right-to-deletion)
client.retrievers.delete_interaction("int_abc123")
```
</CodeGroup>
## Bulk Backfill
Migrate existing interaction history into Mixpeek using the batch endpoint. Send up to 1,000 interactions per call with `occurred_at` timestamps so temporal decay weights them by their true age.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/interactions/batch" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"interactions": [
{
"feature_id": "doc_001",
"interaction_type": ["click"],
"position": 0,
"user_id": "user_123",
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"occurred_at": "2026-01-15T10:30:00Z"
},
{
"feature_id": "doc_002",
"interaction_type": ["purchase"],
"position": 2,
"user_id": "user_123",
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"occurred_at": "2026-01-16T14:00:00Z"
}
]
}'
```
```python Python SDK theme={null}
client.retrievers.backfill_interactions([
{
"feature_id": "doc_001",
"interaction_type": ["click"],
"position": 0,
"user_id": "user_123",
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"occurred_at": "2026-01-15T10:30:00Z",
},
{
"feature_id": "doc_002",
"interaction_type": ["purchase"],
"position": 2,
"user_id": "user_123",
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"occurred_at": "2026-01-16T14:00:00Z",
},
])
```
</CodeGroup>
The response includes per-item results: `{created: 2, failed: 0, results: [{index: 0, interaction_id: "...", status: "created"}, ...]}`. Backfilled interactions bypass the real-time session cache — they affect learned fusion weights only after the next ClickHouse aggregation.
## Best Practices
1. **Always capture position** -- recorded for analytics and future position-aware modeling
2. **Store query text in metadata** -- enables query-document pair export for fine-tuning
3. **Track dwell time** -- separates genuine engagement (`long_view`) from accidental clicks
4. **Backfill with timestamps** -- use the [batch endpoint](#bulk-backfill) with `occurred_at` so temporal decay weights by true age
5. **Monitor weekly** -- set up dashboards to track CTR trends and catch relevance regressions early
<Info>
See [Improve Relevance → Analytics](/docs/platform/improve-relevance#analytics) for dashboards and alerting on interaction-driven metrics like CTR, engagement rates, and query quality.
</Info>
# Lineage Traversal
Source: https://docs.mixpeek.com/docs/retrieval/lineage-traversal
Walk a document's lineage chain — its source, provenance, and evidence trail from the original object through every transformation — efficiently, without N+1 round-trips.
Every Mixpeek document carries the full lineage chain that produced it,
from the original bucket object through every transformation. This guide
shows the four patterns for navigating that chain efficiently from the API.
<Note>
Background on the lineage data model: see [Documents → Lineage](/docs/processing/extractors/document).
The TL;DR is that each document has `_internal.lineage` with `root_object_id`,
`root_bucket_id`, `source_document_id`, and a `chain` array recording every
processing step.
</Note>
## When to use what
| Pattern | Use case | Round-trips |
| --------------------------------- | -------------------------------------------------------- | ---------------- |
| `?expand=parent` | "Show me this scene and its source frame on one page" | 1 |
| `?expand=root_object` | "Show me this document with the original video metadata" | 1 |
| `?expand=ancestors` | "Show me the full pipeline that produced this document" | 1 |
| `?expand=children` | "Show me all the segments derived from this scene" | 1 |
| `GET /documents/{id}/ancestors` | Same as `expand=ancestors` but returns only the chain | 1 |
| `GET /documents/{id}/descendants` | Same as `expand=children` but returns only the children | 1 |
| `from_object` filter | "Search across everything derived from this video" | 1 (no GET first) |
The shared rule: never use a list response to grab IDs and then issue per-document
GETs. The `expand` parameter takes a comma-separated list, so a single request
fetches the document plus everything you need from its lineage tree.
## \$expand keywords
Lineage-aware `$expand` keywords resolve relative to a document's own
`_internal.lineage` block. They land under `_expanded.<keyword>` in the response,
matching the existing user-field expand shape.
<Tabs>
<Tab title="parent">
The single document referenced by `_internal.lineage.source_document_id`.
```bash cURL theme={null}
curl "$API/v1/collections/$COL/documents/$DOC?expand=parent" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NS"
```
```python Python theme={null}
client.documents.get(
collection_identifier="col_scenes",
document_id="doc_scene_42",
expand="parent",
)
# Response includes:
# response._expanded.parent — the upstream frame document
```
For a tier-0 document (created directly from a bucket object), `parent` is
absent — there's no upstream document. Use `root_object` instead.
</Tab>
<Tab title="root_object">
The bucket object that started the lineage tree, fetched from the bucket
objects collection (not [MVS](/docs/vector-store/overview)).
```bash theme={null}
curl "$API/v1/collections/$COL/documents/$DOC?expand=root_object"
```
Useful for "show me this document with its source video filename and
upload metadata" without a separate `GET /v1/buckets/.../objects/{id}`.
</Tab>
<Tab title="ancestors">
Every prior step in the chain, in order from the root through the
immediate parent. The document itself is excluded from the list.
```bash theme={null}
curl "$API/v1/collections/$COL/documents/$DOC?expand=ancestors"
```
For a document at tier 3, `ancestors` returns 3 elements: the tier-0,
tier-1, and tier-2 documents along the lineage path. Steps that have
no `document_id` (e.g., the bucket-object source step at tier 0) are
skipped — those don't correspond to fetchable documents.
</Tab>
<Tab title="children">
Direct downstream documents (depth=1) — every document whose
`_internal.lineage.source_document_id` equals this document's ID.
Capped at **100 children per request**. If you need deeper traversal,
combine with the `from_object` filter (see below).
```bash theme={null}
curl "$API/v1/collections/$COL/documents/$DOC?expand=children"
```
</Tab>
</Tabs>
You can request multiple keywords in one call by comma-separating them:
```bash theme={null}
curl "$API/v1/collections/$COL/documents/$DOC?expand=parent,root_object,children"
```
The same `expand` is accepted by `POST /documents/list` (in the request body)
and by retriever response shaping — the document GET endpoint is just the
simplest demonstration.
## Convenience endpoints
For SDKs and UIs that want only the lineage walk without fetching the
document itself, use the dedicated endpoints:
```bash theme={null}
# Returns the chain root → parent (excludes the document itself)
GET /v1/collections/{collection_identifier}/documents/{document_id}/ancestors
# Returns direct depth=1 children (max 100)
GET /v1/collections/{collection_identifier}/documents/{document_id}/descendants
```
Both endpoints return a `List<DocumentResponse>` with the same shape as
`GET /documents/{id}` per element.
## Filter aliases
When you want to *search* the lineage tree (find every document derived
from one root), use the filter aliases instead of expand. They work
in document list endpoints, retriever filter stages, and aggregations.
| Alias | Resolves to | Use for |
| ----------------- | ---------------------------------------- | ------------------------------------------------------- |
| `from_object` | `_internal.lineage.root_object_id` | Everything derived from this bucket object |
| `from_bucket` | `_internal.lineage.root_bucket_id` | Everything derived from this bucket |
| `from_document` | `_internal.lineage.source_document_id` | Direct children of one upstream document |
| `from_collection` | `_internal.lineage.source_collection_id` | Documents whose immediate parent was in this collection |
```json theme={null}
// "Show me all scene documents in col_scenes that came from this video"
{
"AND": [
{ "field": "from_object", "operator": "eq", "value": "obj_video_123" }
]
}
```
```json theme={null}
// "Direct children of one specific frame document"
{
"AND": [
{ "field": "from_document", "operator": "eq", "value": "doc_frame_42" }
]
}
```
These aliases are equivalent to the underscore-prefixed paths
(`_internal.lineage.*`) — they exist purely so you don't have to learn
the internal schema. Mix them freely with normal user fields:
```json theme={null}
{
"AND": [
{ "field": "from_object", "operator": "eq", "value": "obj_video_123" },
{ "field": "metadata.scene_score", "operator": "gte", "value": 0.8 }
]
}
```
## End-to-end example: decomposition tree
To render a decomposition tree for one bucket object — every document at
every tier that descended from it — make one filtered list call per
collection in the namespace using `from_object`. The result is
already structured by collection, and each document's `_internal.lineage.chain`
tells you where to draw the edges.
```python theme={null}
def decomposition_tree(client, namespace, root_object_id):
namespaces = {}
for collection in client.collections.list(namespace=namespace):
docs = client.documents.list(
collection.collection_id,
filters={
"AND": [
{"field": "from_object", "operator": "eq", "value": root_object_id}
]
},
)
if docs:
namespaces[collection.collection_id] = docs
return namespaces
```
For a deeper materialized view (the chain edges with parent/child resolved
inline), use the dedicated decomposition tree endpoint:
```bash theme={null}
GET /v1/buckets/{bucket_id}/objects/{object_id}/decomposition-tree
```
That endpoint pre-joins everything in one call and is what the Studio
namespace detail page uses to draw lineage diagrams.
## Limits & caveats
* **Maximum 50 unique user-field references per `expand` request** —
doesn't apply to lineage keywords (those are bounded by chain length
for `ancestors` and by the children cap for `children`).
* **`expand=children` is capped at 100 children per parent.** For deeper
traversal or wider fan-out, fall back to a `from_document` filter.
* **Recursive expansion is not supported** — `expand=parent` resolves one
level. To walk further, use `expand=ancestors` (full chain) or call
`/ancestors` then re-`expand` from there.
* **Lineage is immutable provenance.** If an ancestor is deleted, its
`document_id` reference in the chain remains. The `ancestors` expand
silently skips unresolved references — never returns `null` slots — but
client code should still be ready for shorter-than-expected chains.
# Multi-Stage Retrieval
Source: https://docs.mixpeek.com/docs/retrieval/multi-stage-deep-dive
The composable pipeline architecture that makes Mixpeek a warehouse, not a database
<Frame>
<img alt="Multi-stage retrieval: query flows through filter (WHERE), sort (ORDER BY), reduce (LIMIT), enrich (JOIN), apply (SELECT) stages to produce ranked results" />
</Frame>
Single-query search returns a flat list of results ranked by one signal. That works for simple lookups. It falls apart the moment you need to combine signals, cross-reference collections, reshape output, or enforce business logic at query time. Multi-stage retrieval solves this by turning your search into a composable pipeline: a sequence of typed stages that filter, sort, reduce, enrich, and transform results in a single deterministic execution.
<Note>
This is the definitive guide to multi-stage retrieval. For ready-to-copy pipeline configs, see the [Retrieval Cookbook](/docs/retrieval/cookbook). For the full stage catalog and parameter schemas, see [Retrievers](/docs/retrieval/retrievers).
</Note>
<Warning>
Several pipelines on this page use a `score_linear` stage to blend multiple score
fields with weights. That stage is not implemented. Sending a pipeline containing
it returns HTTP 400.
We left the examples in place because the pipelines around them are accurate and
worth reading, and because swapping in a stage that merely looks plausible would
be harder to spot. Treat every `score_linear` block as a sketch of intent.
There is no weighted multi-field blend today, and no combination of existing
stages produces one. The closest you can get is `score_normalize` to bring each
score into a common range, then `sort_attribute` on the single field that matters
most — you pick one signal, you do not blend them. The live stage list is at
[Retrievers](/docs/retrieval/retrievers).
</Warning>
## Why Single-Query Search Isn't Enough
Traditional search systems give you one query, one index, one ranked list. This creates three problems that compound as your data grows:
**1. Signal collapse.** You want to find content that matches a face *and* contains a specific logo *and* has negative sentiment. A single vector query can only encode one of these signals. You end up running three separate queries and stitching results together in application code.
**2. N+1 enrichment.** After retrieving results, you need to join them with metadata from another collection, call an external API for licensing info, or classify each result against a taxonomy. Without pipeline-level enrichment, every result triggers a separate round-trip from your application.
**3. Brittle application logic.** Filtering, ranking, deduplication, and reshaping all live in your application layer. Every new use case means new glue code. Every change to ranking logic means a redeploy.
Multi-stage retrieval moves all of this into the retriever definition itself --- a declarative pipeline that the engine executes in a single pass.
## The SQL Analogy
If you know SQL, you already understand multi-stage retrieval. Each stage type maps to a SQL clause:
| Stage Type | SQL Equivalent | What It Does |
| ---------- | ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| **filter** | `WHERE` | Narrow the document set based on conditions --- semantic similarity, metadata predicates, feature thresholds |
| **sort** | `ORDER BY` | Reorder documents by score, attribute, or cross-encoder reranking |
| **reduce** | `LIMIT` / `GROUP BY` | Collapse results --- top-k sampling, deduplication, aggregation, summarization |
| **enrich** | `JOIN` | Add data from other collections, LLM-generated fields, or taxonomy classifications |
| **apply** | `SELECT` / `TRANSFORM` | Reshape output, call external APIs, execute custom code, run web searches |
A SQL query like:
```sql theme={null}
SELECT t.title, t.risk_score, r.license_type
FROM media_library t
JOIN rights_database r ON t.asset_id = r.asset_id
WHERE similarity(t.face_embedding, @query) > 0.72
AND similarity(t.logo_embedding, @brand) > 0.6
ORDER BY t.risk_score DESC
LIMIT 10
```
Becomes a retriever pipeline:
```json theme={null}
{
"stages": [
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.face_embedding}}", "top_k": 100, "min_score": 0.72}], "final_top_k": 100, "fusion": "rrf"}}},
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.brand_embedding}}", "top_k": 100, "min_score": 0.6}], "final_top_k": 100, "fusion": "rrf"}}},
{"stage_name": "score_linear", "stage_type": "sort", "config": {"stage_id": "score_linear", "parameters": {"weights": {"risk_score": 1.0}}}},
{"stage_name": "document_enrich", "stage_type": "enrich", "config": {"stage_id": "document_enrich", "parameters": {"target_collection_id": "col_rights", "source_field": "asset_id", "target_field": "asset_id"}}},
{"stage_name": "sample", "stage_type": "reduce", "config": {"stage_id": "sample", "parameters": {"count": 10}}}
]
}
```
The difference: this pipeline works over multimodal embeddings, not just relational columns. You can filter on face vectors, sort by sentiment scores, and enrich with LLM-generated classifications --- all in one execution.
## The Five Stage Types
### Filter --- Narrow the Candidate Set
Filter stages reduce the number of documents flowing through the pipeline. They are the `WHERE` clause of your retrieval query. Every pipeline starts with at least one filter.
**Use filter stages to:**
* Run semantic similarity search against any extracted feature
* Apply metadata predicates (equality, range, set membership)
* Chain multiple filters for compound conditions (face match AND logo match AND date range)
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
# Semantic search filter: find faces matching a reference
face_filter = {
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.face_embedding}}",
"top_k": 100,
"min_score": 0.72
}
],
"final_top_k": 100,
"fusion": "rrf"
}
}
}
# Metadata filter: restrict to a date range
date_filter = {
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "published_date", "operator": "gte", "value": "2025-01-01"},
{"field": "status", "operator": "eq", "value": "published"}
]
}
}
}
}
# Chain them: both conditions must pass
retriever = client.retrievers.create(
name="filtered-face-search",
stages=[face_filter, date_filter]
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "filtered-face-search",
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.face_embedding}}",
"top_k": 100,
"min_score": 0.72
}
],
"final_top_k": 100,
"fusion": "rrf"
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "published_date", "operator": "gte", "value": "2025-01-01"},
{"field": "status", "operator": "eq", "value": "published"}
]
}
}
}
}
]
}'
```
</CodeGroup>
<Tip>
Chain multiple filter stages to express AND logic. Each successive filter operates on the output of the previous one, progressively narrowing the candidate set.
</Tip>
### Sort --- Control Ranking
Sort stages reorder the document set without adding or removing documents. They are the `ORDER BY` clause. Place them after filters to control which results appear first.
**Use sort stages to:**
* Apply weighted linear scoring across multiple signals
* Rerank results with a cross-encoder model for higher precision
* Sort by a metadata attribute (date, price, popularity)
<CodeGroup>
```python Python theme={null}
# Weighted linear scoring across three signals
sort_stage = {
"stage_name": "score_linear",
"stage_type": "sort",
"config": {
"stage_id": "score_linear",
"parameters": {
"weights": {
"audio.sentiment": 0.6,
"recency": 0.3,
"engagement": 0.1
}
}
}
}
# Cross-encoder reranking for maximum precision
rerank_stage = {
"stage_name": "cross_encoder_rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query_text}}"
}
}
}
```
```bash cURL theme={null}
# Weighted linear scoring
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "scored-results",
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.query}}", "top_k": 100}], "final_top_k": 100, "fusion": "rrf"}
}
},
{
"stage_name": "score_linear",
"stage_type": "sort",
"config": {
"stage_id": "score_linear",
"parameters": {
"weights": {
"audio.sentiment": 0.6,
"recency": 0.3,
"engagement": 0.1
}
}
}
}
]
}'
```
</CodeGroup>
### Reduce --- Collapse and Limit
Reduce stages collapse the result set. They are the `LIMIT`, `GROUP BY`, and `DISTINCT` clauses. Use them to control result count, remove duplicates, or aggregate values.
**Use reduce stages to:**
* Sample the top-k results after sorting
* Deduplicate by a field (e.g., one result per source URL)
* Summarize results into an aggregated output
<CodeGroup>
```python Python theme={null}
# Top-k sampling: keep the 10 highest-ranked results
sampling_stage = {
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {
"count": 10
}
}
}
# Deduplication: one result per source domain
dedup_stage = {
"stage_name": "dedup",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {
"strategy": "field",
"fields": ["metadata.source_url"]
}
}
}
# Combine: deduplicate first, then take top 10
retriever = client.retrievers.create(
name="deduped-top-10",
stages=[
{"stage_name": "feature_search", "stage_type": "filter",
"config": {"stage_id": "feature_search",
"parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.image}}", "top_k": 100}], "final_top_k": 100, "fusion": "rrf"}}},
dedup_stage,
sampling_stage
]
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "deduped-top-10",
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.image}}", "top_k": 100}], "final_top_k": 100, "fusion": "rrf"}
}
},
{
"stage_name": "dedup",
"stage_type": "reduce",
"config": {
"stage_id": "deduplicate",
"parameters": {"strategy": "field", "fields": ["metadata.source_url"]}
}
},
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {"count": 10}
}
}
]
}'
```
</CodeGroup>
### Enrich --- Join External Knowledge
Enrich stages add data to each document without changing the result set size. They are the `JOIN` clause. Use them to attach metadata from other collections, generate LLM-powered annotations, or classify documents against taxonomies.
**Use enrich stages to:**
* Cross-collection joins (product data + catalog info + pricing)
* LLM enrichment (generate summaries, extract entities, assess risk)
* Taxonomy classification (label documents against a controlled vocabulary)
<CodeGroup>
```python Python theme={null}
# Cross-collection join: attach rights/licensing data
rights_enrich = {
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_rights_database",
"source_field": "metadata.asset_id",
"target_field": "asset_id",
"fields_to_merge": ["license_type", "expiry_date", "rights_holder"],
"output_field": "rights_info"
}
}
}
# LLM enrichment: generate a risk assessment for each result
llm_enrich = {
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"prompt": "Assess the IP risk level (low/medium/high) for this content based on the match confidence score {{DOC.score}} and rights status {{DOC.rights_info.license_type}}. Return a JSON object with 'risk_level' and 'reasoning' fields.",
"output_field": "risk_assessment",
"model": "gpt-4o-mini"
}
}
}
# Taxonomy classification: label by content category
taxonomy_enrich = {
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "tax_content_categories",
"top_k": 5,
"min_score": 0.5
}
}
}
```
```bash cURL theme={null}
# Cross-collection join
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "enriched-search",
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.audio}}", "top_k": 100, "min_score": 0.8}], "final_top_k": 100, "fusion": "rrf"}
}
},
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_rights_database",
"source_field": "metadata.asset_id",
"target_field": "asset_id",
"fields_to_merge": ["license_type", "expiry_date", "rights_holder"],
"output_field": "rights_info"
}
}
}
]
}'
```
</CodeGroup>
<Note>
Enrich stages execute per-document but are batched internally. A `document_enrich` join resolves all lookups in a single batch query to the target collection, not one query per document.
</Note>
### Apply --- Transform and Reshape
Apply stages transform the structure or content of each document. They are the `SELECT` and function-call layer of your pipeline. Use them to reshape output for downstream consumers, call external APIs, execute custom code, or search the web.
**Use apply stages to:**
* Reshape JSON output with Jinja2 templates
* Call external APIs (Stripe, Salesforce, internal services)
* Execute custom Python/TypeScript/JavaScript in sandboxed environments
* Run web searches to augment results with external context
<CodeGroup>
```python Python theme={null}
# JSON transform: reshape output for a frontend
json_transform = {
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": '{"id": "{{DOC.document_id}}", "title": "{{DOC.metadata.title}}", "risk": "{{DOC.risk_assessment.risk_level}}", "thumbnail": "{{DOC.metadata.thumbnail_url}}"}',
"fail_on_error": False
}
}
}
# External API call: check licensing status
api_call = {
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://licensing.internal/v1/check/{{DOC.metadata.asset_id}}",
"method": "GET",
"allowed_domains": ["licensing.internal"],
"auth": {
"type": "bearer",
"secret_ref": "licensing_api_key"
},
"output_field": "metadata.license_check",
"on_error": "skip"
}
}
}
# Custom code execution: compute a composite score
code_exec = {
"stage_name": "code_execution",
"stage_type": "apply",
"config": {
"stage_id": "code_execution",
"parameters": {
"language": "python",
"code": "output = {'composite_score': doc['score'] * 0.7 + doc.get('metadata', {}).get('popularity', 0) * 0.3}",
"output_field": "computed"
}
}
}
```
```bash cURL theme={null}
# JSON transform
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "transformed-output",
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.query}}", "top_k": 100}], "final_top_k": 100, "fusion": "rrf"}
}
},
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"id\": \"{{DOC.document_id}}\", \"title\": \"{{DOC.metadata.title}}\"}",
"fail_on_error": false
}
}
}
]
}'
```
</CodeGroup>
## Building Multi-Stage Pipelines
The power of multi-stage retrieval is in composition. Here are three production pipelines that demonstrate how stages chain together to solve complex problems that no single query can address.
### Pipeline 1: Brand Safety Scanner
**Problem:** A media company needs to find scenes where their talent appears near competitor products in negative-sentiment content --- before the content goes live.
**Pipeline logic:** Find faces matching talent roster, then check for competitor logos in the same scenes, rank by sentiment risk, take the worst offenders, and attach brand safety context.
<CodeGroup>
```python Python theme={null}
retriever = client.retrievers.create(
name="brand-safety-scanner",
namespace="media-library",
stages=[
# Stage 1: Find scenes containing talent faces
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.talent_embedding}}",
"top_k": 500,
"min_score": 0.72
}
],
"final_top_k": 500,
"fusion": "rrf"
}
}
},
# Stage 2: Narrow to scenes that also contain competitor logos
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.competitor_logo_embedding}}",
"top_k": 500,
"min_score": 0.65
}
],
"final_top_k": 500,
"fusion": "rrf"
}
}
},
# Stage 3: Rank by weighted risk (sentiment + recency + engagement)
{
"stage_name": "score_linear",
"stage_type": "sort",
"config": {
"stage_id": "score_linear",
"parameters": {
"weights": {
"audio.sentiment": 0.6,
"recency": 0.3,
"engagement": 0.1
}
}
}
},
# Stage 4: Take the 10 highest-risk scenes
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {"count": 10}
}
},
# Stage 5: Attach brand safety scores from reference collection
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_brand_safety_scores",
"source_field": "metadata.content_id",
"target_field": "content_id",
"fields_to_merge": ["safety_rating", "advertiser_category", "risk_flags"],
"output_field": "brand_context"
}
}
}
]
)
# Execute the pipeline
results = client.retrievers.execute(
retriever_id=retriever.id,
inputs={
"talent_embedding": celebrity_face_vector,
"competitor_logo_embedding": competitor_logo_vector
}
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "brand-safety-scanner",
"stages": [
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.talent_embedding}}", "top_k": 500, "min_score": 0.72}], "final_top_k": 500, "fusion": "rrf"}}},
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.competitor_logo_embedding}}", "top_k": 500, "min_score": 0.65}], "final_top_k": 500, "fusion": "rrf"}}},
{"stage_name": "score_linear", "stage_type": "sort", "config": {"stage_id": "score_linear", "parameters": {"weights": {"audio.sentiment": 0.6, "recency": 0.3, "engagement": 0.1}}}},
{"stage_name": "sample", "stage_type": "reduce", "config": {"stage_id": "sample", "parameters": {"count": 10}}},
{"stage_name": "document_enrich", "stage_type": "enrich", "config": {"stage_id": "document_enrich", "parameters": {"target_collection_id": "col_brand_safety_scores", "source_field": "metadata.content_id", "target_field": "content_id", "fields_to_merge": ["safety_rating", "advertiser_category", "risk_flags"], "output_field": "brand_context"}}}
]
}'
```
</CodeGroup>
**Stage flow:** 500 face matches --> \~50 with competitor logos --> sorted by risk --> top 10 --> enriched with brand context
***
### Pipeline 2: IP Clearance Pipeline
**Problem:** Before publishing new content, a legal team needs to check it against a database of copyrighted material across audio fingerprints, visual similarity, and metadata --- then attach licensing information for review.
**Pipeline logic:** Match audio fingerprints, check visual similarity for the same assets, filter by rights status, sort by match confidence, and attach the full licensing record.
<CodeGroup>
```python Python theme={null}
retriever = client.retrievers.create(
name="ip-clearance-pipeline",
namespace="rights-catalog",
stages=[
# Stage 1: Audio fingerprint matching against known works
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.audio_fingerprint}}",
"top_k": 200,
"min_score": 0.8
}
],
"final_top_k": 200,
"fusion": "rrf"
}
}
},
# Stage 2: Visual similarity check on the same content
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.visual_frames}}",
"top_k": 200,
"min_score": 0.7
}
],
"final_top_k": 200,
"fusion": "rrf"
}
}
},
# Stage 3: Exclude already-licensed content
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{"field": "license_status", "operator": "ne", "value": "cleared"}
]
}
}
}
},
# Stage 4: Rank by match confidence weighted with rights severity
{
"stage_name": "score_linear",
"stage_type": "sort",
"config": {
"stage_id": "score_linear",
"parameters": {
"weights": {
"match_confidence": 0.8,
"rights_severity": 0.2
}
}
}
},
# Stage 5: Attach full licensing records for legal review
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "col_licensing_records",
"source_field": "metadata.rights_id",
"target_field": "rights_id",
"fields_to_merge": ["rights_holder", "license_type", "territory", "expiry_date", "contact_email"],
"output_field": "licensing"
}
}
},
# Stage 6: LLM-generated risk summary for each match
{
"stage_name": "llm_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "llm_enrich",
"parameters": {
"prompt": "Based on the match confidence ({{DOC.score}}) and licensing status ({{DOC.licensing.license_type}}), provide a one-sentence risk assessment and recommended action (clear/review/block).",
"output_field": "legal_summary",
"model": "gpt-4o-mini"
}
}
}
]
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "ip-clearance-pipeline",
"stages": [
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.audio_fingerprint}}", "top_k": 200, "min_score": 0.8}], "final_top_k": 200, "fusion": "rrf"}}},
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.visual_frames}}", "top_k": 200, "min_score": 0.7}], "final_top_k": 200, "fusion": "rrf"}}},
{"stage_name": "attribute_filter", "stage_type": "filter", "config": {"stage_id": "attribute_filter", "parameters": {"conditions": {"AND": [{"field": "license_status", "operator": "ne", "value": "cleared"}]}}}},
{"stage_name": "score_linear", "stage_type": "sort", "config": {"stage_id": "score_linear", "parameters": {"weights": {"match_confidence": 0.8, "rights_severity": 0.2}}}},
{"stage_name": "document_enrich", "stage_type": "enrich", "config": {"stage_id": "document_enrich", "parameters": {"target_collection_id": "col_licensing_records", "source_field": "metadata.rights_id", "target_field": "rights_id", "fields_to_merge": ["rights_holder", "license_type", "territory", "expiry_date", "contact_email"], "output_field": "licensing"}}},
{"stage_name": "llm_enrich", "stage_type": "enrich", "config": {"stage_id": "llm_enrich", "parameters": {"prompt": "Based on the match confidence ({{DOC.score}}) and licensing status ({{DOC.licensing.license_type}}), provide a one-sentence risk assessment and recommended action (clear/review/block).", "output_field": "legal_summary", "model": "gpt-4o-mini"}}}
]
}'
```
</CodeGroup>
**Stage flow:** 200 audio matches --> \~30 with visual matches --> exclude cleared --> sorted by risk --> licensing data attached --> LLM risk summary generated
***
### Pipeline 3: Content Moderation
**Problem:** A platform needs to scan user-uploaded content across multiple safety dimensions (NSFW, violence, toxicity), aggregate risk scores, and route flagged content to a moderation queue.
**Pipeline logic:** Filter for NSFW content above threshold, check text toxicity, sort by combined risk, take the worst offenders, classify against a moderation taxonomy, and push to the review queue.
<CodeGroup>
```python Python theme={null}
retriever = client.retrievers.create(
name="content-moderation",
namespace="user-uploads",
stages=[
# Stage 1: Flag visually unsafe content
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.unsafe_reference}}",
"top_k": 1000,
"min_score": 0.6
}
],
"final_top_k": 1000,
"fusion": "rrf"
}
}
},
# Stage 2: Check text-based toxicity in the same content
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": {"input_mode": "text", "value": "{{INPUT.toxic_terms}}"},
"top_k": 1000,
"min_score": 0.5
}
],
"final_top_k": 1000,
"fusion": "rrf"
}
}
},
# Stage 3: Aggregate risk signals into a combined score
{
"stage_name": "score_linear",
"stage_type": "sort",
"config": {
"stage_id": "score_linear",
"parameters": {
"weights": {
"nsfw_score": 0.4,
"violence_score": 0.3,
"toxicity_score": 0.3
}
}
}
},
# Stage 4: Take the top 50 highest-risk items
{
"stage_name": "sample",
"stage_type": "reduce",
"config": {
"stage_id": "sample",
"parameters": {"count": 50}
}
},
# Stage 5: Classify against moderation taxonomy
{
"stage_name": "taxonomy_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "taxonomy_enrich",
"parameters": {
"taxonomy_id": "tax_moderation_categories",
"top_k": 5,
"min_score": 0.5
}
}
},
# Stage 6: Push to external moderation queue
{
"stage_name": "api_call",
"stage_type": "apply",
"config": {
"stage_id": "api_call",
"parameters": {
"url": "https://moderation.internal/v1/review-queue",
"method": "POST",
"allowed_domains": ["moderation.internal"],
"auth": {
"type": "bearer",
"secret_ref": "moderation_api_key"
},
"output_field": "metadata.review_ticket",
"on_error": "skip"
}
}
}
]
)
```
```bash cURL theme={null}
curl -X POST https://api.mixpeek.com/v1/retrievers \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "content-moderation",
"stages": [
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding", "query": "{{INPUT.unsafe_reference}}", "top_k": 1000, "min_score": 0.6}], "final_top_k": 1000, "fusion": "rrf"}}},
{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.toxic_terms}}"}, "top_k": 1000, "min_score": 0.5}], "final_top_k": 1000, "fusion": "rrf"}}},
{"stage_name": "score_linear", "stage_type": "sort", "config": {"stage_id": "score_linear", "parameters": {"weights": {"nsfw_score": 0.4, "violence_score": 0.3, "toxicity_score": 0.3}}}},
{"stage_name": "sample", "stage_type": "reduce", "config": {"stage_id": "sample", "parameters": {"count": 50}}},
{"stage_name": "taxonomy_enrich", "stage_type": "enrich", "config": {"stage_id": "taxonomy_enrich", "parameters": {"taxonomy_id": "tax_moderation_categories", "top_k": 5, "min_score": 0.5}}},
{"stage_name": "api_call", "stage_type": "apply", "config": {"stage_id": "api_call", "parameters": {"url": "https://moderation.internal/v1/review-queue", "method": "POST", "allowed_domains": ["moderation.internal"], "auth": {"type": "bearer", "secret_ref": "moderation_api_key"}, "output_field": "metadata.review_ticket", "on_error": "skip"}}}
]
}'
```
</CodeGroup>
**Stage flow:** 1000 NSFW candidates --> \~200 also toxic --> sorted by combined risk --> top 50 --> taxonomy labels attached --> pushed to moderation queue
## Performance Characteristics
Multi-stage pipelines avoid the N+1 problem that plagues application-level orchestration. Here is how:
**1. Filter stages execute server-side against indexes.** A `feature_search` filter runs directly against the MVS vector index. No data leaves the engine until the candidate set is narrowed. Chaining two filter stages does not mean two round-trips from your application --- both execute within the engine in sequence.
**2. Enrich stages batch internally.** A `document_enrich` join across 50 results resolves in a single batch query to the target collection, not 50 separate lookups. LLM enrichment stages batch prompts where possible.
**3. Reduce stages shrink the working set early.** Place a `sample` or `deduplicate` stage as early as possible to minimize the number of documents flowing through expensive downstream stages (LLM enrichment, API calls).
**4. The pipeline streams, not materializes.** Documents flow through stages incrementally. A 6-stage pipeline does not create 6 intermediate copies of the full result set. Each stage processes and passes documents forward.
<Warning>
Stage ordering matters for performance. Place cheap, high-selectivity filters first (metadata filters, feature searches with high thresholds) and expensive stages last (LLM enrichment, external API calls). A pipeline that enriches 1000 documents and then filters to 10 is dramatically slower than one that filters to 10 and then enriches.
</Warning>
## When to Use Which Stage Type
Use this decision guide when designing your pipeline:
<AccordionGroup>
<Accordion title="I need to narrow down results based on content or metadata">
Use a **filter** stage. Start with `feature_search` for semantic/vector-based filtering, or `attribute_filter` for structured attribute filtering. Chain multiple filters for compound conditions.
</Accordion>
<Accordion title="I need to reorder results by relevance or business logic">
Use a **sort** stage. Choose `rerank` for high-precision reranking with a cross-encoder model, `sort_relevance` to order by search relevance score, or `sort_attribute` for simple field-based ordering.
</Accordion>
<Accordion title="I need fewer results, or deduplicated results">
Use a **reduce** stage. Choose `limit` for top-k limits, `sample` for random sampling, `deduplicate` for deduplication by field, or `summarize` for LLM-powered aggregation of results into a single summary.
</Accordion>
<Accordion title="I need to add data from another collection, an LLM, or a taxonomy">
Use an **enrich** stage. Choose `document_enrich` for cross-collection joins, `llm_enrich` for AI-generated fields, or `taxonomy_enrich` for classification against a controlled vocabulary.
</Accordion>
<Accordion title="I need to reshape output, call an API, or run custom logic">
Use an **apply** stage. Choose `json_transform` for output reshaping, `api_call` for external service integration, `code_execution` for custom Python/TypeScript/JavaScript, or `external_web_search` for web augmentation.
</Accordion>
</AccordionGroup>
### Stage Ordering Rules of Thumb
1. **Filter first.** Every pipeline should start with one or more filter stages to narrow the candidate set.
2. **Sort second.** Apply ranking after filtering so you are sorting a smaller set.
3. **Reduce third.** Cut the result set to a manageable size before enrichment.
4. **Enrich fourth.** Add external data only to the documents that survived filtering, sorting, and reduction.
5. **Apply last.** Reshape output and trigger side effects at the end of the pipeline.
<Note>
These are guidelines, not hard rules. Some pipelines benefit from enriching before sorting (e.g., sort by a field that only exists after enrichment). Design your pipeline around your data flow, not a rigid template.
</Note>
## Related Resources
<CardGroup>
<Card title="Retrievers" icon="magnifying-glass" href="/retrieval/retrievers">
Full stage catalog, parameter schemas, and retriever configuration reference
</Card>
<Card title="Retrieval Cookbook" icon="book" href="/retrieval/cookbook">
Ready-to-copy pipeline configurations for common use cases
</Card>
<Card title="Stage Reference" icon="layer-group" href="/retrieval/retrievers#stage-catalog">
Detailed documentation for every stage type and stage ID
</Card>
<Card title="Caching" icon="bolt" href="/best-practices/caching-strategies">
Configure retriever-level caching for repeated queries
</Card>
</CardGroup>
# Query Optimization & Explain
Source: https://docs.mixpeek.com/docs/retrieval/query-optimization
How Mixpeek automatically optimizes retriever pipelines, and how to inspect the execution plan with explain
Mixpeek optimizes every retriever before it runs — reordering, fusing, and pushing work down into the vector store — then lets you inspect exactly what it did with the **explain** endpoint. You write the pipeline you find readable; the optimizer makes it fast.
## Automatic optimizations
When you execute a retriever, the planner rewrites your stage list before execution. These transformations are automatic — you don't configure them:
| Optimization | What it does | Why it helps |
| ------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| **Filter push-down** | Moves attribute filters ahead of vector search | Shrinks the candidate set before the expensive embedding search runs |
| **Stage fusion** | Merges adjacent compatible stages into one | Fewer passes over the result set |
| **Grouping optimization** | Rewrites group/reduce stages to run database-side | Avoids materializing intermediate results |
| **Computation push-down** | Runs data-plane stages (`feature_search`, `attribute_filter`, `sort_attribute`, `aggregate`) **inside MVS** | Eliminates a network round-trip and lets the vector store filter/sort where the data lives |
| **Parallel sub-queries** | Runs independent operations (search + count, search + facet) concurrently | Lower wall-clock latency |
| **Over-fetch hints** | Fetches extra candidates when a later stage will filter them out | Preserves recall after post-filtering |
<Tip>
Because the optimizer pushes filters down for you, **write filters wherever they read most clearly** — you don't need to hand-order stages for performance. Use `explain` to confirm what was pushed.
</Tip>
The retriever is also fetched and optimized **once** per request, then reused across a batch — so `POST /v1/retrievers/{id}/execute/batch` amortizes planning across all queries.
## Inspect the plan with explain
`POST /v1/retrievers/{retriever_id}/explain` returns the **optimized** execution plan without running the query — per-stage cost and latency estimates, bottlenecks, and exactly which optimizations were applied. Pass hypothetical `inputs` to see how the plan changes with different parameters.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/explain" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "inputs": { "query": "people discussing electric vehicles" } }'
```
```json Example response theme={null}
{
"retriever_id": "ret_abc123",
"execution_plan": [
{
"stage_index": 0,
"stage_name": "attribute_filter",
"stage_type": "filter",
"estimated_input": 10000,
"estimated_output": 5000,
"estimated_efficiency": 0.5,
"estimated_cost_credits": 0.01,
"estimated_duration_ms": 20,
"cache_likely": true,
"optimization_notes": ["Pushed down from stage 2"],
"warnings": []
},
{
"stage_index": 1,
"stage_name": "feature_search",
"stage_type": "filter",
"estimated_input": 5000,
"estimated_output": 100,
"estimated_efficiency": 0.02,
"estimated_cost_credits": 0.5,
"estimated_duration_ms": 200,
"cache_likely": false,
"optimization_notes": [],
"warnings": ["High cost stage - consider reducing top_k"]
}
],
"estimated_cost": { "total_credits": 0.51, "total_duration_ms": 220 },
"bottleneck_stages": ["feature_search"],
"optimization_applied": true,
"optimization_details": {
"original_stage_count": 3,
"optimized_stage_count": 2,
"stage_reduction_pct": 33.3,
"decisions": [
{
"rule_type": "push_down_filters",
"applied": true,
"reason": "Moved attribute_filter before feature_search to reduce search scope"
}
]
},
"optimization_suggestions": [
{ "type": "reduce_limit", "stage": "feature_search", "message": "Consider reducing top_k to improve latency" }
]
}
```
<Note>
`estimated_cost_credits` and `total_credits` are legacy fields expressed in the internal ledger unit (1 credit = \$0.001). Customer-facing pricing is in dollars — see [Billing](/docs/platform/billing).
</Note>
### How to read it
| Field | Use it to… |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `execution_plan[].estimated_input/output` | See how each stage narrows the set — a stage that barely reduces the set may be unnecessary |
| `estimated_efficiency` | Spot low-selectivity stages (close to 1.0 = passes almost everything through) |
| `estimated_cost_credits` / `estimated_duration_ms` | Budget in dollars before running (1 credit = \$0.001); find the expensive stage |
| `bottleneck_stages` | The stages dominating latency — optimize these first |
| `cache_likely` | Whether a stage will likely hit the [cache](/docs/resources/best-practices) |
| `optimization_details.decisions` | Exactly which automatic rewrites fired (and why) |
| `optimization_suggestions` | Concrete, actionable tuning hints |
| `warnings` | Per-stage red flags (e.g. high-cost stage, overly broad `top_k`) |
<Note>
The `execution_plan` reflects the **optimized** pipeline, not your original stage list. Compare `optimization_details.original_stage_count` vs `optimized_stage_count` to see how much the planner collapsed.
</Note>
### Execution-plan variant
`POST /v1/retrievers/{retriever_id}/execute/explain` returns the same plan in a MongoDB-`explain`-style shape if you prefer that format. Both are read-only and never execute the query.
## Typical workflow
<Steps>
<Step title="Explain before you ship">
Run `explain` with representative inputs to see estimated cost, bottlenecks, and applied optimizations.
</Step>
<Step title="Act on bottlenecks & suggestions">
Reduce `top_k` on high-cost searches, add a selective `attribute_filter` (the optimizer pushes it down), or drop low-selectivity stages.
</Step>
<Step title="Verify in production">
Use [retriever analytics](/docs/platform/operations) to confirm real latency and cache-hit rates match the estimate.
</Step>
</Steps>
## Related
* [Multi-Stage Retrieval](/docs/retrieval/multi-stage-deep-dive) — how stages compose
* [Feature Search](/docs/retrieval/stages/feature-search) — the most common bottleneck stage
* [Evaluations](/docs/retrieval/evaluations) — measure quality alongside cost
* [Best Practices](/docs/resources/best-practices) — caching and cost optimization
# Deep Research Patterns
Source: https://docs.mixpeek.com/docs/retrieval/research
Compose multi-stage retrievers for investigations, literature reviews, and analysis
Deep research workflows orchestrate multiple retriever executions, enrichment passes, and synthesis steps to answer complex questions. Mixpeek's stage catalog—filter, sort, reduce, apply, enrich—gives you the primitives to build these flows without bespoke infrastructure.
## Building Blocks
| Stage | Examples | Use in Research |
| ------------------- | -------------------------------------------------- | ---------------------------------------------------------------------------- |
| Search (filter) | `feature_search`, `query_expand` | Gather candidate documents across modalities; expand queries for recall |
| Narrow (filter) | `attribute_filter`, `llm_filter` | Restrict to relevant time ranges, entities, or sentiment |
| Web (apply) | `external_web_search`, `web_scrape` | Pull public sources from the open web |
| Enrich | `document_enrich`, `taxonomy_enrich`, `llm_enrich` | Attach related docs, taxonomy tags, or LLM-extracted facts |
| Synthesize (reduce) | `summarize` | Aggregate results into a single synthesized brief with citations |
| Compose (apply) | `api_call`, `cross_compare` | Call external services (e.g., fact-check APIs) or compare across collections |
## Common Patterns
### Literature Review
1. **Seed search** using `feature_search` to retrieve recent papers across text and figures.
2. **Narrow** by publication date and venue with `attribute_filter`.
3. **Classify** by research area with `taxonomy_enrich`.
4. **Synthesize** findings with citations using `summarize`.
5. Store summaries alongside `feature_id` references for auditability.
### Competitive Intelligence
1. Use `external_web_search` + `web_scrape` stages to pull public announcements.
2. Join with internal product docs via `document_enrich` to compare specs.
3. Apply an `attribute_filter` to spotlight price or feature gaps.
4. Generate a briefing memo with the `summarize` stage.
### Incident Investigation
1. Collect relevant runbooks/logs via `feature_search` over internal collections.
2. Use `attribute_filter` to isolate the incident window.
3. Enrich with taxonomy-based tags (`taxonomy_enrich`) for impacted systems.
4. Summarize timeline and root cause via `summarize`, keeping citations.
## Orchestrating Multi-Retriever Flows
Use the `document_enrich` stage to call a sub-retriever based on previous stage output:
```json theme={null}
{
"stage_name": "retriever",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"retriever_id": "ret_internal_logs",
"input_mappings": {
"query_text": "{{INPUT.primary_question}}",
"time_range": "{{STAGE.filter.time_range}}"
},
"merge_strategy": "append"
}
}
}
```
This pattern lets you create macro retrievers that orchestrate domain-specific sub-searches, enabling modular reuse.
## Capturing Feedback
* Record user signals with the [Interactions API](/docs/retrieval/interactions) (`click`, `long_view`, `positive_feedback`, etc.).
* Feed interactions back into [auto-tuning](/docs/retrieval/auto-tune) or `attribute_filter` stages ("hide documents seen in this session").
* Use [Observability](/docs/operations/observability) to optimize parameter choices (e.g., increase a `feature_search` stage's `final_top_k` if users often tap beyond the top 10).
## Operational Tips
1. **Persist execution IDs** – each `execute` response includes an execution id; link it to your research session for audit trails.
2. **Monitor stage telemetry** – `stage_statistics` identifies bottlenecks (e.g., LLM stages dominating latency).
3. **Budget controls** – set `budget_limits` on retrievers to cap time or spend for exploratory workflows.
4. **Cache intermediate results** – cache expensive discovery steps, especially when analysts reiterate queries.
5. **Use tasks** – schedule enrichment batches (clusters, taxonomies) ahead of time so research pipelines stay low-latency.
## Suggested Architecture
```
Orchestration App
├─ Calls macro retriever (with document_enrich compose stages)
├─ Logs execution IDs + user prompts
├─ Stores generated summaries & citations
└─ Sends interactions back to Mixpeek
```
Behind the scenes, Mixpeek handles stage execution, caching, and lineage tracking. You focus on stitching together the right stages and presenting the synthesized output.
## Next Steps
* Review [Retrievers](/docs/retrieval/retrievers) for stage configuration details.
* Learn how [Filters](/docs/retrieval/filters) and [Taxonomies](/docs/enrichment/taxonomies) contribute structure to exploratory pipelines.
* Use [Operations → Observability](/docs/operations/observability) to monitor research workloads in production.
# Reward Signals
Source: https://docs.mixpeek.com/docs/retrieval/reward-signals
Configure how different interaction types influence learned fusion weights
When a user interacts with a search result, that interaction carries a **reward value** that adjusts the learned fusion weights. A purchase is a stronger signal than a click; negative feedback is a penalty. The reward map controls these magnitudes.
## Default Reward Map
If you do not provide a custom `reward_map` in `learning_config`, the system uses these defaults:
| Interaction Type | Default Reward | Signal Strength | Description |
| ------------------- | :------------: | ----------------- | ------------------------------------------------------ |
| `impression` | `0.0` | Neutral | Result was rendered on screen (passive signal) |
| `view` | `0.0` | Neutral | User viewed a result (not in default reward map) |
| `click` | `1.0` | Moderate positive | User clicked a result |
| `dwell` | `0.5` | Weak positive | User lingered on a result |
| `long_view` | `1.0` | Moderate positive | Sustained engagement (dwell time > 30s) |
| `purchase` | `3.0` | Strong positive | Conversion event |
| `add_to_cart` | `2.0` | Positive | Intent to purchase |
| `wishlist` | `0.0` | Neutral | User added to wishlist (not in default reward map) |
| `bookmark` | `1.5` | Positive | User saved for later |
| `share` | `1.5` | Positive | User shared the result |
| `positive_feedback` | `2.0` | Strong positive | Explicit thumbs up |
| `negative_feedback` | `-2.0` | Strong negative | Explicit thumbs down |
| `query_refinement` | `0.0` | Neutral | User modified their search (not in default reward map) |
| `zero_results` | `0.0` | Neutral | Query yielded no results (not in default reward map) |
| `filter_toggle` | `0.0` | Neutral | User modified filters (not in default reward map) |
| `skip` | `-0.5` | Weak negative | Result was shown but ignored |
| `return_to_results` | `-0.5` | Weak negative | User bounced back quickly |
Interaction types not listed in the reward map contribute a reward of `0.0` — they are recorded but do not influence fusion weights.
## Custom Reward Maps
Override the defaults by setting `reward_map` in `learning_config`:
```json theme={null}
{
"fusion": "learned",
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {
"click": 1.0,
"purchase": 5.0,
"add_to_cart": 2.5,
"positive_feedback": 3.0,
"negative_feedback": -3.0,
"skip": -1.0
}
}
}
```
<Info>
When you provide a custom `reward_map`, it **replaces** the defaults entirely. Only interaction types present in your map will influence fusion weights. Include every type you want to count.
</Info>
The reward value is computed at interaction-write time and stored as `reward_value` in the interaction metadata. When an interaction has multiple types, the reward with the largest absolute value is used (not summed). For example, `['click', 'purchase']` yields `3.0` (the purchase reward), not `4.0`. This means changing the `reward_map` only affects future interactions — previously recorded interactions retain their original reward values.
## Negative Signals
Negative rewards (`negative_feedback`, `skip`, `return_to_results`) penalize the feature that surfaced the result. Mechanically, a negative reward increments the Beta distribution's `beta` parameter, making it less likely that the associated feature receives high weight in future queries:
```
positive reward → alpha += reward → feature weight trends up
negative reward → beta += abs(reward) → feature weight trends down
```
<Warning>
Negative signals should generally have smaller absolute values than positive signals. A single `negative_feedback: -5.0` would outweigh five `click: 1.0` interactions, which can cause rapid weight swings. Start conservative and tune based on evaluation results.
</Warning>
## Position Bias
Results shown at position 0 get clicked more often than results at position 10, regardless of relevance. This is **position bias** — a well-known problem in learning-to-rank systems.
Auto-Tune records the `position` field on every interaction for analytics and audit purposes. However, the current reward computation does **not** weight interactions by position — a click at position 8 receives the same reward value as a click at position 0. Position bias correction is a planned enhancement but is not yet implemented.
<Tip>
Include `position` when posting interactions. Although position does not currently affect reward computation, it is stored alongside each interaction for future position-aware modeling and for your own analytics. Recording it now means you won't need to backfill when position-based weighting is added.
</Tip>
## Temporal Decay
User preferences change over time. Auto-Tune applies exponential decay to older interactions so that recent behavior matters more:
```
effective_reward = reward * (decay_factor ^ days_ago)
```
| `decay_factor` | After 30 days | After 90 days | After 180 days | After 365 days |
| :--------------: | :-----------: | :-----------: | :------------: | :------------: |
| `1.0` (no decay) | 100% | 100% | 100% | 100% |
| `0.999` | 97% | 91% | 84% | 69% |
| `0.995` | 86% | 64% | 41% | 16% |
| `0.990` | 74% | 41% | 17% | 3% |
| `0.980` | 55% | 16% | 3% | \~0% |
Configure via `learning_config`:
```json theme={null}
{
"learning_config": {
"decay_factor": 0.995,
"decay_window_days": 365
}
}
```
Interactions older than `decay_window_days` are ignored entirely (not just decayed to near-zero, but excluded from the aggregation query).
### Backfilling historical interactions
By default the server timestamps each interaction at the moment it's recorded. If you're **migrating existing click/purchase logs** into Mixpeek, pass `occurred_at` (ISO 8601) so temporal decay weights each interaction by its *true* age instead of treating everything as brand-new:
```json theme={null}
POST /v1/retrievers/interactions
{
"feature_id": "doc_123",
"interaction_type": ["purchase"],
"position": 0,
"user_id": "user_42",
"feature_uri": "mixpeek://text_extractor@v1/...",
"occurred_at": "2026-01-15T10:30:00Z"
}
```
```python Python SDK theme={null}
client.retrievers.create_interaction(
feature_id="doc_123",
interaction_type=["purchase"],
user_id="user_42",
feature_uri="mixpeek://text_extractor@v1/...",
occurred_at="2026-01-15T10:30:00Z", # backfill with the real timestamp
)
```
<Note>
Omit `occurred_at` for live interactions — the server stamps "now". A naive datetime is interpreted as UTC, and a future value is clamped to now. Backfilled events (with `occurred_at`) bypass the real-time within-session cache so historical data can't pollute live, in-session adaptation.
</Note>
For large histories, **send interactions in bulk** (1–1000 per call) instead of one request each:
```python Python SDK theme={null}
result = client.retrievers.backfill_interactions([
{"feature_id": "d1", "interaction_type": ["purchase"], "user_id": "u1",
"feature_uri": "mixpeek://text_extractor@v1/...",
"occurred_at": "2026-01-15T10:30:00Z"},
{"feature_id": "d2", "interaction_type": ["click"], "user_id": "u2",
"feature_uri": "mixpeek://text_extractor@v1/...",
"occurred_at": "2026-01-16T11:00:00Z"},
# ... up to 1000 per call
])
# -> {"created": 2, "failed": 0, "errors": [], "results": [{"index": 0, "interaction_id": "int_abc123", "status": "created"}, ...]}
```
```bash REST theme={null}
POST /v1/retrievers/interactions/batch
{ "interactions": [ { "feature_id": "d1", "interaction_type": ["purchase"], "occurred_at": "2026-01-15T10:30:00Z" }, ... ] }
```
Each row is enriched (reward value, feature/context promotion) exactly like a single create; one bad row doesn't sink the batch (it's reported in `errors`). The response now also includes per-item `results` with assigned `interaction_id`s.
<Note>
The private API uses `feature_id` while published (public) retrievers use `document_id` — they refer to the same field. Use whichever matches the endpoint you're calling.
</Note>
## Examples
### E-commerce: purchases matter most
```json theme={null}
{
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {
"purchase": 5.0,
"add_to_cart": 2.0,
"click": 1.0,
"bookmark": 1.5,
"negative_feedback": -2.0,
"return_to_results": -1.0
},
"decay_factor": 0.995,
"min_interactions": 3
}
}
```
A user who purchases products found via text search will see their text feature weight increase faster than a user who only clicks.
### Content platform: engagement over clicks
```json theme={null}
{
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {
"long_view": 2.0,
"share": 3.0,
"bookmark": 1.5,
"click": 0.5,
"skip": -1.5
},
"decay_factor": 0.990,
"min_interactions": 5
}
}
```
Clicks are downweighted relative to deep engagement (long views, shares). Skips are penalized more heavily — a result shown but ignored is a stronger negative signal than a mere absence of clicks.
### Internal search: explicit feedback only
```json theme={null}
{
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {
"positive_feedback": 3.0,
"negative_feedback": -3.0
},
"decay_factor": 1.0,
"min_interactions": 10
}
}
```
Only explicit thumbs up/down influence weights. Clicks and views are ignored. No temporal decay — in an internal tool, preferences tend to be stable. Higher `min_interactions` threshold because explicit feedback is sparse.
## Related
* [Auto-Tune](/docs/retrieval/auto-tune) — overview of the full feedback loop
* [Interactions](/docs/retrieval/interactions) — how to capture user behavior
* [Rollout & Safety](/docs/retrieval/auto-tune-rollout) — safely deploying learned fusion
# Agentic Enrich
Source: https://docs.mixpeek.com/docs/retrieval/stages/agentic-enrich
Classify documents using a multi-turn reasoning agent with tool access
<Frame>
<img alt="Agentic Enrich stage showing multi-turn reasoning agent with tool access for document classification" />
</Frame>
The Agentic Enrich stage uses a multi-turn reasoning agent (default: Claude) that can call tools — taxonomy lookup, example search, and content analysis — to produce high-quality structured classifications for each document.
<Note>
**Stage Category**: ENRICH (Enriches documents)
**Transformation**: N documents → N documents (with agent-produced classification added)
</Note>
## When to Use
| Use Case | Description |
| --------------------------------- | ----------------------------------------------------------- |
| **Complex classification** | Ambiguous categories requiring multi-step reasoning |
| **Multimodal analysis** | Video/image content needing perceptual analysis + reasoning |
| **Taxonomy-aware classification** | Agent looks up taxonomy definitions before deciding |
| **Few-shot classification** | Agent queries already-classified examples for reference |
## When NOT to Use
| Scenario | Recommended Alternative |
| ---------------------------------- | ------------------------------- |
| Simple single-shot extraction | `llm_enrich` (faster, cheaper) |
| Vector-based taxonomy matching | `taxonomy_enrich` (no LLM cost) |
| High-throughput batch processing | `llm_enrich` with batch API |
| Deterministic field transformation | `json_transform` |
## Parameters
| Parameter | Type | Default | Description |
| ------------------------ | ------- | ---------------------------- | ----------------------------------------------------------------------------------------------------- |
| `system_prompt` | string | *Required* | System prompt for the reasoning agent. Supports `{{INPUT.*}}`, `{{DOC.*}}`, `{{CONTEXT.*}}` templates |
| `output_schema` | object | *Required* | JSON schema for the structured output the agent must produce |
| `output_field` | string | `metadata.classification` | Dot-path where classification is stored on each document |
| `provider` | string | `anthropic` | LLM provider for the reasoning agent |
| `model_name` | string | `claude-sonnet-4-5-20250929` | Model for the reasoning agent |
| `api_key` | string | `null` | BYOK API key. Supports `{{secrets.*}}` |
| `taxonomy_id` | string | `null` | Taxonomy to load via `get_taxonomy_categories` tool |
| `example_collection_ids` | array | `null` | Collections to search for classified examples |
| `analysis_provider` | object | Google/Gemini | Secondary LLM config for `analyze_content` tool |
| `enabled_tools` | array | `null` | Explicit tool list. Auto-detected when null |
| `max_turns` | integer | `8` | Maximum agent reasoning turns (1-20) |
| `timeout_seconds` | float | `60.0` | Max wall-clock seconds per document (5-300) |
| `temperature` | float | `0.0` | Sampling temperature for the agent |
| `when` | object | `null` | Conditional filter — only enrich matching documents |
| `max_concurrency` | integer | `2` | Parallel agent loops (1-5) |
## Available Tools
The agent has access to three tools, auto-enabled based on configuration:
| Tool | Enabled When | Description |
| ------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `get_taxonomy_categories` | `taxonomy_id` is set | Loads full taxonomy definition (categories, hierarchy, descriptions) from the database |
| `query_examples` | `example_collection_ids` is set | Vector search against already-classified collections, optionally filtered by category label |
| `analyze_content` | Always available | Delegates to a secondary LLM (default: Gemini) for specialized content analysis (video, image, audio) |
## Configuration Examples
<CodeGroup>
```json Video Classification (Claude + Gemini) theme={null}
{
"stage_name": "agentic_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "agentic_enrich",
"parameters": {
"provider": "anthropic",
"model_name": "claude-sonnet-4-5-20250929",
"system_prompt": "You are an expert IAB content classifier. Use the available tools to: 1) Load the IAB taxonomy categories, 2) Analyze the video content with the analyze_content tool, 3) Query for similar already-classified examples. Make your classification decision with detailed reasoning.",
"output_schema": {
"type": "object",
"properties": {
"iab_tier1": {"type": "string"},
"iab_tier2": {"type": "string"},
"confidence": {"type": "number"},
"reasoning": {"type": "string"}
},
"required": ["iab_tier1", "confidence", "reasoning"]
},
"output_field": "iab_classification",
"taxonomy_id": "tax_iab_content",
"example_collection_ids": ["col_classified_videos"],
"analysis_provider": {
"provider": "google",
"model_name": "gemini-2.5-flash"
},
"max_turns": 10,
"when": {"field": "_internal.modality", "operator": "eq", "value": "video"}
}
}
}
```
```json Simple Text Classification theme={null}
{
"stage_name": "agentic_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "agentic_enrich",
"parameters": {
"system_prompt": "Analyze the document and classify it into a technology category with confidence score and reasoning.",
"output_schema": {
"type": "object",
"properties": {
"category": {"type": "string"},
"confidence": {"type": "number"},
"reasoning": {"type": "string"}
},
"required": ["category", "confidence", "reasoning"]
},
"output_field": "metadata.classification",
"max_turns": 3,
"timeout_seconds": 30
}
}
}
```
```json Taxonomy-Aware with Examples theme={null}
{
"stage_name": "agentic_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "agentic_enrich",
"parameters": {
"system_prompt": "You are a product classifier. First load the taxonomy to see available categories, then search for similar already-classified products. Use those references to classify this product accurately.",
"output_schema": {
"type": "object",
"properties": {
"category_id": {"type": "string"},
"category_name": {"type": "string"},
"confidence": {"type": "number"},
"similar_products": {"type": "array", "items": {"type": "string"}}
},
"required": ["category_id", "category_name", "confidence"]
},
"output_field": "product_classification",
"taxonomy_id": "tax_product_categories",
"example_collection_ids": ["col_classified_products"],
"max_turns": 8
}
}
}
```
```json Conditional Enrichment theme={null}
{
"stage_name": "agentic_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "agentic_enrich",
"parameters": {
"system_prompt": "Classify this image content by subject matter and artistic style.",
"output_schema": {
"type": "object",
"properties": {
"subject": {"type": "string"},
"style": {"type": "string"},
"confidence": {"type": "number"}
}
},
"output_field": "image_classification",
"max_turns": 5,
"when": {"field": "_internal.modality", "operator": "eq", "value": "image"}
}
}
}
```
</CodeGroup>
## How It Works
For each document, the stage runs a multi-turn agent loop:
1. **Initialize**: Agent receives the document content + system prompt + available tools
2. **Reason**: Agent analyzes the document and optionally calls tools (taxonomy lookup, example search, content analysis)
3. **Observe**: Tool results are fed back to the agent as context
4. **Iterate**: Loop continues until the agent produces a final answer (or `max_turns`/`timeout_seconds` is reached)
5. **Output**: The agent's structured JSON response is merged into the document at `output_field`
## Output Examples
### With Taxonomy + Examples
```json theme={null}
{
"document_id": "doc_abc123",
"content": "A product review video discussing...",
"iab_classification": {
"iab_tier1": "Technology & Computing",
"iab_tier2": "Consumer Electronics",
"confidence": 0.92,
"reasoning": "The video discusses smartphone features and pricing. Taxonomy lookup confirmed 'Consumer Electronics' under 'Technology & Computing'. Similar classified videos (col_classified_videos) showed consistent T&C categorization for product review content."
}
}
```
### Simple Classification
```json theme={null}
{
"document_id": "doc_def456",
"content": "Introduction to machine learning algorithms...",
"metadata": {
"classification": {
"category": "Artificial Intelligence",
"confidence": 0.88,
"reasoning": "Document covers supervised and unsupervised learning methods, neural network architectures, and model evaluation."
}
}
}
```
### Conditional Skip (When Condition)
```json theme={null}
{
"document_id": "doc_ghi789",
"_internal": {"modality": "text"},
"content": "Plain text document...",
"iab_classification": null
}
```
Documents that don't match the `when` condition are passed through unchanged.
## Performance
| Metric | Value |
| ----------------- | ----------------------------------------------- |
| **Latency** | 2-30s per document (depends on turns and tools) |
| **LLM calls** | 3-15 per document |
| **Max documents** | 10 per execution |
| **Parallel** | Up to `max_concurrency` (default 2) |
<Warning>
Agentic enrichment makes multiple LLM calls per document. Use the `when` condition to limit which documents are processed, and keep `max_turns` low for simple tasks.
</Warning>
## Common Pipeline Patterns
### Search + Agentic Classify + Filter
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 20
}
],
"final_top_k": 20
}
}
},
{
"stage_name": "agentic_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "agentic_enrich",
"parameters": {
"system_prompt": "Classify this content by IAB category using the taxonomy and examples.",
"output_schema": {
"type": "object",
"properties": {
"iab_category": {"type": "string"},
"confidence": {"type": "number"}
}
},
"output_field": "classification",
"taxonomy_id": "tax_iab",
"example_collection_ids": ["col_labeled"],
"max_turns": 6
}
}
},
{
"stage_name": "attribute_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "classification.confidence",
"operator": "gte",
"value": 0.8
}
}
}
]
```
### Multimodal Analysis Pipeline
```json theme={null}
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 10
}
],
"final_top_k": 10
}
}
},
{
"stage_name": "agentic_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "agentic_enrich",
"parameters": {
"system_prompt": "Use the analyze_content tool to examine this media, then classify by topic and sentiment.",
"output_schema": {
"type": "object",
"properties": {
"topic": {"type": "string"},
"sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
"summary": {"type": "string"}
}
},
"output_field": "media_analysis",
"analysis_provider": {
"provider": "google",
"model_name": "gemini-2.5-flash"
},
"max_turns": 5
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"query": "{{INPUT.query}}",
"document_field": "content"
}
}
}
]
```
## Bring Your Own Key (BYOK)
Use your own LLM API keys instead of Mixpeek's default keys for both the reasoning agent and the analysis provider.
<Steps>
<Step title="Store your API keys as secrets">
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/organizations/secrets" \
-H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"secret_name": "anthropic_api_key",
"secret_value": "sk-ant-..."
}'
```
</Step>
<Step title="Reference secrets in your stage config">
```json theme={null}
{
"stage_name": "agentic_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "agentic_enrich",
"parameters": {
"provider": "anthropic",
"model_name": "claude-sonnet-4-5-20250929",
"api_key": "{{secrets.anthropic_api_key}}",
"system_prompt": "Classify this content.",
"output_schema": {"type": "object", "properties": {"category": {"type": "string"}}},
"analysis_provider": {
"provider": "google",
"model_name": "gemini-2.5-flash",
"api_key": "{{secrets.google_api_key}}"
}
}
}
}
```
</Step>
</Steps>
<Note>
When `api_key` is not specified, the stage uses Mixpeek's default API keys and usage is charged to your Mixpeek account.
</Note>
## Stage Metadata
The stage returns execution metadata for observability:
| Field | Description |
| --------------------- | ---------------------------------------------------- |
| `documents_enriched` | Number of documents processed by the agent |
| `documents_skipped` | Number of documents skipped (when condition) |
| `total_cost` | Total LLM API cost across all documents |
| `total_tokens_input` | Total input tokens consumed |
| `total_tokens_output` | Total output tokens generated |
| `reasoning_traces` | Per-document traces with tool calls and turn history |
| `conditional` | Whether a `when` condition was applied |
## Error Handling
| Error | Behavior |
| ---------------------- | --------------------------------------------------------------- |
| Agent timeout | Returns best result so far, or null |
| Max turns reached | Loop ends; latest structured output used |
| Schema validation fail | Raw text stored in `output_field` |
| Tool execution error | Error message returned to agent; it can retry or skip |
| Missing system\_prompt | Stage fails with validation error |
| Invalid taxonomy\_id | `get_taxonomy_categories` tool returns error to agent |
| Empty content | Agent receives empty document; classification based on metadata |
| Invalid API key | Error returned with auth failure |
## Cost Considerations
| Setting | Cost Impact |
| --------------------- | --------------------------------------------- |
| `max_turns: 3` | Low — simple direct classification |
| `max_turns: 10` | Medium — multi-tool research workflow |
| `max_concurrency: 1` | Sequential, slower but controlled cost |
| `when` condition | Skip documents that don't need classification |
| `timeout_seconds: 30` | Cap per-document spend |
<Tip>
Start with `max_turns: 3` and increase only if the agent consistently needs more iterations. Most straightforward classifications finish in 2-4 turns.
</Tip>
## Related
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) — Single-shot LLM enrichment (faster, cheaper)
* [Taxonomy Enrich](/docs/retrieval/stages/taxonomy-enrich) — Vector-based taxonomy matching (no LLM cost)
* [Agent Search](/docs/retrieval/stages/agent-search) — Multi-turn agent for search (different use case)
# Document Enrich
Source: https://docs.mixpeek.com/docs/retrieval/stages/document-enrich
Join documents across collections for cross-reference enrichment
<Frame>
<img alt="Document Enrich stage showing collection joins and cross-reference lookups" />
</Frame>
The Document Enrich stage performs collection joins by looking up related documents from other Mixpeek collections. This enables cross-reference enrichment without external database calls.
<Note>
**Stage Category**: ENRICH (Enriches documents)
**Transformation**: N documents → N documents (with joined data added)
</Note>
## When to Use
| Use Case | Description |
| -------------------------- | ------------------------------------------- |
| **Cross-collection joins** | Link products to reviews, users to profiles |
| **Reference resolution** | Expand foreign keys to full documents |
| **Denormalization** | Flatten related data for display |
| **Multi-index search** | Combine results from different collections |
## When NOT to Use
| Scenario | Recommended Alternative |
| ------------------------ | ----------------------- |
| External database joins | `sql_lookup` |
| Single collection search | No enrichment needed |
| Real-time external data | `api_call` |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------- | --------------- | -------------------------------------- |
| `collection_id` | string | *Required* | Target collection to join from |
| `lookup_field` | string | *Required* | Field in target to match against |
| `source_field` | string | *Required* | Field in source document to use as key |
| `result_field` | string | `enriched_data` | Field to store joined data |
| `select_fields` | array | `null` | Specific fields to return (null = all) |
| `multiple` | boolean | `false` | Return multiple matching documents |
| `limit` | integer | `10` | Max documents when `multiple: true` |
## Configuration Examples
<CodeGroup>
```json Basic Collection Join theme={null}
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "user_profiles",
"target_field": "user_id",
"source_field": "metadata.author_id",
"output_field": "author"
}
}
}
```
```json Select Specific Fields theme={null}
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "products",
"target_field": "product_id",
"source_field": "metadata.product_id",
"output_field": "product_details",
"fields_to_merge": ["name", "price", "category", "image_url"]
}
}
}
```
```json Conditional Enrichment theme={null}
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "reviews",
"target_field": "product_id",
"source_field": "document_id",
"output_field": "reviews"
}
}
}
```
```json Nested Field Lookup theme={null}
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "categories",
"target_field": "category_code",
"source_field": "metadata.taxonomy.primary_category",
"output_field": "category_info"
}
}
}
```
</CodeGroup>
## Output Schema
### Single Document Join
```json theme={null}
{
"document_id": "doc_123",
"content": "Product review content...",
"metadata": {
"product_id": "prod_456"
},
"product_details": {
"name": "Wireless Headphones",
"price": 199.99,
"category": "Electronics",
"image_url": "https://..."
}
}
```
### Multiple Documents Join
```json theme={null}
{
"document_id": "prod_456",
"content": "Product description...",
"reviews": [
{
"document_id": "rev_1",
"rating": 5,
"text": "Great product!"
},
{
"document_id": "rev_2",
"rating": 4,
"text": "Good value"
}
]
}
```
### No Match Found
```json theme={null}
{
"document_id": "doc_123",
"content": "...",
"enriched_data": null
}
```
## Performance
| Metric | Value |
| ---------------------- | --------------------------- |
| **Latency** | 5-20ms per document |
| **Batch processing** | Automatic batching |
| **Index usage** | Uses collection indexes |
| **Parallel execution** | Up to 10 concurrent lookups |
<Tip>
Ensure `lookup_field` is indexed in the target collection for optimal performance. Use `select_fields` to reduce payload size.
</Tip>
## Common Pipeline Patterns
### Search + Author Enrichment
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 20 }
],
"final_top_k": 20
}
}
},
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "users",
"target_field": "user_id",
"source_field": "metadata.author_id",
"output_field": "author",
"fields_to_merge": ["name", "avatar", "bio"]
}
}
}
]
```
### Product Search with Reviews
```json theme={null}
[
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 10 }
],
"final_top_k": 10
}
}
},
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "reviews",
"target_field": "product_id",
"source_field": "document_id",
"output_field": "recent_reviews",
"fields_to_merge": ["rating", "text", "author_name", "created_at"]
}
}
}
]
```
### Hierarchical Category Enrichment
```json theme={null}
[
{
"stage_name": "semantic_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{ "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
],
"final_top_k": 50
}
}
},
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "categories",
"target_field": "category_id",
"source_field": "metadata.category_id",
"output_field": "category"
}
}
},
{
"stage_name": "document_enrich",
"stage_type": "enrich",
"config": {
"stage_id": "document_enrich",
"parameters": {
"target_collection_id": "categories",
"target_field": "category_id",
"source_field": "category.parent_id",
"output_field": "parent_category"
}
}
}
]
```
## Error Handling
| Error | Behavior |
| -------------------- | ---------------------------- |
| Collection not found | Stage fails |
| No matching document | `result_field` set to `null` |
| Invalid field path | Stage fails with error |
| Timeout | Continues with null result |
## vs Other Enrichment Stages
| Feature | document\_enrich | sql\_lookup | api\_call |
| ----------- | ------------------- | ------------------- | --------------- |
| Data source | Mixpeek collections | SQL database | External API |
| Latency | 5-20ms | 10-100ms | 50-500ms |
| Best for | Cross-collection | External relational | REST APIs |
| Setup | None | Connection config | Endpoint config |
## Related
* [SQL Lookup](/docs/retrieval/stages/sql-lookup) - External database joins
* [API Call](/docs/retrieval/stages/api-call) - REST API enrichment
* [Taxonomy Enrich](/docs/retrieval/stages/taxonomy-enrich) - Classification enrichment
# Batch Diagnostics & Troubleshooting
Source: https://docs.mixpeek.com/docs/troubleshoot/batch-diagnostics
Use the API to diagnose, troubleshoot, and fix batch processing issues without accessing infrastructure.
The Mixpeek API provides complete observability into batch processing jobs. You can diagnose issues, cancel stuck jobs, retry failed tiers with modified resources, and trigger self-healing — all through the API.
## Quick Diagnosis
Call the diagnose endpoint to get a complete picture of a batch's health:
<CodeGroup>
```python Python theme={null}
import requests
response = requests.get(
"https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/diagnose",
headers={"Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "your-namespace"}
)
diagnostic = response.json()
print(f"Status: {diagnostic['status']}")
print(f"Failure category: {diagnostic['failure_category']}")
print(f"Failed docs: {diagnostic['failed_document_count']}")
for rec in diagnostic['recommendations']:
print(f" → {rec}")
```
```javascript JavaScript theme={null}
const response = await fetch(
`https://api.mixpeek.com/v1/buckets/${bucketId}/batches/${batchId}/diagnose`,
{ headers: { "Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "your-namespace" } }
);
const diagnostic = await response.json();
console.log(`Status: ${diagnostic.status}`);
console.log(`Failure: ${diagnostic.failure_category}`);
diagnostic.recommendations.forEach(r => console.log(` → ${r}`));
```
```bash cURL theme={null}
curl -X GET "https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/diagnose" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
</CodeGroup>
The response includes:
* **status** and **failure\_category** — programmatic failure classification (infrastructure, timeout, orphaned, pipeline)
* **infrastructure\_events** — OOM, preemption, node failures, Ray bugs with timestamps
* **per\_tier** — timing, submission params, and resource details per tier
* **failed\_documents\_sample** — first 10 failed documents with error details
* **recommendations** — actionable next steps based on the failure type
## Common Failure Scenarios
### Out of Memory (OOM)
The diagnose endpoint will show `failure_category: "infrastructure"` with an infrastructure event of type `oom`.
**Fix:** Retry the failed tier with more resources:
<CodeGroup>
```python Python theme={null}
requests.post(
f"https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/tiers/{tier_num}/retry",
headers={"Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "your-namespace"},
json={"requires_gpu": True, "priority": 50}
)
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/tiers/{tier_num}/retry" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace" \
-H "Content-Type: application/json" \
-d '{"requires_gpu": true, "priority": 50}'
```
</CodeGroup>
### Stuck Job
If a tier shows `IN_PROGRESS` but `last_activity_at` is stale (minutes old), the job may be stuck.
**Fix:** Run stuck detection, then cancel the stuck tier:
<CodeGroup>
```python Python theme={null}
# Detect stuck jobs
requests.post(
f"https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/tiers/{tier_num}/heal",
headers={"Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "your-namespace"},
json={"action": "detect_stuck"}
)
# Cancel the stuck tier
requests.post(
f"https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/tiers/{tier_num}/cancel",
headers={"Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "your-namespace"}
)
```
```bash cURL theme={null}
# Detect stuck
curl -X POST ".../tiers/{tier_num}/heal" -H "..." -d '{"action": "detect_stuck"}'
# Cancel
curl -X POST ".../tiers/{tier_num}/cancel" -H "..."
```
</CodeGroup>
### Duplicate Jobs
If multiple Ray jobs are running for the same extractor in a tier:
<CodeGroup>
```python Python theme={null}
requests.post(
f"https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/tiers/{tier_num}/heal",
headers={"Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "your-namespace"},
json={"action": "kill_duplicates"}
)
```
```bash cURL theme={null}
curl -X POST ".../tiers/{tier_num}/heal" -d '{"action": "kill_duplicates"}'
```
</CodeGroup>
### Cancel a Single Job (Not the Whole Batch)
If one extractor job in a multi-extractor tier is failing but others are fine:
<CodeGroup>
```python Python theme={null}
requests.post(
f"https://api.mixpeek.com/v1/buckets/{bucket_id}/batches/{batch_id}/tiers/{tier_num}/jobs/{ray_job_id}/cancel",
headers={"Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "your-namespace"}
)
```
```bash cURL theme={null}
curl -X POST ".../tiers/{tier_num}/jobs/{ray_job_id}/cancel" -H "..."
```
</CodeGroup>
## Submission Parameters
Every batch tier now persists its submission parameters — the resources, GPU setting, plugins, and entrypoint used when the job was submitted. Access them in the batch response:
```json theme={null}
{
"tier_tasks": [{
"submission_params": {
"entrypoint": "python -m engine.pipelines.entrypoint",
"deployment_mode": "gke",
"requires_gpu": true,
"num_cpus": 1,
"memory_bytes": 8589934592,
"priority": 100,
"plugin_archives": null,
"extractor_name": "universal_extractor_v1"
}
}]
}
```
## Stage Timing Breakdown
The batch progress now includes `stage_history` — a timing breakdown of each completed processing stage:
```json theme={null}
{
"progress": {
"stage_history": [
{"name": "loading", "duration_seconds": 2.5},
{"name": "processing", "duration_seconds": 45.3},
{"name": "writing", "duration_seconds": 8.1}
]
}
}
```
Use this to identify bottlenecks — if "processing" takes 90% of the time, the extractor itself is the bottleneck. If "writing" is slow, the vector store may be under pressure.
# Self-Improving CV Pipeline
Source: https://docs.mixpeek.com/docs/tutorials/annotations-improve-features
Deploy a YOLO model, annotate detections, fine-tune from corrections, and compound accuracy over time using annotations, taxonomies, and clusters
<Tip>This tutorial builds a computer vision pipeline that gets smarter with use. You'll deploy YOLO as a custom extractor, review detections with annotations, export corrections as training data, and close the loop by uploading improved weights — all through Mixpeek primitives.</Tip>
<Frame>
<img alt="Self-improving CV pipeline: YOLO extractor → human review → fine-tune → redeploy, with taxonomies and clusters feeding back into the loop" />
</Frame>
## What You'll Build
A closed-loop object detection system that compounds accuracy over time:
<Steps>
<Step title="Detect">
Deploy YOLO as a custom extractor. Every image ingested produces bounding boxes, class labels, and detection embeddings.
</Step>
<Step title="Review">
Surface low-confidence detections for human review. Annotate each detection as confirmed, corrected, false positive, or missed.
</Step>
<Step title="Fine-Tune">
Export annotations as YOLO-format training data. Fine-tune externally and upload improved weights as a new extractor version.
</Step>
<Step title="Compound">
Taxonomies auto-classify future detections against your curated ground truth. Clusters discover new categories. Retroactive reapplication improves old data.
</Step>
</Steps>
**Prerequisites:** A Mixpeek namespace with an API key. Familiarity with [custom extractors](/docs/processing/custom-extractors) and the [model registry](/docs/processing/model-registry) helps but isn't required.
***
## 1. Deploy YOLO as a Custom Extractor
Package a YOLO-based detector as a custom extractor. The extractor reads images, runs inference, and outputs detection features — bounding boxes, class labels, and confidence scores.
<AccordionGroup>
<Accordion title="manifest.py — extractor metadata and feature definitions">
```python theme={null}
feature_extractor_name = "yolo_detector"
version = "1.0.0"
description = "YOLOv8 object detection with bounding boxes and class embeddings"
dependencies = ["ultralytics==8.2.0", "torch>=2.0"]
features = [
{
"feature_type": "json",
"feature_name": "detections",
},
{
"feature_type": "embedding",
"feature_name": "detection_embedding",
"embedding_dim": 512,
"distance_metric": "cosine",
},
]
output_schema = {
"detections": {
"type": "array",
"items": {
"type": "object",
"properties": {
"class": {"type": "string"},
"confidence": {"type": "number"},
"bbox": {
"type": "object",
"properties": {
"x": {"type": "number"},
"y": {"type": "number"},
"w": {"type": "number"},
"h": {"type": "number"},
},
},
},
},
},
"detection_embedding": {
"type": "array",
"items": {"type": "number"},
"description": "512-dim CLIP embedding of the highest-confidence crop",
},
}
input_mappings = {"image": "image"}
tier = 1
tier_label = "OBJECT_DETECTION"
compute_profile = {"resource_type": "gpu"}
```
<Warning>
Use the exact key names: `feature_type`, `feature_name`, `embedding_dim`, `distance_metric`. Using `name`/`type`/`dimensions`/`distance` will silently create zero vector indexes.
</Warning>
</Accordion>
<Accordion title="pipeline.py — YOLO inference with LazyModelMixin">
```python theme={null}
import numpy as np
import pandas as pd
from engine.models.lazy import LazyModelMixin
from engine.inference.services import BaseBatchInferenceService
from engine.io import parallel_io
class YOLODetector(LazyModelMixin, BaseBatchInferenceService):
model_id = "ultralytics/yolov8m"
model_source = "huggingface"
def _instantiate_model(self, cached_data):
from ultralytics import YOLO
model = YOLO("yolov8m.pt")
model.to(self._detect_device())
return model, None
def _process_batch(self, batch):
model, _ = self.get_model()
images = parallel_io(batch["data"].tolist())
results = model(images, conf=0.25)
all_detections = []
all_embeddings = []
for result in results:
detections = []
for box in result.boxes:
detections.append({
"class": result.names[int(box.cls)],
"confidence": float(box.conf),
"bbox": {
"x": float(box.xywh[0][0]),
"y": float(box.xywh[0][1]),
"w": float(box.xywh[0][2]),
"h": float(box.xywh[0][3]),
},
})
all_detections.append(detections)
if detections:
best = max(detections, key=lambda d: d["confidence"])
crop = result.orig_img[
int(best["bbox"]["y"] - best["bbox"]["h"]/2):int(best["bbox"]["y"] + best["bbox"]["h"]/2),
int(best["bbox"]["x"] - best["bbox"]["w"]/2):int(best["bbox"]["x"] + best["bbox"]["w"]/2),
]
embedding = self._embed_crop(crop)
else:
embedding = np.zeros(512).tolist()
all_embeddings.append(embedding)
batch["detections"] = all_detections
batch["detection_embedding"] = all_embeddings
return batch
def _embed_crop(self, crop):
# Replace with CLIP or similar for production
return np.random.randn(512).astype(np.float32).tolist()
def build_steps(extractor_request=None, base_steps=None, **kwargs):
steps = list(base_steps or [])
steps.append(YOLODetector())
return {"steps": steps, "prepare": lambda ds: ds}
def extract(extractor_request=None, base_steps=None, **kwargs):
result = build_steps(
extractor_request=extractor_request,
base_steps=base_steps, **kwargs
)
class PipelineResult:
def __init__(self, steps, prepare):
self.steps = steps
self.prepare = prepare
return PipelineResult(result["steps"], result["prepare"])
```
</Accordion>
</AccordionGroup>
### Upload and Deploy
```bash theme={null}
# Package
zip -r yolo_detector.zip yolo_detector/
# Upload
UPLOAD=$(curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/plugins/uploads" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "yolo_detector", "version": "1.0.0", "file_size_bytes": 50000}')
UPLOAD_ID=$(echo $UPLOAD | jq -r '.upload_id')
PRESIGNED_URL=$(echo $UPLOAD | jq -r '.presigned_url')
curl -s -X PUT "$PRESIGNED_URL" \
-H "Content-Type: application/zip" \
--data-binary @yolo_detector.zip
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/plugins/uploads/$UPLOAD_ID/confirm" \
-H "Authorization: Bearer $MP_API_KEY"
# Deploy
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/plugins/yolo_detector_1_0_0/deploy?deployment_type=batch_only" \
-H "Authorization: Bearer $MP_API_KEY"
```
<Info>
Your extractor is now available at feature URI `mixpeek://yolo_detector@1.0.0/detection_embedding`. This URI is the stable contract — retrievers, taxonomies, and clusters all reference it, so you can swap model versions without breaking downstream consumers.
</Info>
***
## 2. Create a Collection and Ingest Images
Bind the YOLO extractor to a bucket so every uploaded image gets processed automatically.
<CodeGroup>
```bash cURL theme={null}
# Create bucket
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/buckets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "security_footage",
"bucket_schema": {
"properties": {
"image": {"type": "image", "required": true},
"camera_id": {"type": "text"},
"timestamp": {"type": "text"}
}
}
}'
# Create collection with YOLO extractor
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/collections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "detected_objects",
"feature_extractor": {
"feature_extractor_name": "yolo_detector",
"version": "1.0.0"
},
"source": {
"type": "bucket",
"bucket_ids": ["bkt_security_footage"]
}
}'
# Upload images
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/buckets/$BUCKET_ID/objects" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"metadata": {"camera_id": "cam_lobby_01", "timestamp": "2026-05-03T14:30:00Z"},
"blobs": [{
"property": "image",
"type": "image",
"data": {"url": "s3://my-bucket/footage/frame_001.jpg"}
}]
}'
```
```python Python SDK theme={null}
from mixpeek import Mixpeek
mp = Mixpeek(api_key="API_KEY")
bucket = mp.buckets.create(
name="security_footage",
bucket_schema={
"properties": {
"image": {"type": "image", "required": True},
"camera_id": {"type": "text"},
"timestamp": {"type": "text"},
}
},
)
collection = mp.collections.create(
name="detected_objects",
feature_extractor={
"feature_extractor_name": "yolo_detector",
"version": "1.0.0",
},
source={"type": "bucket", "bucket_ids": [bucket.bucket_id]},
)
mp.buckets.objects.create(
bucket_id=bucket.bucket_id,
metadata={"camera_id": "cam_lobby_01", "timestamp": "2026-05-03T14:30:00Z"},
blobs=[{
"property": "image",
"type": "image",
"data": {"url": "s3://my-bucket/footage/frame_001.jpg"},
}],
)
```
</CodeGroup>
Trigger batch processing to run YOLO across all uploaded images:
```bash theme={null}
# Buckets/batches are top-level, keyed by the X-Namespace header.
# A batch is created (objects + collections) then submitted.
H=(-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $NS_ID" -H "Content-Type: application/json")
OBJIDS=$(curl -s "$MP_API_URL/v1/buckets/$BUCKET_ID/objects" "${H[@]}" | jq -c '[.results[].object_id]')
BATCH_ID=$(curl -s -X POST "$MP_API_URL/v1/buckets/$BUCKET_ID/batches" "${H[@]}" \
-d "{\"batch_name\":\"yolo-run\",\"object_ids\":$OBJIDS,\"collection_ids\":[\"$COLLECTION_ID\"]}" | jq -r '.batch_id')
curl -s -X POST "$MP_API_URL/v1/buckets/$BUCKET_ID/batches/$BATCH_ID/submit" "${H[@]}" \
-d "{\"collection_ids\":[\"$COLLECTION_ID\"]}"
```
***
## 3. Build a Retriever for Detection Review
Create a retriever that surfaces detections for human review. Filter by confidence to focus reviewers on borderline cases where the model is least sure.
<CodeGroup>
```bash cURL theme={null}
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "detection_review",
"stages": [{
"stage_name": "low_confidence",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://yolo_detector@1.0.0/detection_embedding",
"query": "{{INPUT.query}}"
}],
"filters": {
"AND": [{
"field": "detections.0.confidence",
"operator": "lt",
"value": 0.7
}]
}
}
}
}]
}'
```
```python Python SDK theme={null}
retriever = mp.retrievers.create(
name="detection_review",
stages=[{
"stage_name": "low_confidence",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "mixpeek://yolo_detector@1.0.0/detection_embedding",
"query": "{{INPUT.query}}",
}],
"filters": {
"AND": [{
"field": "detections.0.confidence",
"operator": "lt",
"value": 0.7,
}]
},
},
},
}],
)
```
</CodeGroup>
<Tip>
**Focus reviewers on uncertainty.** Annotating high-confidence correct detections adds little value. Filtering for confidence \< 0.7 routes reviewers to the cases where YOLO is least sure — exactly the training signal you need for the next fine-tune.
</Tip>
***
## 4. Annotate Detections
Reviewers examine each detection and record their decision. The `payload` carries the corrected bounding boxes and class labels — this is what becomes training data.
### Label Vocabulary
Before annotating, establish a consistent label vocabulary. The stats endpoint groups by exact string match, so consistency matters.
<CardGroup>
<Card title="confirmed" icon="check">
Detection is correct as-is. Becomes a **positive training sample** that reinforces the model.
</Card>
<Card title="corrected" icon="pen">
Bounding box or class was adjusted. **Highest-value sample** — teaches the model its mistakes.
</Card>
<Card title="false_positive" icon="xmark">
No real object at this location. Becomes a **hard negative** that reduces false alarms.
</Card>
<Card title="missed" icon="plus">
Object exists but wasn't detected. Added as **new ground truth** for the next training run.
</Card>
</CardGroup>
### Recording Decisions
<Tabs>
<Tab title="Individual Annotations">
<CodeGroup>
```bash cURL theme={null}
# Correct detection — class was wrong
curl -X POST "$MP_API_URL/v1/annotations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"document_id": "doc_frame_001_det_3",
"collection_id": "col_detected_objects",
"label": "corrected",
"confidence": 1.0,
"reasoning": "Model predicted car, actual object is delivery van.",
"payload": {
"predicted_class": "car",
"true_class": "delivery_van",
"bbox": {"x": 340, "y": 220, "w": 180, "h": 120},
"image_width": 1920,
"image_height": 1080
},
"retriever_id": "ret_detection_review",
"execution_id": "exec_review_batch_01"
}'
# Confirm correct detection
curl -X POST "$MP_API_URL/v1/annotations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"document_id": "doc_frame_001_det_1",
"collection_id": "col_detected_objects",
"label": "confirmed",
"confidence": 1.0,
"payload": {
"predicted_class": "person",
"true_class": "person",
"bbox": {"x": 640, "y": 300, "w": 90, "h": 200}
},
"retriever_id": "ret_detection_review",
"execution_id": "exec_review_batch_01"
}'
# Reject false positive
curl -X POST "$MP_API_URL/v1/annotations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"document_id": "doc_frame_001_det_5",
"collection_id": "col_detected_objects",
"label": "false_positive",
"reasoning": "Shadow on wall, not an actual object.",
"retriever_id": "ret_detection_review",
"execution_id": "exec_review_batch_01"
}'
```
```python Python SDK theme={null}
# Correct detection — class was wrong
mp.annotations.create(
document_id="doc_frame_001_det_3",
collection_id="col_detected_objects",
label="corrected",
confidence=1.0,
reasoning="Model predicted car, actual object is delivery van.",
payload={
"predicted_class": "car",
"true_class": "delivery_van",
"bbox": {"x": 340, "y": 220, "w": 180, "h": 120},
"image_width": 1920,
"image_height": 1080,
},
retriever_id="ret_detection_review",
execution_id="exec_review_batch_01",
)
# Confirm correct detection
mp.annotations.create(
document_id="doc_frame_001_det_1",
collection_id="col_detected_objects",
label="confirmed",
confidence=1.0,
payload={
"predicted_class": "person",
"true_class": "person",
"bbox": {"x": 640, "y": 300, "w": 90, "h": 200},
},
retriever_id="ret_detection_review",
execution_id="exec_review_batch_01",
)
# Reject false positive
mp.annotations.create(
document_id="doc_frame_001_det_5",
collection_id="col_detected_objects",
label="false_positive",
reasoning="Shadow on wall, not an actual object.",
retriever_id="ret_detection_review",
execution_id="exec_review_batch_01",
)
```
</CodeGroup>
</Tab>
<Tab title="Bulk Review (up to 1,000)">
Process an entire review queue in a single call:
```bash theme={null}
curl -X POST "$MP_API_URL/v1/annotations/bulk" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"create": [
{
"document_id": "doc_det_001",
"collection_id": "col_detected_objects",
"label": "confirmed",
"payload": {"true_class": "person", "bbox": {"x": 100, "y": 200, "w": 50, "h": 120}}
},
{
"document_id": "doc_det_002",
"collection_id": "col_detected_objects",
"label": "corrected",
"payload": {"predicted_class": "dog", "true_class": "cat", "bbox": {"x": 300, "y": 150, "w": 80, "h": 60}}
},
{
"document_id": "doc_det_003",
"collection_id": "col_detected_objects",
"label": "false_positive"
}
]
}'
```
Each operation is independent — a failure in one does not roll back the others. The response includes per-operation results so you can retry individual failures.
</Tab>
</Tabs>
<Note>
Always include `retriever_id` and `execution_id` when annotating retriever results. This provenance link lets you measure which retrievers produce the most approved vs. rejected results — critical for evaluating retriever quality over time.
</Note>
***
## 5. Track Model Quality with Stats
Monitor how your model is performing across review cycles. A rising `corrected` or `false_positive` rate signals the model needs retraining.
<CodeGroup>
```bash cURL theme={null}
curl "$MP_API_URL/v1/annotations/stats?collection_id=col_detected_objects" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python SDK theme={null}
stats = mp.annotations.stats(collection_id="col_detected_objects")
# {"total": 500, "by_label": {"confirmed": 340, "corrected": 95, "false_positive": 45, "missed": 20}}
```
</CodeGroup>
### Interpreting Stats for Retraining Decisions
| Metric | Healthy | Action Needed |
| ------------------- | ------- | --------------------------------------------------------------------------- |
| Confirmed rate | > 80% | Model is performing well |
| Corrected rate | > 15% | Class confusion — retrain with corrected examples |
| False positive rate | > 10% | Confidence threshold too low, or hard negatives needed |
| Missed rate | > 5% | Model is missing objects — add missed annotations as positive training data |
<Warning>
Track stats **over time**, not just cumulatively. A model at 90% confirmed overall might be at 60% confirmed on last week's data if the deployment context changed (new camera angle, different lighting, seasonal changes).
</Warning>
***
## 6. Export Annotations as YOLO Training Data
Query your annotations and convert them to YOLO format. Every corrected bounding box and confirmed detection becomes a labeled training sample.
```python theme={null}
import os
confirmed = mp.annotations.list(
collection_id="col_detected_objects",
label="confirmed",
)
corrected = mp.annotations.list(
collection_id="col_detected_objects",
label="corrected",
)
os.makedirs("dataset/labels", exist_ok=True)
class_map = {}
class_counter = 0
for ann in confirmed.items + corrected.items:
payload = ann.payload
true_class = payload.get("true_class", payload.get("predicted_class"))
bbox = payload.get("bbox", {})
img_w = payload.get("image_width", 1920)
img_h = payload.get("image_height", 1080)
if not bbox or not true_class:
continue
if true_class not in class_map:
class_map[true_class] = class_counter
class_counter += 1
# Convert to YOLO format: class x_center y_center width height (normalized)
x_center = bbox["x"] / img_w
y_center = bbox["y"] / img_h
width = bbox["w"] / img_w
height = bbox["h"] / img_h
label_file = f"dataset/labels/{ann.document_id}.txt"
with open(label_file, "a") as f:
f.write(f"{class_map[true_class]} {x_center:.6f} {y_center:.6f} {width:.6f} {height:.6f}\n")
with open("dataset/classes.txt", "w") as f:
for name, idx in sorted(class_map.items(), key=lambda x: x[1]):
f.write(f"{name}\n")
print(f"Exported {len(confirmed.items) + len(corrected.items)} annotations across {len(class_map)} classes")
```
<Info>
The YOLO format expects one `.txt` file per image with lines of `class x_center y_center width height`, all values normalized to `[0, 1]`. The export script handles this conversion from Mixpeek's pixel-coordinate annotation payloads.
</Info>
***
## 7. Fine-Tune and Redeploy
Fine-tune YOLO externally with your exported annotations, then upload the improved weights as a new extractor version.
<Tabs>
<Tab title="Fine-Tune (Ultralytics)">
```python theme={null}
from ultralytics import YOLO
model = YOLO("yolov8m.pt")
model.train(
data="dataset/data.yaml",
epochs=50,
imgsz=640,
batch=16,
name="yolo_v2_finetuned",
)
model.export(format="torchscript")
```
</Tab>
<Tab title="Upload as New Extractor Version">
```bash theme={null}
# Update manifest.py version to "2.0.0" and package
zip -r yolo_detector_v2.zip yolo_detector/
UPLOAD=$(curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/plugins/uploads" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "yolo_detector", "version": "2.0.0", "file_size_bytes": 80000}')
UPLOAD_ID=$(echo $UPLOAD | jq -r '.upload_id')
PRESIGNED_URL=$(echo $UPLOAD | jq -r '.presigned_url')
curl -s -X PUT "$PRESIGNED_URL" \
-H "Content-Type: application/zip" \
--data-binary @yolo_detector_v2.zip
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/plugins/uploads/$UPLOAD_ID/confirm" \
-H "Authorization: Bearer $MP_API_KEY"
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/plugins/yolo_detector_2_0_0/deploy?deployment_type=batch_only" \
-H "Authorization: Bearer $MP_API_KEY"
```
</Tab>
<Tab title="Alternative: Model Registry">
Upload fine-tuned weights to the [Model Registry](/docs/processing/model-registry) as a namespace model. This separates model weights from extractor code, so you can iterate on weights without repackaging the extractor.
```bash theme={null}
# Package weights
tar -czvf yolo_v2_weights.tar.gz ./runs/detect/yolo_v2_finetuned/weights/
# Upload to registry
curl -X POST "$MP_API_URL/v1/namespaces/$NS_ID/models" \
-H "Authorization: Bearer $MP_API_KEY" \
-F "file=@yolo_v2_weights.tar.gz" \
-F "name=yolo-detector" \
-F "version=2.0.0" \
-F "model_format=pytorch" \
-F "task_type=detection" \
-F "num_gpus=1"
# Deploy to Ray object store
curl -X POST "$MP_API_URL/v1/namespaces/$NS_ID/models/yolo-detector_2_0_0/deploy" \
-H "Authorization: Bearer $MP_API_KEY"
```
Then reference in your extractor via `load_namespace_model("yolo-detector_2_0_0")`.
</Tab>
</Tabs>
<Tip>
The new version gets its own feature URI — `mixpeek://yolo_detector@2.0.0/detection_embedding` — so you can run both versions side by side and compare results before switching production traffic.
</Tip>
***
## 8. Auto-Classify Detections with Taxonomies
Once you have enough confirmed annotations, promote them to a reference collection. Then create a taxonomy that auto-classifies future detections by matching against your curated ground truth.
<Steps>
<Step title="Create the taxonomy">
```bash theme={null}
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/taxonomies" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"taxonomy_name\": \"object-catalog\",
\"taxonomy_type\": \"flat\",
\"retriever_id\": \"ret_object_catalog_search\",
\"collection_id\": \"col_verified_objects\",
\"input_mappings\": [{
\"source\": \"detection_embedding\",
\"target\": \"query\"
}],
\"enrichment_fields\": [
{\"source\": \"true_class\", \"target\": \"verified_class\"},
{\"source\": \"category\", \"target\": \"object_category\"}
],
\"threshold\": 0.75,
\"execution_mode\": \"materialize\"
}"
```
</Step>
<Step title="Apply to your detection collection">
Every new image auto-classifies at ingestion time:
```json theme={null}
{
"taxonomy_applications": [
{
"taxonomy_id": "tax_object_catalog",
"execution_mode": "materialize"
}
]
}
```
</Step>
<Step title="Backfill existing data when the reference improves">
When annotations accumulate and your reference collection gets better, trigger retroactive mode to reclassify all existing detections:
```bash theme={null}
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/taxonomies/tax_object_catalog/apply" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"execution_mode": "retroactive", "collection_id": "col_detected_objects"}'
```
</Step>
</Steps>
<Note>
Retroactive reapplication is a first-class operation, not a data migration. When your reference improves — more annotations, better coverage, new categories — old data automatically re-benefits.
</Note>
***
## 9. Discover New Categories with Clusters
YOLO might detect "unknown" objects that don't fit existing classes. Use clustering to group similar unknowns and discover categories you haven't labeled yet.
```bash theme={null}
curl -s -X POST "$MP_API_URL/v1/namespaces/$NS_ID/clusters" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"cluster_name\": \"unknown-objects\",
\"collection_ids\": [\"$COLLECTION_ID\"],
\"cluster_type\": \"vector\",
\"vector_config\": {
\"feature_uris\": [\"mixpeek://yolo_detector@2.0.0/detection_embedding\"],
\"clustering_method\": \"hdbscan\",
\"hdbscan_parameters\": {\"min_cluster_size\": 5}
},
\"llm_labeling\": {
\"enabled\": true,
\"input_mappings\": [{
\"source\": \"payload\",
\"fields\": [\"detections\"]
}]
},
\"dimension_reduction\": {\"method\": \"umap\", \"n_components\": 2}
}"
```
Clusters reveal groups like "delivery trucks," "bicycles," or "strollers" — objects the base YOLO model might lump together or miss entirely.
<Steps>
<Step title="Review cluster labels">
The LLM-generated name gives you a starting point. Review the cluster members to confirm the grouping makes sense.
</Step>
<Step title="Promote to taxonomy node">
The cluster becomes a reference for auto-classification. Future detections matching this cluster auto-label.
</Step>
<Step title="Add to training data">
Confirmed cluster members become training samples for the next YOLO fine-tune — new classes discovered from your own data.
</Step>
</Steps>
***
## 10. Automate the Loop with Webhooks
Wire up webhooks so the pipeline runs without manual intervention. Each annotation event can trigger downstream processing.
```bash theme={null}
curl -X POST "$MP_API_URL/v1/webhooks" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"webhook_name": "detection-review-events",
"url": "https://your-app.com/webhooks/detections",
"events": [
"annotation.created",
"annotation.updated",
"batch.completed"
]
}'
```
### Automation Patterns
| Event | Trigger | Action |
| -------------------- | --------------------------------------------- | ------------------------------------------------------- |
| `annotation.created` | Label is `confirmed` or `corrected` | Add to reference collection, append to training dataset |
| `annotation.created` | Label is `false_positive` | Log as hard negative for next training run |
| Annotation count | Crosses threshold (e.g., 500 new corrections) | Trigger fine-tuning job, export training data |
| `batch.completed` | New extractor version finishes processing | Run evaluation comparing v1 vs. v2 detection quality |
***
## The Compounding Flywheel
Each Mixpeek primitive contributes to a system that gets better with use:
<CardGroup>
<Card title="Custom Extractor" icon="plug">
Runs YOLO, produces detections with stable feature URIs. Versioned — v1 and v2 coexist.
</Card>
<Card title="Annotations" icon="pen-to-square">
Captures human corrections — the highest-quality training signal. Bulk API for review queues.
</Card>
<Card title="Model Registry" icon="database">
Stores fine-tuned weights. Upload, deploy, version — without repackaging the extractor.
</Card>
<Card title="Taxonomies" icon="sitemap">
Auto-classifies detections against curated ground truth. Retroactive mode backfills old data.
</Card>
<Card title="Clusters" icon="circle-nodes">
Discovers object categories you haven't labeled yet. Promote stable clusters to taxonomy nodes.
</Card>
<Card title="Webhooks" icon="bell">
Triggers downstream actions on every annotation event. No polling required.
</Card>
</CardGroup>
<Tip>
The key insight is that these primitives **compose**. Annotations curate the edges where the model was wrong. Those curated edges become training data *and* reference collection entries. The reference collection powers taxonomy auto-classification. Clusters discover what you haven't labeled yet. And every improvement backfills via retroactive taxonomy application — old data re-benefits from every new correction.
</Tip>
## Next Steps
<CardGroup>
<Card title="Custom Extractors" icon="plug" href="/processing/custom-extractors">
Full guide to packaging and deploying custom feature extractors.
</Card>
<Card title="Model Registry" icon="database" href="/processing/model-registry">
Upload fine-tuned weights, manage versions, and deploy to the inference cluster.
</Card>
<Card title="Taxonomies" icon="sitemap" href="/enrichment/taxonomies">
Build flat and hierarchical classification systems with retroactive reapplication.
</Card>
<Card title="Clusters" icon="circle-nodes" href="/enrichment/clusters">
Discover structure in your data with 8 algorithms and LLM-powered labeling.
</Card>
</CardGroup>
# Auto-Labeling Datasets
Source: https://docs.mixpeek.com/docs/tutorials/bootstrap-labeled-dataset
Build a self-improving classification system using taxonomy auto-labeling
**Build a labeled dataset from scratch and auto-classify new data using taxonomy-based matching.**
<Tip>Auto-labeling uses the warehouse's enrichment layer (taxonomies) to classify documents at query time, the multimodal equivalent of a SQL JOIN.</Tip>
This tutorial shows how to:
1. Start with unlabeled data
2. Use feature extraction to find relevant items
3. Manually label a small reference set
4. Automatically classify new items based on the reference set
5. Create a self-improving system that gets better over time
<img alt="Bootstrap Labeled Dataset Workflow" />
***
## Overview
This tutorial demonstrates two approaches to building an auto-labeling system:
* **Option A: Unified Approach** (Recommended) - Single bucket/collection that grows smarter over time
* **Option B: Separate Approach** - Dedicated reference set with production data separated
Both approaches follow the same core workflow:
1. Upload unlabeled data with feature extraction
2. Manually label a small reference set (10-20 examples per category)
3. Configure taxonomy to auto-label new items based on similarity
4. Review and label unknowns to continuously improve
***
## Use Cases
* **Product Recognition**: Label product images, auto-tag new inventory
* **People Identification**: Build a face recognition system from photos
* **Document Classification**: Categorize documents by type or topic
* **Object Detection**: Label objects in images for training data
***
## Option A: Unified Approach (Recommended)
The unified approach uses a single bucket and collection that references itself. As you label items, they immediately become part of the reference set for future matches.
### Step 1: Create Bucket and Collection
Create a bucket and collection with self-referencing taxonomy:
```bash theme={null}
# Create bucket
POST /v1/buckets
{
"bucket_name": "products_unified",
"bucket_schema": {
"properties": {
"product_label": { "type": "text" },
"image_url": { "type": "text" }
}
}
}
# Create retriever (do this first, before collection)
POST /v1/retrievers
{
"retriever_name": "products_unified_classifier",
"collection_identifiers": ["products_unified"],
"stages": [
{
"stage_name": "labeled_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{ "field": "product_label", "operator": "exists", "value": true }
]
}
}
}
},
{
"stage_name": "image_match",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": { "input_mode": "content", "value": "{{INPUT.query_image}}" },
"top_k": 1,
"min_score": 0.30
}
]
}
}
}
]
}
# Create collection that references itself
POST /v1/collections
{
"collection_name": "products_unified",
"source": {
"type": "bucket",
"bucket_ids": ["bkt_products_unified"]
},
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": { "image": "image_url" },
"field_passthrough": ["product_label"]
},
"taxonomy": {
"retriever_id": "ret_products_unified_classifier",
"field_to_enrich": "product_label",
"confidence_threshold": 0.30
}
}
```
### Step 2: Upload Initial Unlabeled Data
```bash theme={null}
POST /v1/buckets/{bucket_id}/objects
{
"key_prefix": "/bootstrap",
"metadata": {
"product_label": null
},
"blobs": [{
"property": "image_url",
"type": "image",
"data": {
"url": "s3://my-bucket/products/shoe-001.jpg"
}
}]
}
```
Upload 50-100 images. Feature extraction happens automatically, but no auto-labeling occurs yet (no labeled examples to match against).
### Step 3: Manually Label Reference Set
Query documents and label them:
```bash theme={null}
# Get documents
GET /v1/collections/{collection_id}/documents?return_presigned_urls=true
# Label via bucket (syncs to collection automatically)
PATCH /v1/buckets/{bucket_id}/objects/{object_id}
{
"metadata": {
"product_label": "Red Running Shoes"
}
}
```
**Labeling tips:**
* Label 10-20 examples per category minimum
* Include diverse examples (angles, lighting, backgrounds)
* Use consistent naming conventions
### Step 4: Upload New Items - Auto-Labeling Works!
Now that you have labeled examples, new uploads auto-label automatically:
```bash theme={null}
POST /v1/buckets/{bucket_id}/objects
{
"key_prefix": "/new-arrivals",
"blobs": [{
"property": "image_url",
"type": "image",
"data": {
"url": "s3://my-bucket/new-arrivals/shoe-new.jpg"
}
}]
}
```
**What happens automatically:**
1. Feature extraction runs on the new image
2. Taxonomy searches your labeled items for similar matches
3. If similarity > 0.30 → Auto-labels (e.g., `"Red Running Shoes"`)
4. If similarity \< 0.30 → Leaves as `null` for manual review
**Check the result:**
```bash theme={null}
GET /v1/collections/{collection_id}/documents/{document_id}
```
**Matched:**
```json theme={null}
{
"metadata": {
"product_label": "Red Running Shoes"
},
"taxonomy_match": {
"matched": true,
"confidence": 0.87,
"source_document_id": "doc_xyz123"
}
}
```
**Unknown (needs manual review):**
```json theme={null}
{
"metadata": {
"product_label": null
},
"taxonomy_match": {
"matched": false,
"confidence": 0.21
}
}
```
### Step 5: Review and Label Unknowns
Find items that need manual labeling:
```bash theme={null}
GET /v1/collections/{collection_id}/documents?filters={
"must": [
{
"key": "product_label",
"match": { "operator": "eq", "value": null }
}
]
}
```
Label them via bucket (automatically syncs to collection):
```bash theme={null}
PATCH /v1/buckets/{bucket_id}/objects/{object_id}
{
"metadata": {
"product_label": "Blue Basketball Shoes"
}
}
```
**Self-improvement in action**: This newly labeled item becomes part of the reference set for future uploads!
***
## Option B: Separate Approach
For more control, keep reference data separate from production data:
* **Reference bucket/collection**: Curated, high-quality labeled examples
* **Production bucket/collection**: All data with auto-labels
**When to use:**
* Need strict quality control on reference set
* Want to prevent noisy auto-labels from affecting matching
* Prefer to manually review before promoting items to reference
### Step 1: Create Reference Bucket and Collection
```bash theme={null}
# Reference bucket
POST /v1/buckets
{
"bucket_name": "product_reference",
"bucket_schema": {
"properties": {
"product_label": { "type": "text" },
"image_url": { "type": "text" }
}
}
}
# Reference collection (no taxonomy needed)
POST /v1/collections
{
"collection_name": "product_reference",
"source": {
"type": "bucket",
"bucket_ids": ["bkt_product_reference"]
},
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": { "image": "image_url" },
"field_passthrough": ["product_label"]
}
}
# Create taxonomy retriever
POST /v1/retrievers
{
"retriever_name": "product_classifier",
"collection_identifiers": ["product_reference"],
"stages": [
{
"stage_name": "labeled_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"conditions": {
"AND": [
{ "field": "product_label", "operator": "exists", "value": true }
]
}
}
}
},
{
"stage_name": "image_match",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": { "input_mode": "content", "value": "{{INPUT.query_image}}" },
"top_k": 1,
"min_score": 0.30
}
]
}
}
}
]
}
```
### Step 2: Upload and Label Reference Set
Upload 50-100 curated images to the reference bucket and manually label them:
```bash theme={null}
# Upload to reference
POST /v1/buckets/bkt_product_reference/objects
{
"metadata": { "product_label": null },
"blobs": [{ "property": "image_url", "type": "image", "data": "https://example.com/image.jpg" }]
}
# Label them
PATCH /v1/buckets/bkt_product_reference/objects/{object_id}
{
"metadata": { "product_label": "Red Running Shoes" }
}
```
### Step 3: Create Production Bucket and Collection
```bash theme={null}
# Production bucket
POST /v1/buckets
{
"bucket_name": "product_catalog",
"bucket_schema": {
"properties": {
"product_label": { "type": "text" },
"image_url": { "type": "text" }
}
}
}
# Production collection with taxonomy
POST /v1/collections
{
"collection_name": "product_catalog",
"source": {
"type": "bucket",
"bucket_ids": ["bkt_product_catalog"]
},
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": { "image": "image_url" },
"field_passthrough": ["product_label"]
},
"taxonomy": {
"retriever_id": "ret_product_classifier",
"field_to_enrich": "product_label",
"confidence_threshold": 0.30
}
}
```
### Step 4: Upload Production Data
New uploads auto-label based on the reference set:
```bash theme={null}
POST /v1/buckets/bkt_product_catalog/objects
{
"blobs": [{ "property": "image_url", "type": "image", "data": "https://example.com/image.jpg" }]
}
```
### Step 5: Promote High-Confidence Items to Reference
Periodically review production data and promote high-confidence matches:
```bash theme={null}
# Find high-confidence items
GET /v1/collections/product_catalog/documents?filters={
"must": [
{
"key": "taxonomy_match.confidence",
"match": { "operator": "gte", "value": 0.85 }
}
]
}
# Copy to reference bucket
POST /v1/buckets/bkt_product_reference/objects
{
"metadata": { "product_label": "..." },
"blobs": [{ ... }]
}
```
***
## Real-World Examples
### Example 1: Face Recognition System
```bash theme={null}
# Create bucket for employee photos
POST /v1/buckets
{
"bucket_name": "employee_photos",
"bucket_schema": {
"properties": {
"person_name": { "type": "text" },
"employee_id": { "type": "text" },
"photo_url": { "type": "text" }
}
}
}
# Bootstrap collection with face extraction
POST /v1/collections
{
"collection_name": "employee_faces",
"source": {
"type": "bucket",
"bucket_ids": ["bkt_employee_photos"]
},
"feature_extractor": {
"feature_extractor_name": "face_identity_extractor",
"version": "v1",
"input_mappings": { "image": "photo_url" },
"field_passthrough": ["person_name", "employee_id"]
}
}
# Upload 50 employee photos → manually label with names
# Create taxonomy retriever
# Security camera footage auto-identifies employees
```
### Example 2: Document Classification
```bash theme={null}
# Create bucket for documents
POST /v1/buckets
{
"bucket_name": "company_documents",
"bucket_schema": {
"properties": {
"document_type": { "type": "text" },
"content": { "type": "text" }
}
}
}
# Bootstrap collection with text extraction
POST /v1/collections
{
"collection_name": "document_types",
"source": {
"type": "bucket",
"bucket_ids": ["bkt_company_documents"]
},
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": { "text": "content" },
"field_passthrough": ["document_type"]
},
"taxonomy": {
"field_to_enrich": "document_type",
"confidence_threshold": 0.35
}
}
# Label 20 invoices, 20 contracts, 20 receipts
# New documents auto-classify by type
```
***
## Advanced Configuration
### Tuning Confidence Thresholds
The `confidence_threshold` determines how conservative auto-labeling is:
| Threshold | Behavior | Use Case |
| ----------- | ------------ | --------------------------------- |
| `0.20-0.25` | Aggressive | High recall, more false positives |
| `0.30-0.35` | Balanced | Good starting point |
| `0.40-0.50` | Conservative | High precision, fewer auto-labels |
| `0.60+` | Very strict | Only exact matches |
**Finding the right threshold:**
1. Start with `0.30`
2. Monitor false positive rate (wrong auto-labels)
3. Check coverage (% of items auto-labeled)
4. Adjust based on cost of errors:
* **High cost of errors** (e.g., medical imaging) → Higher threshold
* **Low cost of errors** (e.g., photo organization) → Lower threshold
### Monitoring & Analytics
Track performance with these queries:
```bash theme={null}
# Get distribution of labels
GET /v1/collections/{collection_id}/analytics/field-distribution?field=product_label
# Check match confidence distribution
GET /v1/collections/{collection_id}/documents?sort_by=taxonomy_match.confidence&limit=100
# Find low-confidence matches for review
GET /v1/collections/{collection_id}/documents?filters={
"must": [
{
"key": "taxonomy_match.matched",
"match": { "operator": "eq", "value": true }
},
{
"key": "taxonomy_match.confidence",
"match": { "operator": "lt", "value": 0.40 }
}
]
}
```
**Key metrics:**
* **Auto-label coverage**: % of new items auto-labeled
* **Manual review queue**: # of items with `label: null`
* **Confidence distribution**: Are matches clustered around threshold?
* **False positive rate**: Sample and manually verify auto-labels
### Best Practices
**Reference set quality:**
* Include diverse examples (angles, lighting, backgrounds)
* Use consistent naming conventions
* Aim for balanced distribution across categories
* Maintain high-quality, unambiguous images
**Labeling guidelines:**
* Create a labeling style guide
* Consider hierarchical labels: `"Shoes > Running > Red"`
* Define rules for edge cases
* Version your taxonomy as it evolves
**Continuous improvement:**
* Review unknowns regularly
* Audit auto-labels periodically
* Add corrected examples when system makes mistakes
* Expand categories as needed
**Production deployment:**
* Start with conservative threshold (0.40+)
* Implement human-in-the-loop for critical applications
* Enable feedback mechanism for corrections
* A/B test threshold changes
***
## Troubleshooting
### Too many unlabeled items
**Causes**: Threshold too high, insufficient reference examples, new categories
**Solutions**:
* Lower `confidence_threshold` to 0.25-0.30
* Add 20+ examples per category to reference set
* Review and label new categories
### False positives (wrong labels)
**Causes**: Threshold too low, similar categories, poor quality references
**Solutions**:
* Raise `confidence_threshold` to 0.40+
* Add diverse examples to distinguish categories
* Clean up reference set
### System not self-improving
**Causes**: Labels not syncing, configuration issues
**Solutions**:
* Verify `field_passthrough` includes label field
* Check retriever filters for non-null labels
* Confirm bucket-to-collection sync is working
***
## Summary
**Workflow:**
1. Create bucket and collection with feature extraction
2. Upload unlabeled data (50-100 items)
3. Manually label reference set (10-20 per category)
4. Create taxonomy retriever pointing to labeled items
5. New uploads auto-label based on similarity
6. Review and label unknowns to improve system
**Key benefits:**
* Start with zero labels, build incrementally
* Automate repetitive labeling
* Self-improving with each manual correction
* Scales from dozens to millions
**Next steps:**
* Choose unified (simpler) or separate (more control) approach
* Start with 50-100 reference items
* Test different confidence thresholds (start at 0.30)
* Monitor auto-label quality and adjust
***
## Discover Clusters
Use clustering to find new categories before defining them manually:
```bash theme={null}
POST /v1/clusters
{
"cluster_name": "product-discovery",
"collection_ids": ["col_products_unified"],
"cluster_type": "vector",
"vector_config": {
"feature_uris": ["mixpeek://image_extractor@v1/google_siglip_base_v1"],
"clustering_method": "hdbscan",
"hdbscan_parameters": { "min_cluster_size": 5 }
},
"llm_labeling": {
"enabled": true,
"input_mappings": [{ "source": "payload", "fields": ["product_label"] }]
},
"dimension_reduction": { "method": "umap", "n_components": 2 }
}
```
Clusters reveal groups you haven't labeled yet — "sandals", "boots", "athletic wear" — without predefined categories. Once a cluster stabilizes, promote it to a taxonomy node so future items auto-classify into it. See [Clusters](/docs/enrichment/clusters).
## Set Up Alerts
Get notified when items fail to auto-label (unknown categories needing manual review). An alert **runs a retriever and fires on its results** — so first create a retriever that surfaces unlabeled items, then point an alert at it.
```bash theme={null}
# 1. Retriever that returns products with no label. attribute_filter as the
# first stage fetches + filters straight from the collection (no search needed).
POST /v1/retrievers
{
"retriever_name": "unlabeled-products",
"collection_identifiers": ["col_products_unified"],
"input_schema": {},
"stages": [
{ "stage_name": "missing_label", "stage_type": "filter",
"config": { "stage_id": "attribute_filter", "parameters": {
"field": "product_label", "operator": "exists", "value": false } } }
]
}
# -> { "retriever_id": "ret_unlabeled" }
# 2. Fire a webhook whenever that retriever returns any results
POST /v1/alerts
{
"name": "unknown-products",
"source": "retriever",
"retriever_id": "ret_unlabeled",
"trigger_on": "results",
"notification_config": {
"channels": [
{ "channel_type": "webhook", "config": { "url": "https://example.com/webhook" } }
]
}
}
```
See [Alerts](/docs/enrichment/alerts) for system-metric alerts and Slack/email channels.
## Set Up Webhooks
Forward ingestion and labeling events to your own systems:
```bash theme={null}
POST /v1/organizations/webhooks
{
"webhook_name": "labeling-events",
"event_types": ["object.created", "object.updated"],
"channels": [
{ "channel": "webhook", "configs": { "url": "https://example.com/webhook", "method": "POST" } }
]
}
```
`event_types` accepts object/collection/cluster/taxonomy/alert lifecycle events (e.g. `object.created`, `object.updated`, `object.deleted`, `collection.created`). See [Webhooks](/docs/operations/webhooks) for the full event list and Slack/email channels.
# Visual Document Retrieval
Source: https://docs.mixpeek.com/docs/tutorials/colpali-visual-document-retrieval
Search PDFs, scanned pages, and figure-heavy reports by visual content using cross-modal embeddings — no OCR required
<Tip>Visual document retrieval treats each page as an image and embeds it with a cross-modal model, so a text query can find the right page by what it *looks like* — tables, charts, diagrams, scans — without relying on OCR.</Tip>
## Why Visual Document Retrieval
Traditional document search pipelines drop information at every stage:
1. **OCR** mangles tables, equations, and low-contrast scans.
2. **Layout parsers** miss chart and diagram semantics.
3. **Text-only embeddings** never see the visual structure of the page.
Mixpeek's `multimodal_extractor` embeds page images directly with Google's Vertex multimodal model into a shared text-image space (`vertex_multimodal_embedding`). Because the space is cross-modal, a **text query retrieves visually-relevant pages** — the words "revenue breakdown by region" can match a page dominated by a financial table, even with no clean extractable text.
<Note>
This is single-vector cross-modal retrieval (one embedding per page). Mixpeek does not currently offer ColPali-style multi-vector *late-interaction* (per-patch MaxSim) scoring. For born-digital, text-heavy PDFs where you want extracted text + OCR, use the [universal extractor](/docs/processing/extractors/universal) instead (see [Document Intelligence](/docs/tutorials/document-intelligence)).
</Note>
## 1. Create a bucket
Hold the page images. Render each PDF page to an image (PNG/JPG) and store the page image URL — the cross-modal model embeds images.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "visual-documents",
"bucket_schema": {
"properties": {
"page_image": { "type": "image" },
"document_title": { "type": "text" },
"doc_type": { "type": "string" },
"page_number": { "type": "integer" }
}
}
}'
```
<Tip>
Render PDF pages to images client-side (e.g. `pdftoppm`, `pdf2image`) — one object per page — so each page becomes an independently retrievable result with its own `page_number`.
</Tip>
## 2. Create a collection
Embed each page image with `multimodal_extractor`. Map the extractor's `image` input to your `page_image` field.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/collections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "visual-doc-index",
"source": { "type": "bucket", "bucket_ids": ["bkt_visual_documents"] },
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": { "image": "page_image" },
"field_passthrough": [
{ "source_path": "document_title" },
{ "source_path": "doc_type" },
{ "source_path": "page_number" }
]
}
}'
```
## 3. Ingest pages
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_visual_documents/objects" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"blobs": [
{ "property": "page_image", "type": "image", "data": "s3://my-bucket/filings/acme-10k-2024/page-042.png" },
{ "property": "document_title", "type": "text", "data": "Acme Corp 2024 10-K" },
{ "property": "doc_type", "type": "text", "data": "financial" },
{ "property": "page_number", "type": "text", "data": "42" }
]
}'
```
## 4. Process
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_visual_documents/batches" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "object_ids": ["obj_001"] }'
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_visual_documents/batches/{batch_id}/submit" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
## 5. Create a retriever
Search the cross-modal page embeddings with a text query.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "visual-doc-retriever",
"collection_identifiers": ["visual-doc-index"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
}
],
"final_top_k": 10
}
}
}
]
}'
```
## 6. Query
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"inputs": { "query": "segment revenue breakdown by region with year-over-year growth" },
"filters": { "field": "doc_type", "operator": "eq", "value": "financial" }
}'
```
Each result is a page, ranked by cross-modal similarity, carrying its `document_title` and `page_number` so you can deep-link to the exact page.
## When to use visual vs text retrieval
<CardGroup>
<Card title="Use visual retrieval when" icon="image">
* Pages are visually complex (tables, charts, infographics, equations)
* OCR quality is unreliable (scans, handwriting, multi-column layouts)
* Figures and diagrams carry meaning text alone cannot capture
</Card>
<Card title="Use text retrieval when" icon="font">
* Documents are born-digital and text-only (use [universal extractor](/docs/processing/extractors/universal))
* You need extracted text, NER, or summaries (see [Document Intelligence](/docs/tutorials/document-intelligence))
* You want exact-keyword matching (add a [lexical/BM25 search](/docs/retrieval/stages/feature-search#lexical-bm25-search))
</Card>
</CardGroup>
## Next steps
<CardGroup>
<Card title="Classify by layout" icon="tags" href="/enrichment/taxonomies">
Auto-classify pages (financial report, slide deck, research paper) with a taxonomy.
</Card>
<Card title="Discover layout patterns" icon="diagram-project" href="/enrichment/clusters">
Cluster page embeddings to surface visual document patterns.
</Card>
<Card title="Combine with text" icon="layer-group" href="/tutorials/document-intelligence">
Pair visual retrieval with extracted-text search for hybrid document QA.
</Card>
<Card title="Get notified" icon="bell" href="/enrichment/alerts">
Alert when new documents of a given type are indexed.
</Card>
</CardGroup>
## Further reading
* [ColPali: Efficient Document Retrieval with Vision Language Models](https://arxiv.org/abs/2407.01449) — the late-interaction approach that inspired visual document retrieval
* [ViDoRe Benchmark](https://huggingface.co/spaces/vidore/vidore-leaderboard) — visual document retrieval leaderboard
# Custom Extractor Quickstart
Source: https://docs.mixpeek.com/docs/tutorials/custom-extractor-quickstart
Build, test, and query a custom text embedding extractor
<Tip>Custom extractors extend the warehouse's Decompose layer with your own feature extraction logic. For the full reference, see [Custom Extractors](/docs/processing/custom-extractors).</Tip>
## What You'll Build
A custom text embedding extractor that:
1. Generates 128-dimensional embeddings from text (batch + real-time)
2. Validates locally with `lint` and `test` — no API key needed
3. Powers search through a retriever
<Info>
**Where this runs.** Authoring + `lint` + `test` work everywhere. The **upload/deploy** steps require a [dedicated deployment](/docs/processing/custom-extractors#availability) — the shared `api.mixpeek.com` API does not expose custom-extractor uploads. On the shared API, swap in a built-in extractor (e.g. `text_extractor`) at Step 4 and skip Step 3, or ship your extractor via [Submissions](/docs/processing/extractor-marketplace). The ingest + search steps (4–5) run on any plan.
</Info>
Set these first:
```bash theme={null}
export MIXPEEK_API_KEY="mxp_sk_..."
export MIXPEEK_NAMESPACE="ns_..."
export MIXPEEK_API_URL="https://api.mixpeek.com" # dedicated deployments use your tenant URL
```
<Note>These are the same three variables the `plugins.py` CLI reads, so the CLI and the raw `curl` examples below use one consistent convention.</Note>
## Step 1: Create Extractor Files
Create a directory `text_embed/` with three files.
### manifest.py
```python theme={null}
feature_extractor_name = "text_embed"
version = "1.0.0"
description = "Text embedding extractor"
dependencies = []
features = [
{
"feature_type": "embedding",
"feature_name": "text_embed_v1_embedding",
"embedding_dim": 128,
"distance_metric": "cosine",
}
]
# Declares real-time inference capability (query embedding for feature_search)
inference_type = "embedding"
# Skip GPU — this extractor is CPU-only
compute_profile = {"resource_type": "cpu"}
```
<Warning>
Use the exact key names: `feature_type`, `feature_name`, `embedding_dim`, `distance_metric`. Using `name`/`type`/`dimensions`/`distance` will silently produce a collection with no vector indexes.
</Warning>
### pipeline.py
```python theme={null}
import hashlib
from typing import List
import numpy as np
import pandas as pd
def text_to_embedding(text: str, dim: int = 128) -> List[float]:
"""Generate a deterministic, L2-normalized embedding from text."""
hash_bytes = hashlib.sha256(text.encode("utf-8")).digest()
seed = int.from_bytes(hash_bytes[:4], byteorder="big")
rng = np.random.default_rng(seed)
embedding = rng.standard_normal(dim).astype(np.float32)
norm = np.linalg.norm(embedding)
if norm > 0:
embedding = embedding / norm
return embedding.tolist()
class TextEmbedBatchProcessor:
def __init__(self, config=None, **kwargs):
# Custom extractors receive blob content in the 'data' column
self.text_column = "data"
self.output_column = "text_embed_v1_embedding"
self.embedding_dim = 128
def __call__(self, batch: pd.DataFrame) -> pd.DataFrame:
if batch.empty:
return batch
batch = batch.reset_index(drop=True)
texts, valid = [], []
for idx, v in enumerate(batch.get(self.text_column, [])):
text = "" if v is None else str(v)
if text.strip():
texts.append(text)
valid.append(idx)
batch[self.output_column] = None
if texts:
embs = [text_to_embedding(t, self.embedding_dim) for t in texts]
for i, orig_idx in enumerate(valid):
batch.at[orig_idx, self.output_column] = embs[i]
return batch
def build_steps(extractor_request=None, container=None,
base_steps=None, dataset_size=None, content_flags=None):
steps = list(base_steps or [])
steps.append(TextEmbedBatchProcessor())
return {"steps": steps, "prepare": lambda ds: ds}
```
<Warning>
Always read from `batch["data"]` — NOT `batch["text"]`. The `data` column holds raw text for text blobs and S3 URLs for binary blobs. The local `test` harness feeds the same `data` column, so a passing `test` matches production.
</Warning>
### realtime.py
Embeds the query at search time so it lands in the same space as your indexed vectors.
```python theme={null}
import hashlib
import numpy as np
class InferenceService:
async def run_inference(self, inputs: dict, parameters: dict) -> dict:
text = inputs.get("text", "")
h = hashlib.sha256(text.encode("utf-8")).digest()
rng = np.random.default_rng(int.from_bytes(h[:4], "big"))
v = rng.standard_normal(128).astype(np.float32)
n = np.linalg.norm(v)
return {"embedding": (v / n if n else v).tolist()}
```
## Step 2: Validate Locally (no API key)
```bash theme={null}
python server/scripts/api/plugins.py lint text_embed
python server/scripts/api/plugins.py test text_embed
```
`lint` validates your manifest + runs the security scanner. `test` runs the pipeline through real Ray Data `map_batches` and confirms your output column is populated. Both are offline.
## Step 3: Deploy (dedicated infrastructure)
<Note>
Upload/deploy is only available on a [dedicated deployment](/docs/processing/custom-extractors#availability) — full HTTP contract in the [Custom Extractor API](/docs/processing/custom-extractor-api) reference. On the shared API, skip to Step 4 with a built-in extractor, or use [Submissions](/docs/processing/extractor-marketplace).
</Note>
The CLI does this for you (`plugins.py push` then `deploy`). Raw HTTP (custom extractors are addressed as `/plugins` on this surface):
```bash theme={null}
zip -r text_embed.zip text_embed/
UPLOAD=$(curl -s -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/plugins/uploads" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "Content-Type: application/json" \
-d '{"name":"text_embed","version":"1.0.0","file_size_bytes":5000}')
UPLOAD_ID=$(echo "$UPLOAD" | jq -r '.upload_id')
PRESIGNED_URL=$(echo "$UPLOAD" | jq -r '.presigned_url')
curl -s -X PUT "$PRESIGNED_URL" -H "Content-Type: application/zip" --data-binary @text_embed.zip
curl -s -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/plugins/uploads/$UPLOAD_ID/confirm" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "Content-Type: application/json" -d '{}'
curl -s -X POST "$MIXPEEK_API_URL/v1/namespaces/$MIXPEEK_NAMESPACE/plugins/text_embed_1_0_0/deploy" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
## Step 4: Ingest Data
<Note>
Buckets, collections, retrievers, and batches are **top-level** resources keyed by the **`X-Namespace` header** — not nested under `/namespaces/{ns}/`.
</Note>
```bash theme={null}
H=(-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $MIXPEEK_NAMESPACE" -H "Content-Type: application/json")
B="https://api.mixpeek.com/v1"
# Bucket
BUCKET_ID=$(curl -s -X POST "$B/buckets" "${H[@]}" \
-d '{"bucket_name":"articles","bucket_schema":{"properties":{"text":{"type":"text"}}}}' | jq -r '.bucket_id')
# Object
curl -s -X POST "$B/buckets/$BUCKET_ID/objects" "${H[@]}" \
-d '{"blobs":[{"property":"text","type":"text","data":"Quantum computing uses qubits to perform calculations exponentially faster than classical computers."}]}'
# Collection — use "text_embed" if deployed, or a built-in like "text_extractor" on the shared API
COLLECTION_ID=$(curl -s -X POST "$B/collections" "${H[@]}" -d "{
\"collection_name\":\"text_embed_articles\",
\"source\":{\"type\":\"bucket\",\"bucket_ids\":[\"$BUCKET_ID\"]},
\"feature_extractor\":{\"feature_extractor_name\":\"text_embed\",\"version\":\"1.0.0\",\"input_mappings\":{\"text\":\"text\"}}
}" | jq -r '.collection_id')
# Batch is two-step: create (objects + collections) then submit
OBJIDS=$(curl -s "$B/buckets/$BUCKET_ID/objects" "${H[@]}" | jq -c '[.results[].object_id]')
BATCH_ID=$(curl -s -X POST "$B/buckets/$BUCKET_ID/batches" "${H[@]}" \
-d "{\"batch_name\":\"run-1\",\"object_ids\":$OBJIDS,\"collection_ids\":[\"$COLLECTION_ID\"]}" | jq -r '.batch_id')
curl -s -X POST "$B/buckets/$BUCKET_ID/batches/$BATCH_ID/submit" "${H[@]}" \
-d "{\"collection_ids\":[\"$COLLECTION_ID\"]}"
# Poll until COMPLETED
while true; do
S=$(curl -s "$B/buckets/$BUCKET_ID/batches/$BATCH_ID" "${H[@]}" | jq -r '.status')
echo "batch: $S"; case "$S" in COMPLETED|COMPLETED_WITH_ERRORS|FAILED|CANCELED) break;; esac; sleep 8
done
```
## Step 5: Create a Retriever and Search
<Warning>
**First-run cold start.** On a cold engine the embedding model can take **several minutes** to load. The batch poll in Step 4 waits for the ingest side, but the **first `execute` immediately after may return 0 results** (with `status` `completed` *or* `degraded`) while the **query-side** model warms up. This is expected on the first call only — **retry after a few seconds** and subsequent searches return ranked results. A warm namespace responds immediately.
</Warning>
```bash theme={null}
RETRIEVER_ID=$(curl -s -X POST "$B/retrievers" "${H[@]}" -d "{
\"retriever_name\":\"text_search\",
\"collection_identifiers\":[\"$COLLECTION_ID\"],
\"input_schema\":{\"query\":{\"type\":\"text\",\"required\":true}},
\"stages\":[{\"stage_name\":\"semantic\",\"stage_type\":\"filter\",\"config\":{\"stage_id\":\"feature_search\",\"parameters\":{
\"searches\":[{\"feature_uri\":\"mixpeek://text_embed@1.0.0/text_embed_v1_embedding\",\"query\":{\"input_mode\":\"text\",\"value\":\"{{INPUT.query}}\"},\"top_k\":10}],
\"final_top_k\":10,\"fusion\":\"rrf\"}}}]
}" | jq -r '.retriever_id')
curl -s -X POST "$B/retrievers/$RETRIEVER_ID/execute" "${H[@]}" \
-d '{"inputs":{"query":"quantum computing"}}' | jq '.documents[:3]'
```
<Note>
The execute body is `{"inputs": {"query": "..."}}`. The `feature_uri` must match your extractor + feature name (for a built-in, fetch it from `GET /v1/namespaces/$MIXPEEK_NAMESPACE/extractors/{extractor_id}`).
</Note>
## Next Steps
* [Custom Extractors](/docs/processing/custom-extractors) — full manifest, SDK, security, and platform-services reference
* [Model Registry](/docs/processing/model-registry) — load HuggingFace or your own fine-tuned weights
* [Taxonomies](/docs/enrichment/taxonomies) and [Clusters](/docs/enrichment/clusters) — auto-classify and group your embeddings
* [Alerts](/docs/enrichment/alerts) and [Webhooks](/docs/operations/webhooks) — monitor new content and processing events
# Document Intelligence
Source: https://docs.mixpeek.com/docs/tutorials/document-intelligence
Extract and query data from PDFs, images, and scanned documents
<Tip>Document intelligence uses the warehouse's decompose layer to parse text and layout from PDFs and scanned documents, then makes them queryable through retrieval. Every code block below uses the real API field names.</Tip>
<Frame>
<img alt="Document Intelligence Pipeline" />
</Frame>
## How It Works
When you ingest a document, the `universal_extractor` runs a multi-stage pipeline:
1. **Content extraction** — text is parsed from native PDFs, with OCR fallback for scanned pages.
2. **Chunking** — documents are split into searchable segments.
3. **Embedding** — each document is embedded for semantic search.
4. **Indexing** — segments are stored with metadata for filtered vector search.
At query time, the retriever runs semantic search over the document embeddings and can filter by metadata such as document type.
## Feature Extractors
| Extractor | Use For |
| ------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `universal_extractor@v1` | Parse any file (PDF, image, scanned doc), OCR fallback, and produce a Gemini embedding for semantic search |
| `text_extractor@v1` | Text embeddings, NER, and summarization over already-extracted text |
<Note>
Mixpeek does not have separate `pdf_extractor` / `table_extractor` extractors. The `universal_extractor` ingests arbitrary documents (PDFs and scans) and produces a searchable embedding.
</Note>
## 1. Create a bucket
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "contracts",
"bucket_schema": {
"properties": {
"document_url": { "type": "pdf" },
"document_type": { "type": "string" },
"contract_date": { "type": "datetime" }
}
}
}'
```
## 2. Create a collection
Use `universal_extractor` to parse each document and produce a searchable embedding. Map the extractor's `content` input to your bucket's `document_url` field.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/collections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "contracts-text",
"source": { "type": "bucket", "bucket_ids": ["bkt_contracts"] },
"feature_extractor": {
"feature_extractor_name": "universal_extractor",
"version": "v1",
"input_mappings": { "content": "document_url" },
"field_passthrough": [
{ "source_path": "document_type" },
{ "source_path": "contract_date" }
]
}
}'
```
## 3. Ingest documents
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_contracts/objects" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"key_prefix": "/2025/agreements",
"blobs": [
{ "property": "document_url", "type": "pdf", "data": "s3://my-bucket/contracts/vendor-001.pdf" },
{ "property": "document_type", "type": "text", "data": "vendor_agreement" }
]
}'
```
## 4. Process
```bash theme={null}
# Create a batch, then submit it; poll the returned task until COMPLETED
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_contracts/batches" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "object_ids": ["obj_001", "obj_002"] }'
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_contracts/batches/{batch_id}/submit" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
See [Monitoring ingestion](/docs/processing/tasks) for the task-polling loop.
## 5. Create a retriever
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "contract-search",
"collection_identifiers": ["contracts-text"],
"input_schema": {
"query": { "type": "text", "required": true }
},
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://universal_extractor@v1/gemini-embedding-2",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
}
],
"final_top_k": 20
}
}
}
]
}'
```
## 6. Query
Filter by document type at execution time with the `filters` field:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"inputs": { "query": "termination clauses with 30-day notice" },
"filters": {
"field": "document_type",
"operator": "eq",
"value": "vendor_agreement"
}
}'
```
## Multi-page assembly
Retrieve all segments from a source document using lineage:
```bash theme={null}
curl -sS "$MP_API_URL/v1/documents/{document_id}/lineage" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
## Next steps
<CardGroup>
<Card title="Classify documents" icon="tags" href="/enrichment/taxonomies">
Auto-classify documents by type (contract, invoice, NDA) with a taxonomy.
</Card>
<Card title="Discover themes" icon="diagram-project" href="/enrichment/clusters">
Cluster document embeddings to surface recurring contract patterns.
</Card>
<Card title="Get notified" icon="bell" href="/enrichment/alerts">
Trigger alerts when new documents match a query.
</Card>
<Card title="Add NER & summaries" icon="font" href="/processing/extractors/text">
Pair with `text_extractor` for named-entity recognition and summaries.
</Card>
</CardGroup>
# Build a Feedback Loop
Source: https://docs.mixpeek.com/docs/tutorials/feedback-loop
Start with static fusion weights, capture interaction signals, and let Thompson Sampling learn optimal weights automatically
<Tip>This tutorial walks through the full lifecycle: static weights → interaction capture → learned fusion → convergence monitoring. You don't need interaction data to start — the system gracefully degrades to uniform weights with zero signal.</Tip>
## What You'll Build
A search retriever that starts with manually tuned weights and progressively learns the optimal blend of features from user behavior. By the end, your retriever adapts per-user (or per-segment) without manual tuning.
**Prerequisites:** A namespace with at least two collections producing different embedding types (e.g., text + multimodal). See [Semantic Search](/docs/tutorials/semantic-search) or [Video Understanding](/docs/tutorials/video-understanding) to set those up first.
## 1. Start with Static Weights
Begin with `weighted` fusion. This gives you a deterministic baseline to measure against later.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"retriever_name": "product-search",
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100,
"weight": 0.6
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100,
"weight": 0.4
}
],
"fusion": "weighted",
"final_top_k": 25
}
}
}
]
}'
```
```python Python SDK theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="your-api-key")
retriever = client.retrievers.create(
retriever_name="product-search",
stages=[{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100,
"weight": 0.6
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100,
"weight": 0.4
}
],
"fusion": "weighted",
"final_top_k": 25
}
}
}]
)
```
</CodeGroup>
Run searches against this retriever and record the result quality. These static-weight results are your baseline.
## 2. Instrument Your Application with Interaction Signals
Before switching to learned fusion, you need to emit signals. Add interaction tracking wherever users engage with search results.
<CodeGroup>
```javascript Client-Side (Browser) theme={null}
document.querySelectorAll('.search-result').forEach((el, index) => {
el.addEventListener('click', () => {
fetch(`${MP_API_URL}/v1/retrievers/interactions`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${MP_API_KEY}`,
'X-Namespace': MP_NAMESPACE
},
body: JSON.stringify({
feature_id: el.dataset.documentId,
interaction_type: ['click'],
position: index,
metadata: { query: currentQuery },
user_id: userId,
session_id: sessionId
})
});
});
});
```
```python Server-Side (Python) theme={null}
client.retrievers.create_interaction(
feature_id="doc_product_789",
interaction_type=["purchase"],
position=2,
metadata={
"query": original_query,
"order_value": 49.99
},
user_id=user_id,
session_id=session_id
)
```
</CodeGroup>
<Tip>
**Shortcut: `create_interaction_from_result()`** — pass the full execute response and a position, and the SDK extracts `feature_id`, `execution_id`, `retriever_id`, and `feature_uri` automatically:
```python theme={null}
results = client.retrievers.execute(retriever_id, inputs={"query": "earbuds", "user_id": "user_456"})
# User clicked the first result
client.retrievers.create_interaction_from_result(results, position=0, user_id="user_456")
# User purchased the third result
client.retrievers.create_interaction_from_result(results, position=2, interaction_type=["purchase"], user_id="user_456")
```
</Tip>
<Warning>
**Always include `position`.** It is recorded for analytics and evaluation — helping you understand which result positions drive engagement. Position is not currently used in reward computation, but is required for accurate NDCG and other rank-aware metrics.
</Warning>
**Which signals to capture depends on your domain:**
| Domain | Primary Signals | Why |
| ----------------- | ---------------------------------------- | ------------------------------------------------------ |
| E-commerce | `purchase`, `add_to_cart`, `click` | Conversion is the strongest relevance indicator |
| Media / Video | `long_view`, `click`, `share` | Watch completion > click for engagement |
| Enterprise Search | `positive_feedback`, `click`, `bookmark` | Clicks may be obligatory; explicit feedback is clearer |
| Content Matching | `click`, `positive_feedback`, `skip` | Editor accept/reject on matched content |
See the [Signal Strength Matrix](/docs/retrieval/interactions#signal-strength-matrix) for the full list of 17 signal types and how they're weighted.
## 3. Switch to Learned Fusion
Once interactions are flowing, update the retriever to use `learned` fusion. You can do this at any time — even with zero interactions (it falls back to uniform weights).
<CodeGroup>
```bash cURL theme={null}
curl -X PATCH "$MP_API_URL/v1/retrievers/{retriever_id}" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"stages": [
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"fusion": "learned",
"final_top_k": 25
}
}
}
]
}'
```
```python Python SDK theme={null}
client.retrievers.update(
retriever_id=retriever.retriever_id,
stages=[{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": "{{INPUT.query}}",
"top_k": 100
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": "{{INPUT.query}}",
"top_k": 100
}
],
"fusion": "learned",
"final_top_k": 25
}
}
}]
)
```
</CodeGroup>
No `weights` field needed — the system samples weights from Beta distributions on every query.
## 4. Understand Cold Start Behavior
With learned fusion enabled, the system handles sparse data automatically through hierarchical fallback:
| User Interactions | What Happens | Effective Behavior |
| ------------------------------------ | ------------------------------------------- | ---------------------------------------- |
| **0** | Beta(1,1) = uniform prior for all features | Equivalent to RRF |
| **\< min\_interactions** (default 5) | Falls back to demographic or global weights | Shared weights from segment or all users |
| **>= min\_interactions** | Personal weights from this user's history | Per-user individually tuned weights |
The threshold for trusting personal weights is the `min_interactions` parameter (default **5**). Below that, the system falls back up the hierarchy: personal → demographic → global → uniform prior.
<Note>
If you pass `user_id` in your search requests, the system tracks personal-level weights automatically. Without `user_id`, you still get global-level learning — the system learns which features are better *overall*, just not per-user.
</Note>
## 5. Execute Searches with User Context
Pass `user_id` on every search request so the bandit can build per-user weight profiles:
<CodeGroup>
```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{
"inputs": {
"query": "wireless noise canceling earbuds",
"user_id": "user_456"
}
}'
```
```python Python SDK theme={null}
results = client.retrievers.execute(
retriever_id=retriever.retriever_id,
inputs={"query": "wireless noise canceling earbuds", "user_id": "user_456"}
)
```
</CodeGroup>
Behind the scenes, the Thompson Sampler:
1. Looks up `user_456`'s interaction history in ClickHouse
2. Computes Beta(α, β) per feature: `α = 1 + clicks`, `β = 1 + (impressions - clicks)`
3. Samples a weight from each Beta distribution
4. Normalizes weights to sum to 1
5. Executes each feature search and fuses results using the sampled weights
If `user_456` has consistently clicked text-matched results over image-matched ones, the text feature's Beta distribution is peaked higher — so sampled weights skew toward text.
## 6. Monitor Convergence
Check whether the learned weights are stabilizing using the analytics endpoint:
```bash theme={null}
curl "$MP_API_URL/v1/analytics/retrievers/{retriever_id}/signals?signal_type=learned_weights&hours=168" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
**What to look for:**
* **Weights stabilizing** — variance decreasing over time means the system is converging
* **Feature dominance** — if one feature's weight approaches 1.0, the other features may not be contributing value
* **Per-segment differences** — different user segments learning different weights validates that personalization is working
## 7. Measure Improvement
Use [evaluations](/docs/retrieval/evaluations) to compare learned fusion against your static baseline. Create an evaluation with the same queries and judge whether learned fusion produces better-ranked results.
The key metrics to track:
| Metric | What It Tells You |
| ------------------------ | -------------------------------------------------- |
| **CTR at position 1-3** | Are top results more clickable? |
| **Mean Reciprocal Rank** | Is the first relevant result appearing earlier? |
| **Interaction rate** | Are users engaging more overall? |
| **Weight variance** | Is the system still exploring or has it converged? |
<Tip>
**Recommended rollout:** Run learned fusion on 10% of traffic alongside your static baseline. Compare metrics over 1-2 weeks. If learned fusion wins or ties, ramp to 100%.
</Tip>
## When to Use Each Strategy
| Starting Point | Recommendation |
| ------------------------------------------- | ------------------------------------------------------------------- |
| No interaction data, launching today | Start with `rrf` — strong default, no tuning needed |
| Domain expert knows feature importance | Start with `weighted` — encode expert knowledge as initial weights |
| Have 100+ interactions flowing | Switch to `learned` — let the data decide |
| Multiple user segments with different needs | `learned` with `user_id` — per-segment and per-user personalization |
| Need deterministic, reproducible results | Stay with `weighted` — learned fusion is stochastic by design |
## 8. Monitor Personalization
Once learned fusion is running, check per-user weights to verify personalization is working:
<CodeGroup>
```bash cURL theme={null}
curl "$MP_API_URL/v1/retrievers/$RETRIEVER_ID/learned-fusion/weights/user_123" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python SDK theme={null}
weights = client.retrievers.get_user_weights(
retriever_id=retriever.retriever_id,
user_id="user_123"
)
print(weights["context_level"]) # "personal", "demographic", or "global"
print(weights["sampled_weights"]) # {"text@v1/embed": 0.72, "image@v1/embed": 0.28}
```
</CodeGroup>
A user with enough interactions shows `context_level: "personal"`. New users fall back to `"demographic"` or `"global"` until they cross the `min_interactions` threshold.
## 9. Configure Reward Signals
Not all interactions are equal. Customize the `reward_map` to weight different signals:
```json theme={null}
{
"learning_config": {
"context_features": ["INPUT.user_id"],
"reward_map": {
"click": 1.0,
"purchase": 3.0,
"add_to_cart": 2.0,
"positive_feedback": 2.0,
"negative_feedback": -2.0,
"skip": -1.0
}
}
}
```
Negative values act as penalties -- `negative_feedback` actively pushes the weight *away* from the feature that produced the disliked result. See the [Reward Signals reference](/docs/retrieval/reward-signals) for all 17 signal types and their defaults.
## 10. Safe Rollout
Before rolling learned fusion to all traffic, use the built-in rollout controls:
1. **Shadow mode** -- compute learned weights but serve static results. Compare offline.
2. **Traffic splitting** -- route a percentage of users to learned fusion (`rollout_pct: 10`).
3. **Kill switch** -- instantly disable learned fusion if something goes wrong.
4. **Per-user opt-out** -- exclude internal test accounts or specific users.
```bash theme={null}
# Enable shadow mode (logs learned weights, serves static results)
curl -X PATCH "$MP_API_URL/v1/retrievers/$RETRIEVER_ID" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{"stages": [{"stage_name": "feature_search", "stage_type": "filter", "config": {"stage_id": "feature_search", "parameters": {"fusion": "learned", "learning_config": {"shadow_mode": true, "rollout_pct": 10.0}}}}]}'
```
See the full [Rollout Guide](/docs/retrieval/auto-tune-rollout) for step-by-step rollout instructions.
## 11. Evaluate Learned vs Static
Generate an evaluation dataset from real interactions and compare learned fusion against your static baseline:
<CodeGroup>
```bash cURL theme={null}
# Generate eval dataset from interaction history
curl -X POST "$MP_API_URL/v1/retrievers/$RETRIEVER_ID/evaluations/generate-from-interactions" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{"dataset_name": "learned-vs-static", "min_interactions": 5}'
# Start the evaluation — returns 202 with an evaluation_id
curl -X POST "$MP_API_URL/v1/retrievers/$RETRIEVER_ID/evaluations" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-d '{"dataset_name": "learned-vs-static", "evaluation_config": {"k_values": [1, 5, 10, 20]}}'
# Poll until status is completed, then read NDCG, MRR, Precision
curl "$MP_API_URL/v1/retrievers/$RETRIEVER_ID/evaluations/$EVALUATION_ID" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
```python Python SDK theme={null}
import httpx
# Generate eval dataset from interaction history
resp = httpx.post(
f"{base_url}/v1/retrievers/{retriever_id}/evaluations/generate-from-interactions",
headers={"Authorization": f"Bearer {api_key}", "X-Namespace": namespace},
json={"dataset_name": "learned-vs-static", "min_interactions": 5},
)
dataset = resp.json()
# Start the evaluation — this returns 202, metrics are not ready yet
resp = httpx.post(
f"{base_url}/v1/retrievers/{retriever_id}/evaluations",
headers={"Authorization": f"Bearer {api_key}", "X-Namespace": namespace},
json={
"dataset_name": dataset["dataset_name"],
"evaluation_config": {"k_values": [1, 5, 10, 20]},
},
)
evaluation_id = resp.json()["evaluation_id"]
# Poll until status is completed
resp = httpx.get(
f"{base_url}/v1/retrievers/{retriever_id}/evaluations/{evaluation_id}",
headers={"Authorization": f"Bearer {api_key}", "X-Namespace": namespace},
)
result = resp.json()
print(result["overall_metrics"]["ndcg_at_10"])
```
</CodeGroup>
<Tip>
Run evaluations on a regular cadence (e.g., weekly or after every 1,000 interactions). If learned fusion regresses vs. static, the kill switch gives you an instant rollback path.
</Tip>
## Next Steps
<CardGroup>
<Card title="Auto-Tune Concept" icon="brain" href="/retrieval/auto-tune">
Full overview of the auto-tune system — how it works, when to use it, and configuration options.
</Card>
<Card title="Learned Fusion Deep Dive" icon="flask" href="/relevance/learned-fusion">
Thompson Sampling internals, session adaptation, temporal decay, weight clamping, and exploration decay.
</Card>
<Card title="Reward Signals" icon="signal" href="/retrieval/reward-signals">
All 17 interaction types, reward weighting, negative signals, and position bias correction.
</Card>
<Card title="Rollout Guide" icon="shield" href="/retrieval/auto-tune-rollout">
Shadow mode, traffic splitting, kill switch, per-user opt-out, and preference reset.
</Card>
<Card title="Fusion Strategies" icon="scale-balanced" href="/relevance/fusion-strategies">
Compare all 5 fusion strategies: RRF, DBSF, Weighted, Max, and Learned.
</Card>
<Card title="Evaluations" icon="chart-line" href="/retrieval/evaluations">
Set up benchmarks to measure whether learned fusion is improving result quality.
</Card>
</CardGroup>
# Ingest Video from S3
Source: https://docs.mixpeek.com/docs/tutorials/ingest-video-from-s3
Connect an S3 bucket, sync video files automatically, and make them searchable end-to-end
This guide walks the full path from an S3 bucket of videos to a working search index: create a storage connection, point a sync at your S3 prefix, and let Mixpeek ingest and process new files automatically.
<Note>
This is the automated/continuous path. To upload a single file by URL instead, see [Video Understanding](/docs/tutorials/video-understanding). For the AWS IAM policy and role setup, see [AWS S3](/docs/integrations/object-storage/s3).
</Note>
## Prerequisites
* An S3 bucket containing video files.
* AWS credentials — either an access key pair or an IAM role ARN (see [AWS S3](/docs/integrations/object-storage/s3) for the IAM policy).
* A Mixpeek API key and namespace.
## 1. Create a storage connection
A connection stores your S3 credentials once and can be reused across buckets. Credentials are validated before the connection is saved (`test_before_save` defaults to `true`).
<CodeGroup>
```bash Access key theme={null}
curl -sS -X POST "$MP_API_URL/v1/organizations/connections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "prod-video-s3",
"provider_type": "s3",
"provider_config": {
"provider_type": "s3",
"region": "us-east-1",
"credentials": {
"type": "access_key",
"access_key_id": "AKIA...",
"secret_access_key": "..."
}
}
}'
```
```bash IAM role (recommended) theme={null}
curl -sS -X POST "$MP_API_URL/v1/organizations/connections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "prod-video-s3",
"provider_type": "s3",
"provider_config": {
"provider_type": "s3",
"region": "us-east-1",
"credentials": {
"type": "iam_role",
"role_arn": "arn:aws:iam::123456789012:role/mixpeek-read",
"external_id": "your-external-id"
}
}
}'
```
</CodeGroup>
The response includes a `connection_id` (e.g. `conn_abc123`). Verify connectivity any time with:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/organizations/connections/conn_abc123/test" \
-H "Authorization: Bearer $MP_API_KEY"
```
<Tip>
Connections are org-level and not namespace-scoped — no `X-Namespace` header is needed for connection calls. Buckets, syncs, and collections below are namespace-scoped, so they require `X-Namespace`.
</Tip>
## 2. Create a bucket
The bucket declares the schema for each synced object. Use the Mixpeek type `video` for the file field.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "video-catalog",
"bucket_schema": {
"properties": {
"video_url": { "type": "video" }
}
}
}'
```
## 3. Create a collection
One collection with `multimodal_extractor` produces both visual (`vertex_multimodal_embedding`) and speech (`multilingual_e5_large_instruct_v1`) features per segment.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/collections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "video-moments",
"source": { "type": "bucket", "bucket_ids": ["bkt_videos"] },
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": { "video": "video_url" }
}
}'
```
## 4. Sync the S3 prefix into the bucket
A sync watches an S3 path and ingests matching files. With `sync_mode: "continuous"`, Mixpeek polls for new files and ingests them automatically; `initial_only` runs a single backfill.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_videos/syncs" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"connection_id": "conn_abc123",
"source_path": "s3://my-bucket/videos/",
"sync_mode": "continuous",
"polling_interval_seconds": 300,
"batch_size": 50,
"file_filters": { "extensions": [".mp4", ".mov"] },
"schema_mapping": {
"mappings": {
"video_url": {
"target_type": "blob",
"source": { "type": "file" },
"blob_type": "video",
"blob_property": "video_url"
}
}
}
}'
```
| Field | Default | Description |
| -------------------------- | ------------ | ---------------------------------------------------------------- |
| `connection_id` | required | The storage connection from step 1. |
| `source_path` | required | S3 URI/prefix to watch (e.g. `s3://my-bucket/videos/`). |
| `sync_mode` | `continuous` | `continuous` polls for new files; `initial_only` backfills once. |
| `polling_interval_seconds` | `300` | How often to check for new files (30–900). |
| `batch_size` | `50` | Files processed per batch (1–100). |
| `file_filters` | — | Restrict by extension/pattern. |
| `schema_mapping` | — | Map each file to a bucket property. |
| `skip_duplicates` | `true` | Skip files already ingested (by source ID). |
The response includes a `sync_config_id` (e.g. `sync_xyz`).
## 5. Run and monitor the sync
`continuous` syncs start on their own, but you can trigger the first run immediately and watch its job:
```bash theme={null}
# Trigger a run now
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_videos/syncs/sync_xyz/trigger" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
# Check sync metrics (files discovered, ingested, failed)
curl -sS "$MP_API_URL/v1/buckets/bkt_videos/syncs/sync_xyz/metrics" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
| Operation | Endpoint |
| ------------------ | ------------------------------------------------------------- |
| Trigger now | `POST /v1/buckets/{bucket_id}/syncs/{sync_config_id}/trigger` |
| Pause / resume | `POST .../pause` · `POST .../resume` |
| Job status | `GET .../jobs/{sync_job_id}` |
| Failed files (DLQ) | `POST .../dlq` |
<Note>
The sync ingests files as objects and (unless `skip_batch_submission` is set) submits them for processing automatically — so your `multimodal_extractor` collection populates without any extra step. Vectors are searchable within 10–30s of each batch completing.
</Note>
## 6. Search
Create a retriever over the collection and execute it:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "video-search",
"collection_identifiers": ["video-moments"],
"input_schema": { "query": { "type": "text", "required": true } },
"stages": [
{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100, "weight": 0.6
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100, "weight": 0.4
}
],
"final_top_k": 20,
"fusion": "weighted"
}
}
}
]
}'
```
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "inputs": { "query": "people discussing electric vehicles" } }'
```
## Keep it fresh
Because the sync is `continuous`, new videos dropped into the S3 prefix are ingested and indexed automatically. To re-run clustering or enrichment on a schedule as new content lands, see [Triggers](/docs/platform/triggers).
## Other storage providers
The same connection → bucket → sync flow works for every supported provider — swap `provider_type` and `provider_config`:
<CardGroup>
<Card title="Google Cloud Storage" href="/integrations/object-storage/gcs" />
<Card title="Azure Blob" href="/integrations/object-storage/azure-blob" />
<Card title="Cloudflare R2" href="/integrations/object-storage/r2" />
</CardGroup>
# Add a New Understanding to Existing Content
Source: https://docs.mixpeek.com/docs/tutorials/reprocess-existing-content
Run a new extractor over an already-ingested corpus — scoped, cost-safe, and priced before you run — without re-uploading or re-paying for existing work
You've ingested a corpus and built search over it. Now you want to add another
understanding — a new extractor, a new model version, a classifier — over the
**same content**, without re-uploading anything and without re-paying for the
extraction you already ran.
This is a first-class workflow. Three pieces make it cost-safe by default:
1. **Batch scoping** — `collection_ids` on a batch runs *only* that collection's
extractor over the bucket.
2. **Dedup protection** — `dedup_strategy: "skip"` (the default) never re-runs
extraction for content a collection has already processed, even if you
forget to scope.
3. **Pre-flight pricing** — `POST /batches/{id}/estimate-cost` is a true dry
run: it tells you exactly what would run and what it costs *before* you
commit, including an `already_extracted_count` showing the work you will
NOT be re-billed for.
<Note>
Everything below operates on content already in a bucket. Buckets hold your
source objects once; collections are independent processing pipelines over
them. That separation is what makes iterating on models cheap.
</Note>
## 1. Create a second collection over the same bucket
Your existing collection keeps serving queries untouched. The new
understanding gets its own collection pointed at the same bucket:
<CodeGroup>
```python Python theme={null}
new_col = client.collections.create(
collection_name="scripts-v2-understanding",
source={"type": "bucket", "bucket_id": bucket_id},
features=["text_search"], # or a feature key from GET /v1/collections/features
)
```
```javascript JavaScript theme={null}
const newCol = await client.collections.create({
collection_name: "scripts-v2-understanding",
source: { type: "bucket", bucket_id: bucketId },
features: ["text_search"],
});
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/collections" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "scripts-v2-understanding",
"source": {"type": "bucket", "bucket_id": "'$BUCKET_ID'"},
"features": ["text_search"]
}'
```
</CodeGroup>
If the extractor behind your chosen feature isn't registered on the namespace
yet, the platform registers it for you at create time. (On older deployments a
422 tells you the exact `PATCH /v1/namespaces` body to run first.)
## 2. Create a batch scoped to the new collection
Scope the batch with `collection_ids` so only the new collection's pipeline
runs:
<CodeGroup>
```python Python theme={null}
batch = client.batches.create(
bucket_id=bucket_id,
object_ids=object_ids, # or filters selecting the slice you want
collection_ids=[new_col.collection_id],
)
```
```javascript JavaScript theme={null}
const batch = await client.batches.create({
bucket_id: bucketId,
object_ids: objectIds,
collection_ids: [newCol.collection_id],
});
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/batches" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE" \
-H "Content-Type: application/json" \
-d '{"object_ids": ["obj_..."], "collection_ids": ["'$NEW_COLLECTION_ID'"]}'
```
</CodeGroup>
## 3. Price it before you run it
`estimate-cost` is a dry run — nothing processes, nothing is charged. (For quoting *planned* ingestion in dollars by modality and feature before you even build the batch, use [`POST /v1/organizations/billing/estimate`](/docs/platform/billing#estimate-before-you-run) — same rating engine that bills you.)
<CodeGroup>
```python Python theme={null}
estimate = client.batches.estimate_cost(bucket_id=bucket_id, batch_id=batch.batch_id)
print(estimate)
# {
# "object_count": 8,
# "already_extracted_count": 0, # nothing skipped — all new work
# "estimated_credits": 10, # legacy field: internal ledger unit (1 credit = $0.001 → $0.01)
# "extractors": ["text_extractor"]
# }
```
```javascript JavaScript theme={null}
const estimate = await client.batches.estimateCost(bucketId, batch.batch_id);
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/batches/$BATCH_ID/estimate-cost" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: $NAMESPACE"
```
</CodeGroup>
**The trap probe worth knowing:** if you estimate an *unscoped* batch over a
bucket that an existing collection already processed, the estimate shows that
prior work under `already_extracted_count` and prices it at zero — the default
`dedup_strategy: "skip"` refuses to re-run (and re-bill) extraction whose
inputs haven't changed. Your existing collection's GPU spend is protected even
when you forget to scope.
## 4. Submit, then verify only new work ran
Submit the batch and let it complete. Two checks confirm the cost boundary
held:
* The **new** collection has documents; the **old** collection's documents show
no changed `updated_at`.
* The batch record carries its own cost:
`batch.cost → {"credits_consumed": 10, "cost_usd": 0.01}` — `cost_usd` is the
number that matters (the `credits_consumed` field is the internal ledger unit,
1 credit = $0.001). Batches bill at least the $0.01 minimum.
For a multi-batch rollout, sum the `cost_usd` of the rollout's batches —
each batch is its own attribution record.
## 5. Compare the two understandings side by side
Both collections index the same source objects, so retrieval comparisons need
no second corpus copy: run the same query against each collection (or one
retriever spanning both) and compare. For systematic comparison, build a small
[evaluation](/docs/retrieval/evaluations) dataset once and run it against both.
When a new *version* of an extractor ships (versions coexist in the catalog —
e.g. v1 and v2 side by side), the same pattern applies: pin the new version in
a new collection, roll a scoped batch, compare, and cut over when the numbers
say so. For whole-namespace model swaps, use
[model migration](/docs/processing/model-migration) instead.
## 6. Re-run only failures
If some objects fail, you don't resubmit the batch:
* `GET /v1/buckets/{bucket_id}/batches/{batch_id}/failed-documents` lists
failures with per-tier detail and what's retryable.
* `POST /v1/buckets/{bucket_id}/batches/{batch_id}/retry` re-runs just those.
## Next steps
<CardGroup>
<Card title="Feature Extractors" icon="microchip" href="/processing/feature-extractors">
How a single extractor is configured — inputs, outputs, inference cache.
</Card>
<Card title="Model Migration" icon="arrows-rotate" href="/processing/model-migration">
Swap a namespace's embedding model wholesale with validation and dry-run.
</Card>
<Card title="Multi-Tier Extraction" icon="diagram-project" href="/processing/multi-tier-extractors">
Chain collections into a DAG — transcribe, then embed, then classify.
</Card>
<Card title="Custom Extractor Quickstart" icon="rocket" href="/tutorials/custom-extractor-quickstart">
Build and test your own extractor end-to-end.
</Card>
</CardGroup>
# Reverse Media Search
Source: https://docs.mixpeek.com/docs/tutorials/reverse-search
Find visually similar content using images or videos as your search query
<Card title="Search your own images and video" icon="image" href="https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=docs&utm_medium=docs_reverse_search&utm_campaign=managed_onramp">
Create a namespace, upload a few assets, and run a visual similarity search in minutes — no API key setup required.
</Card>
<Tip>Reverse media search uses the warehouse's Decompose layer to extract visual features, then queries them in the Reassemble layer. See [Multi-Stage Retrieval](/docs/retrieval/multi-stage-deep-dive) for composing pipelines.</Tip>
## Input Methods
| Method | Example | Speed |
| ---------------------- | ------------------------------------------ | ------- |
| Pre-computed embedding | `{"embedding": [0.1, 0.2, ...]}` | Fastest |
| Image URL | `{"url": "https://example.com/img.jpg"}` | Fast |
| Video URL | `{"url": "s3://bucket/video.mp4"}` | Medium |
| Base64 | `{"base64": "data:image/jpeg;base64,..."}` | Fast |
## 1. Create a Bucket
```bash theme={null}
POST /v1/buckets
{
"bucket_name": "visual-assets",
"bucket_schema": {
"properties": {
"asset_url": { "type": "url", "required": true },
"brand": { "type": "text" },
"campaign_id": { "type": "text" }
}
}
}
```
## 2. Create a Collection
**For images:**
```bash theme={null}
POST /v1/collections
{
"collection_name": "product-images",
"source": { "type": "bucket", "bucket_ids": ["bkt_visual_assets"] },
"feature_extractor": {
"feature_extractor_name": "image_extractor",
"version": "v1",
"input_mappings": { "image_url": "asset_url" },
"parameters": { "model": "clip-vit-large-patch14" },
"field_passthrough": [
{ "source_path": "brand" },
{ "source_path": "campaign_id" }
]
}
}
```
**For videos:**
```bash theme={null}
POST /v1/collections
{
"collection_name": "video-segments",
"source": { "type": "bucket", "bucket_ids": ["bkt_visual_assets"] },
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": { "video": "asset_url" },
"parameters": {
"scene_detection_threshold": 0.3,
"extract_keyframes": true
}
}
}
```
## 3. Ingest Assets
```bash theme={null}
POST /v1/buckets/{bucket_id}/objects
{
"key_prefix": "/products/shoes",
"blobs": [
{ "property": "asset_url", "type": "image", "data": "s3://my-bucket/products/sneaker-001.jpg" }
],
"metadata": {
"brand": "Nike",
"campaign_id": "fall-2025"
}
}
```
## 4. Process
```bash theme={null}
POST /v1/buckets/{bucket_id}/batches
{ "object_ids": ["obj_001", "obj_002"] }
POST /v1/buckets/{bucket_id}/batches/{batch_id}/submit
```
## 5. Create a Retriever
```bash theme={null}
POST /v1/retrievers
{
"retriever_name": "reverse-image-search",
"collection_identifiers": ["col_product_images"],
"input_schema": {
"query_image": { "type": "image", "required": true },
"min_similarity": { "type": "number", "default": 0.7 }
},
"stages": [
{
"stage_name": "visual_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"query": {
"input_mode": "content",
"value": "{{INPUT.query_image}}"
},
"top_k": 50
}
]
}
}
},
{
"stage_name": "filter",
"stage_type": "reduce",
"config": {
"stage_id": "score_threshold",
"parameters": {
"min_score": "{{INPUT.min_similarity}}"
}
}
}
]
}
```
## 6. Search
**With image URL:**
```bash theme={null}
POST /v1/retrievers/{retriever_id}/execute
{
"inputs": {
"query_image": "https://example.com/reference.jpg",
"min_similarity": 0.75
}
}
```
**With pre-computed embedding (faster):**
```bash theme={null}
POST /v1/retrievers/{retriever_id}/execute
{
"inputs": {
"embedding": [0.1, 0.2, ...]
}
}
```
**With base64:**
```python theme={null}
import base64
with open("image.jpg", "rb") as f:
image_data = base64.b64encode(f.read()).decode()
response = requests.post(
"https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute",
json={
"inputs": {
"query_image": f"data:image/jpeg;base64,{image_data}"
}
}
)
```
## Cross-Modal Search (Image → Video)
Search videos using a reference image:
```bash theme={null}
{
"stage_name": "cross_modal_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": {
"input_mode": "content",
"value": "{{INPUT.query_image}}"
},
"top_k": 50
}
]
}
}
}
```
## Multi-Collection Search
Search images and videos together:
```bash theme={null}
{
"retriever_name": "visual-federated-search",
"collection_identifiers": ["col_images", "col_videos"],
"stages": [
{
"stage_name": "federated_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
"collections": ["col_images"],
"weight": 0.5,
"top_k": 25
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"collections": ["col_videos"],
"weight": 0.5,
"top_k": 25
}
],
"fusion": "rrf"
}
}
}
]
}
```
## Similarity Thresholds
| Score | Meaning |
| ---------- | -------------- |
| 0.95+ | Near-duplicate |
| 0.85-0.94 | Very similar |
| 0.70-0.84 | Related |
| Below 0.70 | Weak match |
## Classify with Taxonomies
Auto-tag assets by matching against a reference collection of known brands or product types:
```bash theme={null}
POST /v1/taxonomies
{
"taxonomy_name": "brand-classifier",
"taxonomy_type": "flat",
"retriever_id": "ret_reverse_image_search",
"input_mappings": {
"query_image": "mixpeek://image_extractor@v1/google_siglip_base_v1"
},
"source_collection": {
"collection_id": "col_product_images",
"enrichment_fields": [
{ "field_path": "metadata.brand", "merge_mode": "enrich" }
]
}
}
```
New assets automatically get `metadata.brand` enriched when they visually match a known reference. See [Taxonomies](/docs/enrichment/taxonomies) for hierarchical taxonomies.
## Discover Clusters
Find visual themes across your asset library:
```bash theme={null}
POST /v1/clusters
{
"cluster_name": "visual-themes",
"collection_ids": ["col_product_images"],
"cluster_type": "vector",
"vector_config": {
"feature_uris": ["mixpeek://image_extractor@v1/google_siglip_base_v1"],
"clustering_method": "hdbscan",
"algorithm_params": { "min_cluster_size": 10 }
},
"llm_labeling": {
"provider": "openai",
"model_name": "gpt-4o-mini"
},
"dimension_reduction": {
"method": "umap",
"components": 2
}
}
```
Clusters reveal groupings like "product close-ups", "lifestyle shots", and "packaging" without predefined categories. Promote stable clusters to taxonomy nodes. See [Clusters](/docs/enrichment/clusters) for all algorithms.
## Set Up Alerts
Get notified when new assets closely match existing ones (counterfeit detection, duplicate detection):
```bash theme={null}
POST /v1/alerts
{
"alert_name": "duplicate-detection",
"collection_id": "col_product_images",
"condition": { "field": "taxonomy.detected_brand", "operator": "exists" },
"notification": { "type": "webhook", "url": "https://example.com/webhook" }
}
```
## Set Up Webhooks
Track batch processing for large asset uploads:
```bash theme={null}
POST /v1/webhooks
{
"webhook_name": "asset-processing",
"url": "https://example.com/webhook",
"events": ["batch.completed", "batch.failed"]
}
```
# Semantic Search
Source: https://docs.mixpeek.com/docs/tutorials/semantic-search
Build search with vector embeddings and hybrid ranking
<Card title="Create your free workspace" icon="rocket" href="https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=docs&utm_medium=tutorial&utm_campaign=managed_onramp&utm_content=semantic_search_tutorial">
Follow this tutorial on your own data — create a free workspace, upload your documents, and run your first semantic search in minutes.
</Card>
<Tip>Semantic search is one stage in the warehouse's Reassemble layer. This tutorial covers the basics; see [Multi-Stage Retrieval](/docs/retrieval/multi-stage-deep-dive) for composing it with other stages.</Tip>
## 1. Create a Bucket
```bash theme={null}
POST /v1/buckets
{
"bucket_name": "knowledge-base",
"bucket_schema": {
"properties": {
"title": { "type": "text", "required": true },
"content": { "type": "text", "required": true },
"category": { "type": "text" },
"tags": { "type": "array" }
}
}
}
```
## 2. Create a Collection
```bash theme={null}
POST /v1/collections
{
"collection_name": "docs-search",
"source": { "type": "bucket", "bucket_ids": ["bkt_kb"] },
"feature_extractor": {
"feature_extractor_name": "text_extractor",
"version": "v1",
"input_mappings": { "text": "content" },
"parameters": {
"model": "multilingual-e5-large-instruct",
"chunk_strategy": "sentence",
"chunk_size": 512,
"chunk_overlap": 50
},
"field_passthrough": [
{ "source_path": "title" },
{ "source_path": "category" },
{ "source_path": "tags" }
]
}
}
```
**Chunking strategies:**
* `sentence` – Best for Q\&A
* `paragraph` – Best for long-form content
* `fixed` – Predictable token windows
## 3. Ingest Documents
```bash theme={null}
POST /v1/buckets/{bucket_id}/objects
{
"key_prefix": "/docs/api",
"blobs": [
{ "property": "title", "type": "text", "data": "Authentication Guide" },
{ "property": "content", "type": "text", "data": "Mixpeek uses Bearer token authentication..." },
{ "property": "category", "type": "text", "data": "getting-started" }
]
}
```
## 4. Create a Retriever
```bash theme={null}
POST /v1/retrievers
{
"retriever_name": "docs-search",
"collection_identifiers": ["col_docs"],
"input_schema": {
"query": { "type": "text", "required": true }
},
"stages": [
{
"stage_name": "knn_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
}
],
"final_top_k": 50
}
}
}
]
}
```
## 5. Search
```bash theme={null}
POST /v1/retrievers/{retriever_id}/execute
{
"inputs": { "query": "how do I authenticate API requests?" }
}
```
## Hybrid Search (Vector + BM25)
Combine semantic (vector) and keyword (BM25) matching in one stage. BM25 is **not** a separate feature — set `lexical: true` on a search to match the query against the namespace's full-text index instead of embedding it. Use `rrf` fusion so the score-scale mismatch between cosine similarity and BM25 doesn't matter.
```bash theme={null}
{
"stages": [
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
},
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"lexical": true,
"top_k": 100
}
],
"fusion": "rrf",
"final_top_k": 50
}
}
}
]
}
```
<Note>
Lexical search requires a `text` payload index on the field. See [Text Indexes (BM25)](/docs/vector-store/namespaces#text-indexes-bm25) and the [Feature Search](/docs/retrieval/stages/feature-search#lexical-bm25-search) reference.
</Note>
## Pre-Filter by Metadata
Filter before vector search for efficiency:
```bash theme={null}
{
"stages": [
{
"stage_name": "category_filter",
"stage_type": "filter",
"config": {
"stage_id": "attribute_filter",
"parameters": {
"field": "category",
"operator": "eq",
"value": "getting-started"
}
}
},
{
"stage_name": "knn_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 50
}
],
"final_top_k": 50
}
}
}
]
}
```
## Reranking
Use a cross-encoder for better accuracy:
```bash theme={null}
{
"stages": [
{
"stage_name": "knn_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100
}
],
"final_top_k": 100
}
}
},
{
"stage_name": "rerank",
"stage_type": "sort",
"config": {
"stage_id": "rerank",
"parameters": {
"inference_name": "BAAI__bge_reranker_v2_m3",
"top_k": 20
}
}
}
]
}
```
See the [Rerank stage](/docs/retrieval/stages/rerank) reference for available models and parameters.
## Model Options
| Model | Speed | Use Case |
| -------------------------------- | ------ | --------------- |
| `multilingual-e5-base` | Fast | High-volume |
| `multilingual-e5-large-instruct` | Medium | General-purpose |
| `bge-large-en-v1.5` | Medium | English-only |
| `openai/text-embedding-3-large` | Slow | Premium |
## Next steps
<CardGroup>
<Card title="Classify documents" icon="tags" href="/enrichment/taxonomies">
Auto-categorize documents against a taxonomy of reference categories.
</Card>
<Card title="Discover topics" icon="diagram-project" href="/enrichment/clusters">
Cluster document embeddings to surface topic groups without predefined categories.
</Card>
<Card title="Get notified" icon="bell" href="/enrichment/alerts">
Trigger alerts when new documents match a condition.
</Card>
<Card title="Schedule jobs" icon="clock" href="/platform/triggers">
Re-cluster or re-enrich on a cron or interval as new content lands.
</Card>
</CardGroup>
# Video Understanding
Source: https://docs.mixpeek.com/docs/tutorials/video-understanding
Ingest video, extract visual + speech embeddings, and search for moments by what's shown or said
<Tip>This is the full warehouse flow end-to-end: decompose video into searchable segments, store visual and speech embeddings, then reassemble answers through a retriever. Every code block below is copy-pasteable and uses the real API field names.</Tip>
<Frame>
<img alt="Video Understanding Pipeline" />
</Frame>
## How It Works
When you ingest a video, the `multimodal_extractor` runs a multi-stage pipeline:
1. **Chunking** — the video is split into segments (scene detection / fixed intervals).
2. **Visual embeddings** — each segment's keyframes are embedded as `vertex_multimodal_embedding`.
3. **Speech embeddings** — speech is transcribed (Whisper) and embedded (feature URI `mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1`).
4. **Multi-vector indexing** — both embeddings are indexed per segment, so one collection supports hybrid visual + transcript search.
At query time, a retriever searches across both embeddings and fuses the results — finding moments by what's *shown* or what's *said*.
<Note>
A single `multimodal_extractor@v1` produces **both** the visual feature (URI suffix `vertex_multimodal_embedding`) and the speech/transcript feature (URI suffix `multilingual_e5_large_instruct_v1`). You do **not** need separate video/audio extractors.
</Note>
## 1. Create a bucket
The bucket schema declares the fields each video object carries. Use the Mixpeek type `video` for the file field.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"bucket_name": "video-catalog",
"bucket_schema": {
"properties": {
"video_url": { "type": "video" },
"title": { "type": "text" },
"category": { "type": "string" }
}
}
}'
```
## 2. Create a collection
One collection with the `multimodal_extractor` gives you both visual and speech search. Map the extractor's `video` input to your bucket's `video_url` field.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/collections" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"collection_name": "video-moments",
"source": { "type": "bucket", "bucket_ids": ["bkt_videos"] },
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": { "video": "video_url" },
"field_passthrough": [
{ "source_path": "title" },
{ "source_path": "category" }
]
}
}'
```
<Note>
A collection takes exactly **one** `feature_extractor`. To run additional extractors over the same objects, create more collections from the same bucket.
</Note>
## 3. Ingest a video
Register a video object. The blob's content goes in the `data` field (a URL, `s3://` path, base64, or raw content).
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_videos/objects" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"key_prefix": "/marketing/demos",
"blobs": [
{
"property": "video_url",
"type": "video",
"data": "s3://my-bucket/demos/product-launch.mp4"
},
{ "property": "title", "type": "text", "data": "Product Launch Q4 2025" },
{ "property": "category", "type": "text", "data": "marketing" }
]
}'
```
## 4. Process
Create a batch from your object(s), submit it, then poll the returned task until it completes.
```bash theme={null}
# Create a batch (returns batch_id)
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_videos/batches" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "object_ids": ["obj_video_001"] }'
# Submit it for processing (returns task_id)
curl -sS -X POST "$MP_API_URL/v1/buckets/bkt_videos/batches/{batch_id}/submit" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
# Poll until status is terminal: COMPLETED or COMPLETED_WITH_ERRORS
curl -sS "$MP_API_URL/v1/tasks/{task_id}" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE"
```
<Tip>
Prefer the SDK? `client.tasks.get(task_id=...)` returns the same status. See [Monitoring ingestion](/docs/processing/tasks) for a complete polling loop.
</Tip>
## 5. Create a hybrid retriever
The retriever fuses two feature searches — visual and speech — over the same collection. Note the canonical field names: `collection_identifiers`, a flat `input_schema`, and the stage `config` wrapping `stage_id` + `parameters`.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "video-search",
"collection_identifiers": ["video-moments"],
"input_schema": {
"query": { "type": "text", "required": true }
},
"stages": [
{
"stage_name": "hybrid_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100,
"weight": 0.6
},
{
"feature_uri": "mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" },
"top_k": 100,
"weight": 0.4
}
],
"final_top_k": 20,
"fusion": "weighted"
}
}
}
]
}'
```
| Field | Meaning |
| ------------------------ | ---------------------------------------------------------------------------------- |
| `collection_identifiers` | Collections to query (names or IDs). |
| `input_schema` | Flat map of input field → type. Reference inputs in stages as `{{INPUT.<field>}}`. |
| `searches[].feature_uri` | Which embedding index to search. Here: visual + speech. |
| `searches[].weight` | Per-search weight used by `weighted` fusion. |
| `final_top_k` | Results returned after fusion. |
| `fusion` | `rrf` (default) or `weighted`. |
## 6. Search
Execute the retriever. The `inputs` keys must match your `input_schema`.
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"inputs": { "query": "people discussing electric vehicles" }
}'
```
Each result is a video segment with its timestamps and keyframe, ranked by combined visual + transcript relevance.
### Filter by category
Apply ad-hoc filters at execution time without changing the retriever:
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"inputs": { "query": "product demonstration" },
"filters": {
"field": "category",
"operator": "eq",
"value": "marketing"
}
}'
```
### Search by a reference image
Because `vertex_multimodal_embedding` is cross-modal, you can query the visual index with an image to find similar scenes. Add an image input to `input_schema` and set the visual search's `query` to `{ "input_mode": "content", "value": "{{INPUT.query_image}}" }`. See [Feature Search](/docs/retrieval/stages/feature-search) for image/URL query modes.
## Moment-level search
Segments carry start/end timestamps, so you can scope results to part of a video with a filter (field names depend on your collection's output schema):
```bash theme={null}
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{
"inputs": { "query": "pricing discussion" },
"filters": { "field": "start_time", "operator": "gte", "value": 60.0 }
}'
```
## Search the same video other ways
The same segments are searchable across every modality the `multimodal_extractor` produced — combine or swap these into your retriever:
<CardGroup>
<Card title="Search faces" icon="user" href="/processing/extractors/face-identity">
Add the `face_identity_extractor` to find every clip featuring a specific person (ArcFace 1:N).
</Card>
<Card title="Search transcripts" icon="closed-captioning" href="/retrieval/stages/feature-search">
Query `mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1` to find moments by what's *said*.
</Card>
<Card title="Group into moments" icon="film" href="/retrieval/stages/moment-group">
Merge matching frames into continuous time-ranges with the `moment_group` stage.
</Card>
<Card title="Rerank for precision" icon="arrow-up-wide-short" href="/retrieval/stages/rerank">
Add a cross-encoder `rerank` stage to sharpen the top results.
</Card>
<Card title="Filter unsafe content" icon="shield" href="/retrieval/stages/classify">
Drop or flag NSFW segments at query time with the `classify` stage.
</Card>
<Card title="Hybrid keyword search" icon="key" href="/retrieval/stages/feature-search#lexical-bm25-search">
Add a `lexical: true` search to catch exact terms in transcripts (BM25).
</Card>
</CardGroup>
### Search by action or activity
There is no dedicated action-recognition extractor — instead, search **actions as natural language** against the visual embedding. A text query like `"person running"` or `"two people shaking hands"` matches the `vertex_multimodal_embedding` because it understands actions depicted in frames:
```json theme={null}
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "person running on a treadmill" },
"top_k": 50
}
```
For higher precision on a fixed set of actions, enable `run_video_description` / `response_shape` on the extractor to emit structured action labels you can then [`attribute_filter`](/docs/retrieval/stages/attribute-filter) or [`classify`](/docs/retrieval/stages/classify).
## Next steps
<CardGroup>
<Card title="Auto-tag segments" icon="tags" href="/enrichment/taxonomies">
Classify segments ("product demo", "interview") with a taxonomy.
</Card>
<Card title="Discover themes" icon="diagram-project" href="/enrichment/clusters">
Cluster segment embeddings to surface recurring visual themes.
</Card>
<Card title="Get notified" icon="bell" href="/enrichment/alerts">
Trigger alerts when new content matches a query.
</Card>
<Card title="Tune feature search" icon="sliders" href="/retrieval/stages/feature-search">
Full reference for searches, fusion, and query modes.
</Card>
</CardGroup>
# Bring Your Own Object Storage
Source: https://docs.mixpeek.com/docs/vector-store/byo-object-storage
Back the Mixpeek Vector Store with your own S3-compatible or GCS bucket on dedicated and self-hosted deployments
The Mixpeek Vector Store (MVS) persists everything it owns — the write-ahead log, snapshots, and the cold tier — to object storage through a provider-agnostic layer. On **dedicated and self-hosted deployments** you can point that layer at your own bucket, on any supported provider, so your vector data lives in storage you control.
<Note>
This is about where MVS **stores its own data**, not where you ingest from. To connect a bucket as an ingestion *source*, see [Object Storage Integrations](/docs/integrations/object-storage/overview). The managed shared plane uses Mixpeek-operated storage; BYO object storage applies to [single-tenant](/docs/resources/single-tenant) and self-hosted deployments and is configured at the deployment level (not a per-namespace setting).
</Note>
## Supported providers
MVS speaks the S3 API (via boto3) and the native GCS API. Any S3-compatible store works through the same code path — the only per-provider differences are the endpoint, region, and path-style addressing.
| Provider | `MVS_OBJECT_STORE_PROVIDER` | Docs |
| -------------------- | --------------------------- | --------------------------------------------------- |
| Amazon S3 | `s3` | [S3](/docs/integrations/object-storage/s3) |
| Google Cloud Storage | `gcs` | [GCS](/docs/integrations/object-storage/gcs) |
| Cloudflare R2 | `r2` | [R2](/docs/integrations/object-storage/r2) |
| Tigris | `tigris` | [Tigris](/docs/integrations/object-storage/tigris) |
| Wasabi | `wasabi` | [Wasabi](/docs/integrations/object-storage/wasabi) |
| Backblaze B2 | `b2` | [Backblaze](/docs/integrations/object-storage/backblaze) |
| MinIO | `minio` | S3-compatible |
| DigitalOcean Spaces | `spaces` | S3-compatible |
| Ceph | `ceph` | S3-compatible |
| LocalStack (dev) | `localstack` | S3-compatible |
See all providers in the [Object Storage overview](/docs/integrations/object-storage/overview).
## Configuration
The object-store backend is resolved from environment variables on the MVS coordinator and shard pods:
| Variable | Default | Description |
| --------------------------- | ------- | -------------------------------------------------------------------------------------------------- |
| `MVS_OBJECT_STORE_PROVIDER` | `gcs` | Provider key from the table above. |
| `MVS_BUCKET` | — | Bucket that holds all MVS data. (Legacy alias: `MVS_GCS_BUCKET` — both must name the same bucket.) |
| `MVS_OBJECT_STORE_ENDPOINT` | — | Custom S3 endpoint. **Required** for R2, Tigris, and Wasabi; optional for MinIO/Ceph. |
| `MVS_REGION` | — | Region for S3-compatible providers. |
| `MVS_PREFIX` | `mvs/` | Global key prefix for all objects. |
| `MVS_ENCRYPTION_KEY` | — | Optional base64-encoded 32-byte key to encrypt all data at rest (see below). |
<Warning>
The coordinator and the Rust shard must read the **same** bucket. Set `MVS_BUCKET` (or `MVS_GCS_BUCKET`) consistently across both — a split here causes cross-tenant namespace discovery. MVS fails fast if the two env vars disagree.
</Warning>
### Credentials
Credentials are never stored in config — they're resolved from the deployment environment:
* **S3-compatible:** the standard AWS credential chain. Prefer an **IAM role** attached to the pods; otherwise set `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`. R2/Tigris/Wasabi/B2 issue S3 access keys you use the same way.
* **GCS:** Application Default Credentials (workload identity / `GOOGLE_APPLICATION_CREDENTIALS`), or a service-account JSON supplied as a secret.
### Endpoint and addressing notes
* **Endpoint required:** R2 (`https://<account_id>.r2.cloudflarestorage.com`), Tigris, and Wasabi have no usable default — set `MVS_OBJECT_STORE_ENDPOINT` or MVS fails fast with a clear message rather than silently hitting AWS.
* **Path-style addressing** is applied automatically for B2, MinIO, Ceph, and LocalStack (they only serve path-style); AWS/R2/Tigris/Wasabi use virtual-hosted style.
## Encryption at rest
Set `MVS_ENCRYPTION_KEY` (base64-encoded 32 bytes) to transparently encrypt every object with **AES-256-GCM**. The object's storage key is used as additional authenticated data (AAD), binding each ciphertext to its path. Migration needs no backfill: existing unencrypted objects stay readable while new writes are encrypted, so you can enable it on a populated bucket without a backfill.
## Examples
<CodeGroup>
```bash Amazon S3 theme={null}
MVS_OBJECT_STORE_PROVIDER=s3
MVS_BUCKET=my-mvs-data
MVS_REGION=us-east-1
# credentials via attached IAM role (recommended) or AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
```
```bash Cloudflare R2 theme={null}
MVS_OBJECT_STORE_PROVIDER=r2
MVS_BUCKET=my-mvs-data
MVS_OBJECT_STORE_ENDPOINT=https://<account_id>.r2.cloudflarestorage.com
AWS_ACCESS_KEY_ID=<r2_access_key>
AWS_SECRET_ACCESS_KEY=<r2_secret_key>
```
```bash Google Cloud Storage theme={null}
MVS_OBJECT_STORE_PROVIDER=gcs
MVS_BUCKET=my-mvs-data
# credentials via workload identity / GOOGLE_APPLICATION_CREDENTIALS
```
```bash MinIO (self-hosted) theme={null}
MVS_OBJECT_STORE_PROVIDER=minio
MVS_BUCKET=my-mvs-data
MVS_OBJECT_STORE_ENDPOINT=https://minio.internal:9000
AWS_ACCESS_KEY_ID=<minio_access_key>
AWS_SECRET_ACCESS_KEY=<minio_secret_key>
```
</CodeGroup>
## Related
* [Object Storage Integrations](/docs/integrations/object-storage/overview) — connect a bucket as an ingestion source
* [Single-Tenant Deployments](/docs/resources/single-tenant) — fully isolated data plane with a dedicated bucket
* [Vector Store Overview](/docs/vector-store/overview) — using MVS as a standalone vector store
# Collection Lifecycle
Source: https://docs.mixpeek.com/docs/vector-store/collection-lifecycle
Move collections between hot, cold, and archived tiers — pay for fast vector search only where you need it, keep everything else restorable
Not every collection needs to serve low-latency vector search all the time. Seasonal campaigns, completed projects, and long-tail archives can move to cheaper tiers — without losing the data, and without re-processing anything to bring them back.
## Lifecycle states
| State | Vector search | Storage | Restorable |
| -------------------- | --------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------- |
| `active` *(default)* | Fast, fully indexed | Hot vector index | — |
| `cold` | Evicted from the hot index; vectors preserved in object storage | Object-storage backup | Yes — rehydrate to `active` at any time |
| `archived` | Removed | Vectors permanently deleted; documents' metadata remains | No |
Allowed transitions: `active → cold`, `active → archived`, `cold → active`, `cold → archived`. Archival is permanent — the vectors are deleted, not parked.
<Note>
Going `cold` does not delete documents or metadata — it evicts the collection's vectors from the hot index after backing them up. Rehydrating restores search without re-running any extraction or embedding.
</Note>
## Check lifecycle status
```bash theme={null}
curl "https://api.mixpeek.com/v1/collections/{collection_id}/lifecycle" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: my-namespace"
```
Returns the current state plus counts from both the hot index and the object-storage backup, so you can verify a transition completed.
## Transition a collection
```bash theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/collections/{collection_id}/lifecycle" \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: my-namespace" \
-H "Content-Type: application/json" \
-d '{"lifecycle_state": "cold"}'
```
* **`cold`** evicts the collection from the vector store (vectors preserved via object storage).
* **`active`** (from `cold`) rehydrates — the backup is restored into the hot index.
* **`archived`** permanently removes the vectors.
<Warning>
`archived` is irreversible for vectors. If there is any chance you'll search this collection again, use `cold` — restoring from cold is cheap; rebuilding from archive means re-processing the source objects.
</Warning>
## When to use each tier
* **`active`** — anything queried by production retrievers.
* **`cold`** — completed campaigns and historical projects you want *restorable on demand* (for example, an annual review): rehydrate, query, and send back to cold.
* **`archived`** — data you're retaining for record-keeping where the documents' metadata is enough and vector search will never be needed again.
## Related
* [Buckets & storage classes](/docs/platform/data-model) — tiering for the *source objects* (this page covers the *index* tier; the two are independent)
* [Manage data](/docs/vector-store/manage-data) — document-level operations
* [Deduplication & re-processing](/docs/processing/deduplication) — what happens if you do need to rebuild from source
# Documents & Search
Source: https://docs.mixpeek.com/docs/vector-store/documents
Upsert vectors, search with dense/sparse/BM25/hybrid, and manage documents
## Upsert Documents
Insert or update documents with your pre-computed vectors and JSON payload. Up to 1,000 documents per request.
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="mxp_sk_...")
result = client.namespaces.documents.upsert(
namespace_id="product-search",
documents=[
{
"document_id": "prod-001",
"vectors": {"text_embedding": [0.12, -0.34, 0.56]}, # 1536 floats
"payload": {"title": "Noise-Canceling Headphones", "price": 149.99, "in_stock": True},
},
{
"document_id": "prod-002",
"vectors": {"text_embedding": [0.08, 0.22, -0.41]}, # 1536 floats
"payload": {"title": "Bluetooth Speaker", "price": 59.99, "in_stock": True},
},
],
)
# → {"inserted": 2, "document_ids": ["prod-001", "prod-002"]}
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/product-search/documents/upsert" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"documents": [{
"document_id": "prod-001",
"vectors": {"text_embedding": [0.12, -0.34, 0.56]},
"payload": {"title": "Noise-Canceling Headphones", "price": 149.99}
}]
}'
```
</CodeGroup>
Vector dimensions are validated against namespace config. If a document with the same ID exists, it is overwritten.
### Large Imports
Repeat the upsert call in batches of up to 1,000 documents. There is no
separate bulk-import endpoint.
## Search
<Note>
**Querying is unified through retrievers** — Mixpeek has one query path, so there is
no direct `POST /v1/namespaces/{ns}/documents/search` REST endpoint. The `client.search(...)`
helper below is SDK sugar that authors and runs a retriever for you. From raw HTTP/curl,
create a one-stage `feature_search` retriever and execute it — the verified BYO example
immediately below does exactly that in three calls.
</Note>
### Query BYO vectors from raw HTTP (verified)
Bring your own vectors and query them with a raw query vector — no extractor, no re-processing.
The query is an **object** (`{"input_mode": "vector", "value": "{{INPUT.qv}}"}`), and the
retriever's `input_schema` declares the input variable your execute call fills in.
```bash cURL theme={null}
# 1. Create a feature_search retriever over your BYO vector index ("my_vec").
# Note: stage_type is "filter"; the operation is stage_id "feature_search".
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $NS" -H "Content-Type: application/json" \
-d '{
"retriever_name": "byo-knn",
"input_schema": {"qv": {"type": "array", "required": true}},
"stages": [{
"stage_name": "knn", "stage_type": "filter",
"config": {"stage_id": "feature_search", "parameters": {
"searches": [{
"feature_uri": "my_vec",
"query": {"input_mode": "vector", "value": "{{INPUT.qv}}"}
}],
"final_top_k": 10
}}
}]
}'
# → {"retriever_id": "ret_...", ...}
# 2. Execute it with your query vector. pagination.method is required.
curl -X POST "https://api.mixpeek.com/v1/retrievers/$RETRIEVER_ID/execute" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" -H "X-Namespace: $NS" -H "Content-Type: application/json" \
-d '{"inputs": {"qv": [1,0,0,0,0,0,0,0]}, "pagination": {"method": "offset", "page_size": 10, "page_number": 1}}'
# → {"documents": [{"document_id": "d1", "score": 1.0, ...}, ...]}
```
The execute cache is invalidated by upserts — a query re-run after an upsert returns fresh
results (`cache_hit: false`), so the write-then-verify loop is safe.
### Dense (Vector) — SDK
```python theme={null}
results = client.search(
namespace_id="product-search",
queries=[{
"vector_name": "text_embedding",
"vector": query_embedding,
"top_k": 10,
"score_threshold": 0.7,
}],
)
```
### BM25 (Keyword)
Requires a [text index](/docs/vector-store/namespaces#text-indexes-bm25) on the target field.
```python theme={null}
results = client.search(
namespace_id="product-search",
queries=[{"text": "wireless noise canceling", "top_k": 10}],
)
```
### Sparse
```python theme={null}
results = client.search(
namespace_id="product-search",
queries=[{"sparse_vector": {42: 0.8, 1337: 0.5, 9001: 0.3}, "top_k": 10}],
)
```
### Hybrid
Combine multiple query types with RRF or DBSF fusion.
```python theme={null}
results = client.search(
namespace_id="product-search",
queries=[
{"vector_name": "text_embedding", "vector": query_embedding, "top_k": 20},
{"text": "wireless headphones", "top_k": 20},
],
fusion="rrf",
)
```
### Filtered
Add payload filters to any query type. Filters narrow results before scoring.
```python theme={null}
results = client.search(
namespace_id="product-search",
queries=[{
"vector_name": "text_embedding",
"vector": query_embedding,
"top_k": 10,
"filters": {
"must": [
{"key": "price", "range": {"lte": 100}},
{"key": "in_stock", "match": {"value": True}}
]
},
}],
)
```
## Document Operations
| Operation | Method | Endpoint |
| ----------------- | -------- | ---------------------------------------------------- |
| Get by ID | `GET` | `/v1/namespaces/{ns}/documents/{doc_id}` |
| Read many by ID | `POST` | `/v1/documents/batch-get` |
| List and paginate | `POST` | `/v1/documents/list` |
| Update payload | `PATCH` | `/v1/documents/{doc_id}` |
| Delete | `DELETE` | `/v1/collections/{collection_id}/documents/{doc_id}` |
Only upsert and get-by-ID carry the namespace in the path. The rest read it from
the `X-Namespace` header. Delete is collection-scoped, so a document you reach
by namespace is deleted through its collection.
### Read Many by ID
Resolve up to 1,000 documents in one call. The namespace comes from the
`X-Namespace` header here, not from the path.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/documents/batch-get" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{"document_ids": ["prod-001", "prod-002", "prod-404"]}'
# → {"documents": [...], "not_found": ["prod-404"]}
```
| Field | Meaning |
| ----------------------- | ----------------------------------------------------------------- |
| `document_ids` | 1 to 1,000 ids. Required. |
| `return_presigned_urls` | Accepted and ignored on this endpoint. Batch-get always presigns. |
| `documents` | The documents that resolved. |
| `not_found` | Each id that did not resolve, listed individually. |
<Note>
Match returned documents by `document_id` rather than by position. The
response order does not track the order you sent.
An id that does not resolve does not fail the call. It lands in `not_found`
and every other id still returns, so read `not_found` rather than comparing
lengths.
</Note>
### Media URLs
Stored media fields hold a raw storage URI: `gs://` on Mixpeek Cloud, `s3://` on
S3-backed deployments. Whether you get a fetchable `https://` URL back depends on
the endpoint you called, not on the field name.
| Surface | Presigns |
| ------------------------------ | --------------------------------------------------------- |
| Retriever execute | Always. `return_presigned_urls` is not read on this path. |
| `POST /v1/documents/batch-get` | Always. The flag is accepted and ignored. |
| `POST /v1/documents/list` | Only with `"return_presigned_urls": true` in the body. |
| `GET /v1/documents/{doc_id}` | Only with the flag, and these URLs last 1 hour. |
<Warning>
`POST /v1/documents/list` reads that flag from the request body only. A
`?return_presigned_urls=true` query string is ignored, and you get raw `gs://`
values back, which most HTTP clients reject with `unknown url type: gs`.
</Warning>
Selection works on the value, not the name. Any field whose value starts with
`gs://` or `s3://` gets signed, whatever the field is called. These stay raw:
* anything under `metadata`
* values nested inside dicts or lists, on documents carrying a `collection_id`
* `original_url`, `source_blobs`, and `presigned_urls`
* any URI whose signing call fails
Signed URLs last 24 hours everywhere except the single-document `GET` above.
### Update Vectors
Re-upsert the document through `POST /v1/namespaces/{ns}/documents/upsert`.
Sending an existing `document_id` overwrites that document, vectors and payload
together, so send the payload you want to keep alongside the new vectors.
No endpoint replaces vectors while leaving the payload untouched.
## Related
* [Namespace Configuration](/docs/vector-store/namespaces)
* [Promote to Managed](/docs/vector-store/promote)
# Update & Delete Data
Source: https://docs.mixpeek.com/docs/vector-store/manage-data
Update document metadata and delete documents, collections, and objects — including what cascades
How to change and remove data after ingestion, and exactly what each delete cascades to.
## Update document metadata
`PATCH` a document to change its fields. The body is a partial update — send **only the fields you want to change**; you never resend vectors. Any fields you pass are merged into the document.
```bash theme={null}
curl -sS -X PATCH \
"$MP_API_URL/v1/collections/$COLLECTION_ID/documents/$DOCUMENT_ID" \
-H "Authorization: Bearer $MP_API_KEY" \
-H "X-Namespace: $MP_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{ "status": "reviewed", "priority": 3 }'
```
To update many documents at once, use `PATCH /v1/collections/{collection_id}/documents/batch` (bulk update).
<Note>
Updating metadata does **not** re-run extraction or change vectors — it only edits the stored payload. To change the embedding model, see [Migrate Embedding Models](/docs/processing/model-migration).
</Note>
To change **access control** (not metadata), use the dedicated ACL endpoint `PATCH /v1/collections/{collection_id}/documents/{document_id}/acl` — see [Permissions](/docs/platform/permissions).
<Note>
**Object metadata vs document metadata are separate.** The `PATCH` above edits a **document** (a processed, searchable record in a collection). To edit the source **object** in the bucket, use [`PUT /v1/buckets/{bucket_id}/objects/{object_id}`](/docs/api-reference/bucket-objects/update-object) with a `metadata` object (merged with existing). Editing one does not change the other, and re-processing an object regenerates its documents from the object's current state.
</Note>
## Delete data
<Warning>
Deletes are permanent and some **cascade**. Read the cascade column before deleting — removing a bucket object or a collection also destroys derived documents.
</Warning>
| Delete | Endpoint | Cascades to |
| --------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| A document | `DELETE /v1/collections/{collection_id}/documents/{document_id}` | Just that document |
| Many documents | `DELETE /v1/collections/{collection_id}/documents/batch` | The documents you list |
| A bucket object | `DELETE /v1/buckets/{bucket_id}/objects/{object_id}` | **Hard-deletes every collection document derived from that object** |
| A collection | `DELETE /v1/collections/{collection_id}` | **All documents in the collection** |
| A namespace | `DELETE /v1/namespaces/{namespace_id}` | **Everything in the namespace** (buckets, collections, documents, retrievers) |
```bash theme={null}
# Delete a single document
curl -sS -X DELETE \
"$MP_API_URL/v1/collections/$COLLECTION_ID/documents/$DOCUMENT_ID" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
# Delete a bucket object (also removes its derived documents)
curl -sS -X DELETE \
"$MP_API_URL/v1/buckets/$BUCKET_ID/objects/$OBJECT_ID" \
-H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE"
```
<Tip>
Deleting the **object** is the right move when you want the source asset *and* its derived documents gone. Deleting just the **document** leaves the source object in the bucket, so a re-process would recreate the document.
</Tip>
### Synced sources
If documents came from a [storage sync](/docs/platform/syncs), the sync's `reconcile.on_delete` is `true` by default, so deleting a file in the source (S3, Google Drive, etc.) automatically removes the corresponding object and its derived documents. Set `reconcile.on_delete` to `false` to keep Mixpeek objects when the source asset is deleted.
## Related
* [Documents](/docs/vector-store/documents) — document structure and payload
* [Syncs](/docs/platform/syncs) — source-deletion cascade (`on_delete`)
* [Ingest Data](/docs/platform/data-model) — objects, batches, collections
# Namespaces
Source: https://docs.mixpeek.com/docs/vector-store/namespaces
Create standalone namespaces with vector indexes, distance metrics, BM25, and payload indexes
A namespace is the isolation boundary for your vectors, documents, and indexes.
## Create a Namespace
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="mxp_sk_...")
# Minimal — MVS infers vector configs on first write
client.namespaces.create(
namespace_id="product-search",
)
# Or pre-declare configs to set a specific distance metric
client.namespaces.create(
namespace_id="product-search",
vector_configs=[
{"name": "text_embedding", "dimension": 1536, "metric": "cosine"},
{"name": "image_embedding", "dimension": 512, "metric": "dot"},
],
)
```
```bash cURL theme={null}
# Minimal — MVS infers vector configs on first write
curl -X POST "https://api.mixpeek.com/v1/namespaces/standalone" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"namespace_id": "product-search"}'
# Or pre-declare configs to set a specific distance metric
curl -X POST "https://api.mixpeek.com/v1/namespaces/standalone" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespace_id": "product-search",
"vector_configs": [
{"name": "text_embedding", "dimension": 1536, "metric": "cosine"},
{"name": "image_embedding", "dimension": 512, "metric": "dot"}
]
}'
```
</CodeGroup>
### Parameters
<ParamField type="string">
Unique identifier. Lowercase with hyphens (e.g., `product-search`).
</ParamField>
<ParamField type="string">
Must be `"standalone"`.
</ParamField>
<ParamField type="array">
Optional. Pre-declare named vector indexes with specific dimensions and distance metrics. If omitted, MVS uses [schema-on-write](#schema-on-write) — vector indexes are created automatically on first upsert with inferred dimensions and cosine metric.
| Field | Type | Description |
| ----------- | ------- | ----------------------------------------- |
| `name` | string | Index name (e.g., `text_embedding`) |
| `dimension` | integer | Must match your model's output dimension |
| `metric` | string | `cosine` (default), `dot`, or `euclidean` |
</ParamField>
Documents can include vectors for any subset of indexes — not every index needs a vector in every document. When you upsert a document with a new vector name that hasn't been seen before, MVS creates a new vector index automatically.
## Text Indexes (BM25)
Full-text keyword search runs on **payload indexes of type `text`**. You declare indexes with `payload_indexes`, then register them in the store with `ensure-indexes` — which also backfills existing documents with **no re-ingestion or re-embedding**.
**Step 1 — Declare a `text` index** (on an existing namespace, patch `payload_indexes`):
<CodeGroup>
```python Python theme={null}
requests.patch(f"{BASE}/namespaces/product-search", headers=headers, json={
"payload_indexes": [
{"field_name": "title", "type": "text"},
{"field_name": "description", "type": "text"}
]
})
```
```bash cURL theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/namespaces/product-search" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"payload_indexes": [
{"field_name": "title", "type": "text"},
{"field_name": "description", "type": "text"}
]}'
```
</CodeGroup>
**Step 2 — Register and backfill** (idempotent, zero re-extraction):
<CodeGroup>
```python Python theme={null}
requests.post(f"{BASE}/namespaces/product-search/ensure-indexes", headers=headers)
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/product-search/ensure-indexes" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
</CodeGroup>
Pass `?force_rebuild=true` to drop and rebuild an index from the full store — use this to repair a `text`/BM25 index that registered empty (e.g. during a cold start).
<Note>
BM25 matches across **all** `text`-indexed string fields, not a single field. Once a text index exists, run keyword search in a retriever with the [`lexical` option on Feature Search](/docs/retrieval/stages/feature-search#lexical-bm25-search) — and fuse it with a dense vector search under `rrf` for hybrid retrieval.
</Note>
## Payload Indexes
Speed up filtered searches by indexing frequently queried fields. Payload indexes use the same `payload_indexes` declaration + `ensure-indexes` flow as text indexes above — just pick the matching `type`:
<CodeGroup>
```python Python theme={null}
requests.patch(f"{BASE}/namespaces/product-search", headers=headers, json={
"payload_indexes": [{"field_name": "category", "type": "keyword"}]
})
requests.post(f"{BASE}/namespaces/product-search/ensure-indexes", headers=headers)
```
```bash cURL theme={null}
curl -X PATCH "https://api.mixpeek.com/v1/namespaces/product-search" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"payload_indexes": [{"field_name": "category", "type": "keyword"}]}'
curl -X POST "https://api.mixpeek.com/v1/namespaces/product-search/ensure-indexes" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
</CodeGroup>
Types: `keyword`, `integer`, `float`, `bool`, `datetime`, `geo`, `text`, `uuid`. Use dot notation for nested fields (e.g. `metadata.title`). Arrays are indexed **element-wise** — index a multi-value field with its element type (`objects: ["car", "person"]` → `keyword`) and a filter matches if any element matches.
<Note>
**`PATCH` merges, `PUT` replaces.** A `PATCH` is additive: send only the fields
you're adding and every existing index is kept. Use `PUT` when you want the body
to be the complete list — any user index you omit there is **dropped**.
You can also `GET` the namespace, append to `payload_indexes`, and `PATCH` the
whole thing back. The protected system entries in that response are ignored on
input, so you don't need to strip them first.
</Note>
### Adaptive Payload Indexes
MVS monitors your query patterns and **automatically creates payload indexes** when a field appears in filters frequently enough. By default, any field used in >10 queries/hour gets auto-indexed — no manual action needed.
| Setting | Default | Description |
| ----------------------------- | --------------- | -------------------------------------------------------- |
| `auto_create_payload_indexes` | `true` | Auto-create payload indexes when filter threshold is hit |
| `filter_threshold` | 10 queries/hour | How many filter queries before auto-indexing |
| `auto_create_text_indexes` | `false` | BM25 indexes are heavier — suggest-only by default |
| `check_interval_s` | 600 (10 min) | How often the advisor checks query patterns |
Control this with the `MVS_AUTO_INDEX` environment variable (`true`/`false`).
### Index Suggestions
For fields that don't yet meet the auto-create threshold, you can check what MVS recommends:
```bash theme={null}
curl "https://api.mixpeek.com/v1/namespaces/product-search/index-suggestions" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
Returns recommended indexes based on filter frequency, field cardinality, and existing coverage.
### Schema-on-Write
Unlike traditional vector databases that require declaring all vector indexes at creation time, MVS is **schema-on-write**. When you upsert a document with a new vector name, MVS automatically:
1. Infers the dimension from the first write
2. Creates a dense index with the detected dimension and cosine metric
3. Enforces schema on subsequent writes (mismatched dimensions raise an error)
This means you can add new embedding models to an existing namespace without reconfiguring anything — just start writing vectors with the new name.
## Usage Metrics
Get detailed performance and usage metrics for a namespace via `GET /v1/namespaces/{id}/usage`.
```bash theme={null}
curl "https://api.mixpeek.com/v1/namespaces/product-search/usage" \
-H "Authorization: Bearer $MIXPEEK_API_KEY"
```
```json title="Response" theme={null}
{
"namespace_id": "product-search",
"document_count": 125000,
"vector_count": 250000,
"storage_bytes": 1073741824,
"index_count": 3,
"queries_last_24h": 48200,
"vectors": {"total": 250000, "by_name": {"text_embedding": 125000, "image_embedding": 125000}},
"documents": {"total": 125000, "with_all_vectors": 120000, "partial": 5000},
"storage": {
"total_bytes": 1073741824,
"by_tier": {"hot": 1073741824, "cold": 0, "archive": 0},
"vector_bytes": 768000000,
"payload_bytes": 32000000,
"index_bytes": 16384000
},
"queries": {"last_24h": 48200, "last_7d": 312000, "last_30d": 1250000,
"by_type": {"dense_search": 30000, "hybrid_search": 15000, "bm25_search": 3200}},
"writes": {"last_24h": 5200, "last_7d": 35000, "last_30d": 142000},
"shards": {"total": 4, "hot": 4, "cold": 0}
}
```
Key metrics:
| Metric | Description |
| ---------------------------- | ----------------------------------------------------------------------------------------- |
| `vectors.by_name` | Vector count per named index — useful for checking partial coverage |
| `queries.by_type` | Breakdown by search type — informs [cost optimization](/docs/best-practices/cost-optimization) |
| `shards.hot` / `shards.cold` | Shard distribution across hot (in-memory) and cold (object-storage) partitions |
These metrics feed into [usage-based billing](/docs/platform/billing). You can also monitor namespace health via the [observability dashboard](/docs/operations/observability).
## Other Operations
| Operation | Method | Endpoint |
| --------------- | -------- | ------------------------------------- |
| Get info | `GET` | `/v1/namespaces/{id}` |
| Delete | `DELETE` | `/v1/namespaces/{id}` |
| Clone | `POST` | `/v1/namespaces/{id}/clone` |
| Vector metadata | `PUT` | `/v1/namespaces/{id}/vector-metadata` |
## Related
* [Documents & Search](/docs/vector-store/documents)
* [Promote to Managed](/docs/vector-store/promote)
* [Billing & Usage](/docs/platform/billing)
# Vector Store
Source: https://docs.mixpeek.com/docs/vector-store/overview
Use Mixpeek as a standalone vector database — bring your own embeddings, search instantly, promote to managed when ready
Bring your own embeddings, upsert them directly, and query with dense, sparse, BM25, or hybrid search. No collections or extractors required.
<Card title="Create your free vector namespace" icon="database" href="https://studio.mixpeek.com/namespaces/create?mode=standalone&utm_source=docs&utm_medium=vector_store&utm_campaign=mvs_onramp&utm_content=vector_store_overview">
Bring your own vectors and run your first search in minutes — your first 1M vectors are free, no extractors or schema required.
</Card>
## Quickstart
<Steps>
<Step title="Create a namespace">
No schema needed — MVS infers vector dimensions on first write.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/standalone" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"namespace_id": "product-search"}'
```
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="mxp_sk_...")
client.namespaces.create(
namespace_id="product-search",
mode="standalone",
)
```
</CodeGroup>
You can optionally pre-declare vector configs if you want to set a specific distance metric:
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/standalone" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"namespace_id": "product-search",
"vector_configs": [
{"name": "text_embedding", "dimension": 1536, "metric": "dot"}
]
}'
```
```python Python theme={null}
client.namespaces.create(
namespace_id="product-search",
mode="standalone",
vector_configs=[
{"name": "text_embedding", "dimension": 1536, "metric": "dot"}
],
)
```
</CodeGroup>
</Step>
<Step title="Upsert documents with your vectors">
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/product-search/documents/upsert" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"documents": [{
"document_id": "prod-001",
"vectors": {"text_embedding": [0.12, -0.34, 0.56, "...1536 floats"]},
"payload": {"title": "Wireless Headphones", "category": "audio", "price": 79.99}
}]
}'
```
```python Python theme={null}
client.namespaces.documents.upsert(
namespace_id="product-search",
documents=[{
"document_id": "prod-001",
"vectors": {"text_embedding": [0.12, -0.34, 0.56]}, # 1536 floats
"payload": {"title": "Wireless Headphones", "category": "audio", "price": 79.99},
}],
)
```
</CodeGroup>
</Step>
<Step title="Create a retriever (one-time)">
Querying is unified on **retrievers** — one query concept whether you bring your own vectors or promote to managed embedding. Create a retriever once; for a standalone namespace it takes the query vector you computed (`input_mode: vector`). All requests are scoped to the namespace via the `X-Namespace` header.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: product-search" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "product_search",
"input_schema": {"query_vector": {"type": "array", "required": true}},
"stages": [{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "text_embedding",
"query": {"input_mode": "vector", "value": "{{INPUT.query_vector}}"},
"filters": {"must": [{"key": "category", "match": {"value": "audio"}}]},
"top_k": 10
}],
"final_top_k": 10
}
}
}]
}'
```
```python Python theme={null}
retriever = client.retrievers.create(
namespace_id="product-search",
retriever_name="product_search",
input_schema={"query_vector": {"type": "array", "required": True}},
stages=[{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "text_embedding",
"query": {"input_mode": "vector", "value": "{{INPUT.query_vector}}"},
"filters": {"must": [{"key": "category", "match": {"value": "audio"}}]},
"top_k": 10,
}],
"final_top_k": 10,
},
},
}],
)
```
</CodeGroup>
</Step>
<Step title="Execute the retriever">
Run the retriever with your query embedding. Each hit exposes `document_id` and payload fields.
<CodeGroup>
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: product-search" \
-H "Content-Type: application/json" \
-d '{
"inputs": {"query_vector": [0.15, -0.28, 0.44, "...query embedding"]}
}'
```
```python Python theme={null}
results = client.retrievers.execute(
namespace_id="product-search",
retriever_id=retriever.retriever_id,
inputs={"query_vector": [0.15, -0.28, 0.44]}, # query embedding
)
```
</CodeGroup>
</Step>
</Steps>
## Architecture
<Frame>
<img alt="Standalone vector store architecture — your pipeline feeds vectors into Mixpeek's sharded storage with dense, BM25, and payload indexes" />
</Frame>
## Scaling
The index **auto-partitions as you grow** — there's no capacity planning, resharding, or replica provisioning on your side, and no per-vector or per-namespace caps. The same namespace serves thousands or hundreds of millions of vectors.
At large scale (10M–100M+ vectors), two things matter:
* **Tiering.** Hot (in-memory) vectors serve at \~10ms; cold (object-storage) vectors serve at \~100ms via brute-force scan. The vector store keeps your actively-queried set resident automatically — there's no manual tier management on your side.
* **Shard visibility.** The `shards.hot` / `shards.cold` counts in [namespace usage metrics](/docs/vector-store/namespaces) show how the index has partitioned. Dedicated/enterprise deployments can tune shard and index parameters — see [Single-Tenant](/docs/resources/single-tenant).
<Note>
Need concrete latency/recall targets for a specific corpus size and query rate? [Talk to engineers](https://mixpeek.com/contact) — sizing depends on dimensions, filter selectivity, and your hot/cold split.
</Note>
## Standalone vs Managed
Every namespace runs in one of two modes. Start standalone and [promote](/docs/vector-store/promote) when you're ready — no reindexing.
| | Standalone | Managed |
| --------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- |
| **Query latency** | Lower — no embedding at query time | +50-200ms for auto-embedding |
| **Embedding cost** | You pay your provider directly | Included in platform pricing |
| **Model flexibility** | Any model, any fine-tune | Bound to registered inference services |
| **Write path** | Direct upsert only | Collections auto-process + direct upsert |
| **Search input** | Pre-computed vectors, text (BM25), sparse | Also accepts raw text/URLs (auto-embedded) |
| **Feature URIs** | Plain feature names only (e.g. `text_embedding`) — `mixpeek://…` extractor-backed URIs are **not** available | Full `mixpeek://<extractor>@<version>/<model>` URIs |
| **Best for** | Existing ML infra, low-latency, custom models | End-to-end processing, file pipelines |
<Tip>
Start standalone if you already have embeddings. Promotion is additive — all existing data is preserved.
</Tip>
## Features
All features work identically in standalone and managed modes.
| Capability | Description |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Dense search** | Vector similarity with cosine, dot product, or euclidean distance |
| **Sparse search** | Sparse vector queries for learned sparse representations (SPLADE, etc.) |
| **BM25 keyword search** | Full-text search on payload fields via [text indexes](/docs/vector-store/namespaces#text-indexes-bm25) |
| **Hybrid search** | Combine dense + BM25 + sparse in one query with [RRF or DBSF fusion](/docs/vector-store/documents#hybrid) |
| **Metadata filtering** | Filter on any payload field — combine with any search type |
| **Payload indexes** | Manual or [adaptive](/docs/vector-store/namespaces#adaptive-payload-indexes) — auto-created based on query patterns |
| **Schema-on-write** | [Auto-create vector indexes](/docs/vector-store/namespaces#schema-on-write) on first upsert — no upfront declaration needed |
| **Usage metrics** | [Per-namespace breakdowns](/docs/vector-store/namespaces#usage-metrics) of vectors, queries, and writes |
| **Namespace cloning** | Clone namespaces for testing or environment branching |
## Billing
MVS pricing is **pure usage-based** — no per-vector caps, no namespace limits. Tiers gate support level, not features.
| Resource | Price |
| ------------- | -------------------- |
| **Storage** | \$0.023 / GB / month |
| **Hot cache** | \$25 / GB / month |
| **Queries** | \$1 / 1M queries |
| **Writes** | \$1 / 1M writes |
Your first **1M vectors are free** on the Starter tier.
| Tier | Minimum | Support |
| -------------- | -------------------------------------- | ----------------------- |
| **Starter** | \$0/mo | Community |
| **Growth** | \$50/mo (usage applies toward minimum) | Email + SLA |
| **Enterprise** | Custom | Dedicated + SSO + HIPAA |
All search features (dense, sparse, BM25, hybrid, adaptive indexes) are available on every tier. See the [pricing calculator](https://mixpeek.com/mvs#pricing) for cost estimates at scale.
Track usage programmatically with the [vector-backend usage endpoint](/docs/api-reference/organization-billing/get-vector-backend-usage) (`GET /v1/organizations/billing/usage/vector-backend`) or view it in the Studio dashboard under **Billing**.
## Next Steps
<CardGroup>
<Card title="Namespaces" icon="database" href="/vector-store/namespaces">
Vector indexes, metrics, BM25
</Card>
<Card title="Documents & Search" icon="magnifying-glass" href="/vector-store/documents">
Upsert, query, manage
</Card>
<Card title="Promote" icon="arrow-up" href="/vector-store/promote">
Standalone → managed
</Card>
</CardGroup>
***
<Note>
**Ready to go beyond BYO vectors?** Promote your standalone namespace to **managed mode** and add automatic embedding, file processing pipelines, and enrichment — without reindexing. Your retrievers keep working unchanged — after promotion, the same retriever can auto-embed raw text instead of taking a pre-computed vector. [See the migration guide](/docs/vector-store/promote#querying-with-retrievers) for details. [Learn how to promote →](/docs/vector-store/promote)
</Note>
# Promote to Managed
Source: https://docs.mixpeek.com/docs/vector-store/promote
Transition from standalone vectors to auto-embedding and collections — no reindexing required
Promotion converts a standalone namespace to managed mode. Existing vectors and documents stay in place. Mixpeek adds auto-embedding for queries and collection-driven processing for new content.
<Frame>
<img alt="Promotion flow — standalone namespace with vectors becomes managed with inference mappings, no reindexing" />
</Frame>
## Promote
Map existing vector indexes to inference services and optionally add new ones.
<CodeGroup>
```python Python theme={null}
from mixpeek import Mixpeek
client = Mixpeek(api_key="mxp_sk_...")
client.namespaces.promote(
namespace_id="product-search",
vector_mappings=[
{"existing_index": "text_embedding", "inference_service": "openai_text_3_small"}
],
add_vectors=[
{"name": "image_embedding", "dimension": 512, "metric": "cosine", "inference_service": "clip_vit_b32"}
],
)
```
```bash cURL theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/product-search/promote" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"vector_mappings": [
{"existing_index": "text_embedding", "inference_service": "openai_text_3_small"}
],
"add_vectors": [
{"name": "image_embedding", "dimension": 512, "metric": "cosine", "inference_service": "clip_vit_b32"}
]
}'
```
</CodeGroup>
### Parameters
<ParamField type="array">
Map existing indexes to inference services for auto-embedding queries.
| Field | Description |
| ------------------- | --------------------------------------------------------- |
| `existing_index` | Name of an existing vector index |
| `inference_service` | Model to auto-embed queries (e.g., `openai_text_3_small`) |
</ParamField>
<ParamField type="array">
New vector indexes to create during promotion.
| Field | Description |
| ------------------- | ------------------------------------------------ |
| `name` | New index name (must not conflict with existing) |
| `dimension` | Embedding dimension |
| `metric` | `cosine`, `dot`, or `euclidean` |
| `inference_service` | Model for auto-embedding (optional) |
</ParamField>
### Response
```json theme={null}
{
"namespace_id": "product-search",
"previous_mode": "standalone",
"mode": "managed",
"status": "active",
"vector_configs": [
{"name": "text_embedding", "dimension": 1536, "metric": "cosine"},
{"name": "image_embedding", "dimension": 512, "metric": "cosine"}
],
"vector_inference_map": {
"text_embedding": "openai_text_3_small",
"image_embedding": "clip_vit_b32"
}
}
```
## What Changes
| | Before (standalone) | After (managed) |
| ----------------- | --------------------------------- | ------------------------------------------ |
| **Search input** | Must provide pre-computed vectors | Also accepts raw text/URLs — auto-embedded |
| **New content** | Direct upsert only | Collections + extractors auto-process |
| **Existing data** | — | Preserved, no reindexing |
| **Direct upsert** | — | Still works alongside collections |
## Rules
<Warning>
Promotion is one-way. Managed namespaces cannot be demoted.
</Warning>
* Only `standalone` namespaces can be promoted
* `existing_index` must reference an index that exists — errors list available indexes
* `name` in `add_vectors` must not conflict with existing indexes
* Promotion is atomic — if validation fails, nothing changes
* Both `vector_mappings` and `add_vectors` are optional (promote with just one or neither)
## Full Workflow
<Steps>
<Step title="Start standalone">
Create a namespace and upsert your existing embeddings.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/standalone" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"namespace_id": "legal-docs", "vector_configs": [{"name": "text_embedding", "dimension": 1536, "metric": "cosine"}]}'
```
</Step>
<Step title="Load and validate">
Upsert documents and verify retrieval quality before promoting.
```bash theme={null}
# Upsert
curl -X POST "https://api.mixpeek.com/v1/namespaces/legal-docs/documents/upsert" ...
# Execute a retriever to validate (input_mode: vector — you pass the query embedding)
curl -X POST "https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute" \
-H "X-Namespace: legal-docs" \
-d '{"inputs": {"query_vector": [...]}}'
```
</Step>
<Step title="Promote">
Map your embedding to the matching inference service.
```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/namespaces/legal-docs/promote" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"vector_mappings": [{"existing_index": "text_embedding", "inference_service": "openai_text_3_small"}]}'
```
</Step>
<Step title="Switch queries to text input">
Your retrievers already run before and after promotion — no API change. Update the query stage from `input_mode: vector` (you pass the embedding) to `input_mode: text` so Mixpeek auto-embeds. See [Querying with Retrievers](#querying-with-retrievers) below.
</Step>
<Step title="Use managed features">
Create collections with extractors for new content. Existing documents coexist with extractor-processed documents.
</Step>
</Steps>
## Querying with Retrievers
Querying is unified on **retrievers** in both standalone and managed modes — you learn one query concept regardless of whether you bring your own vectors or let Mixpeek embed for you. Promotion doesn't change *how* you query; it only changes *what you pass in*. A standalone retriever takes the query vector you computed (`input_mode: vector`); after promotion the same retriever can take raw text and auto-embed it (`input_mode: text`).
### Before promotion: pass your own vector
The query stage runs in `input_mode: vector` — you compute the embedding and pass it in `inputs`.
```bash theme={null}
# Step 1 — Create a retriever (one-time)
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: product-search" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "product_search",
"input_schema": {"query_vector": {"type": "array", "required": true}},
"stages": [{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "text_embedding",
"query": {"input_mode": "vector", "value": "{{INPUT.query_vector}}"},
"filters": {"must": [{"key": "category", "match": {"value": "audio"}}]},
"top_k": 10
}],
"final_top_k": 10
}
}
}]
}'
# Step 2 — Execute with your query embedding
curl -X POST "https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: product-search" \
-H "Content-Type: application/json" \
-d '{
"inputs": {"query_vector": [0.12, -0.34, 0.56, "...1536 floats"]}
}'
```
### After promotion: pass raw text
Once `text_embedding` is mapped to an inference service, switch the query stage to `input_mode: text`. Mixpeek embeds the text for you — no vectors needed.
```bash theme={null}
# Step 1 — Create (or update) the retriever to take text
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: product-search" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "product_search",
"input_schema": {"q": {"type": "string", "required": true}},
"stages": [{
"stage_name": "search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [{
"feature_uri": "text_embedding",
"query": {"input_mode": "text", "value": "{{INPUT.q}}"},
"filters": {"must": [{"key": "category", "match": {"value": "audio"}}]},
"top_k": 10
}],
"final_top_k": 10
}
}
}]
}'
# Step 2 — Execute with raw text (auto-embedded)
curl -X POST "https://api.mixpeek.com/v1/retrievers/{retriever_id}/execute" \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: product-search" \
-H "Content-Type: application/json" \
-d '{
"inputs": {"q": "wireless headphones"}
}'
```
Each hit in the response exposes `document_id` plus the document's payload fields.
### What Changes
| | Standalone (`input_mode: vector`) | Managed (`input_mode: text`) |
| ----------------- | ------------------------------------------------- | ---------------------------------------------------------- |
| **Query input** | Raw vectors you computed (also text/BM25, sparse) | Raw text, URLs, filters — auto-embedded |
| **Embedding** | You compute and pass vectors | Retriever stages auto-embed via inference services |
| **Query path** | Retriever create + execute | Retriever create + execute — same API |
| **Pipeline** | Single search stage + optional fusion | Multi-stage: search → filter → enrich → rerank → transform |
| **Hybrid search** | Multiple `feature_search` searches + fusion | Multiple `feature_search` stages + sort/reduce stages |
<Tip>
Promotion is additive — your retrievers keep working. Flip the query stage to `input_mode: text` whenever you're ready to let Mixpeek handle embedding, and layer in multi-stage pipelines as your needs grow.
</Tip>
## Related
* [Vector Store Overview](/docs/vector-store/overview)
* [Namespace Configuration](/docs/vector-store/namespaces)
* [Collections](/docs/ingestion/collections) — set up auto-processing after promotion
* [Feature Extractors](/docs/processing/feature-extractors) — available extractors
# Add Objects to Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/add-objects-to-batch
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/objects
Add objects to an existing batch. The batch must be in 'draft' status.
# Create Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/create-batch
post /v1/buckets/{bucket_identifier}/batches
Create a new batch for grouping bucket objects.
# Aggregate Objects
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/aggregate-objects
post /v1/buckets/{bucket_identifier}/objects/aggregate
This endpoint performs aggregation operations on objects in a bucket.
**Aggregation Framework**: Provides MongoDB-style aggregation operations:
- GROUP BY: Group objects by one or more fields
- Aggregations: COUNT, SUM, AVG, MIN, MAX, COUNT_DISTINCT, etc.
- Date Operations: Truncate or extract date parts for time-series analysis
- Filtering: Pre-aggregation filters (WHERE) and post-aggregation filters (HAVING)
- Sorting & Limiting: Control result ordering and size
**Use Cases**:
- Count objects by status or category
- Calculate daily/monthly upload statistics
- Analyze content distribution and trends
- Generate reports with multiple metrics
**Note**: This endpoint works with both MongoDB objects and Qdrant documents
using the same interface. The system automatically selects the appropriate
aggregation provider.
# Create Object
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/create-object
post /v1/buckets/{bucket_identifier}/objects
This endpoint creates a new object in the specified bucket.
The object must conform to the bucket's schema.
**Processing**: By default, objects are created in DRAFT status and require
batch submission for processing. Set `auto_process=true` to automatically
create a batch and submit it for processing (zero-touch workflow).
If the bucket has a unique_key configured, the insertion policy determines behavior:
- insert: Create only. Fail with 409 Conflict if unique key exists.
- update: Update only. Fail with 404 Not Found if unique key doesn't exist.
- upsert: Create if new, update if exists (idempotent).
Policy resolution:
1. Use ?policy= query parameter if provided
2. Fall back to bucket's default_policy if configured
3. Error 400 if neither is specified
# Create Objects in Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/create-objects-in-batch
post /v1/buckets/{bucket_identifier}/objects/batch
This endpoint creates multiple new objects in the specified bucket as a batch.
Each object must conform to the bucket's schema.
**Processing**: By default, objects are created in DRAFT status and require
batch submission for processing. Set `auto_process=true` to automatically
create a processing batch and submit it (zero-touch workflow).
**Partial Success**: This endpoint uses partial success - valid objects are created
even if some fail validation. Failed objects are returned separately with error details,
allowing you to fix and retry only the failed ones.
**Response**: Returns both succeeded and failed objects. The batch succeeds (200 OK) as long
as at least one object is created. Check the `failed` array for objects that need attention.
# Delete Object
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/delete-object
delete /v1/buckets/{bucket_identifier}/objects/{object_identifier}
This endpoint deletes an object from the specified bucket.
# Get Object
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/get-object
get /v1/buckets/{bucket_identifier}/objects/{object_identifier}
This endpoint retrieves an object by its ID from the specified bucket.
**Presigned URLs**: Set `return_presigned_urls=true` query parameter to generate fresh presigned download URLs
for all blobs with S3 storage (default: false). URLs are added to each blob's properties as
`presigned_url` and expire after 1 hour.
**Document count**: `document_count` (how many documents this object produced, via vector-store
lineage) is computed by default. It fans out to the vector partition, which on a cold/serverless
partition can be slow — pass `include_document_count=false` to skip it for latency-sensitive,
interactive views (e.g. an object-detail modal) that don't render the count. Even when requested,
the count is best-effort and bounded by a deadline: it returns `null` rather than blocking the
response if the partition is cold.
# List Objects
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/list-objects
post /v1/buckets/{bucket_identifier}/objects/list
This endpoint lists objects in a bucket with cursor-based pagination, filtering, and sorting.
**Filtering**: Use dot notation for metadata fields
- Example: ?metadata.type=video&metadata.status=ready
**Sorting**: Specify field and direction
- Example: ?sort_field=metadata.created_at&sort_direction=desc
- Direction: asc (ascending) or desc (descending), defaults to asc
**Pagination**: Cursor-based for efficient deep pagination
- First page: ?limit=100 (omit cursor)
- Next pages: ?limit=100&cursor={next_cursor}
- Use next_cursor from response to navigate
- No limit on pagination depth
**Total Count**: Optional (expensive operation)
- Use ?include_total=true to get total count
- Adds 50-200ms to response time
- Returns total, page, page_size, total_pages fields in pagination response
# Partially Update Object
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/partially-update-object
patch /v1/buckets/{bucket_identifier}/objects/{object_identifier}
This endpoint partially updates an existing object in the specified bucket (PATCH operation).
Only provided fields will be updated. At minimum, metadata can always be updated.
Immutable fields like object_id and timestamps cannot be modified.
It does not trigger processing.
# Update Object
Source: https://docs.mixpeek.com/docs/api-reference/bucket-objects/update-object
put /v1/buckets/{bucket_identifier}/objects/{object_identifier}
This endpoint updates an existing object in the specified bucket.
The updated object must conform to the bucket's schema. It does not trigger processing.
# Bulk Mark Dlq Permanently Failed
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/bulk-mark-dlq-permanently-failed
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/dlq/bulk-mark-failed
Mark DLQ entries as permanently failed by error type.
Stops automatic retries for entries matching the specified error types.
Use this for structurally unrecoverable errors (e.g. missing proxy URLs,
unsupported file types) that will never succeed on retry.
# Create Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/create-sync-configuration
post /v1/buckets/{bucket_id}/syncs
Create a sync configuration for automated storage ingestion.
Establishes automated synchronization between an external storage provider
and a Mixpeek bucket. The sync monitors the source path and ingests files
according to the specified mode and filters.
**Supported Providers:** google_drive, s3, snowflake, sharepoint, tigris
**Built-in Robustness:**
- Dead Letter Queue (DLQ): Failed objects tracked with 3 retries
- Idempotent ingestion: Deduplication prevents duplicate objects
- Distributed locking: Prevents concurrent sync execution
- Rate limit handling: Automatic backoff on 429 responses
- Metrics: Duration, files synced/failed, batches created
**Sync Modes** (the request enum accepts exactly these two):
- `initial_only`: Single bulk import, then sync stops
- `continuous`: Polling-based monitoring (``polling_interval_seconds``,
max 900)
(Docstring previously advertised ``one_time``/``scheduled``, which the
enum rejects with a 422 — modes here must match ``SyncCreateRequest``.)
# Delete Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/delete-sync-configuration
delete /v1/buckets/{bucket_id}/syncs/{sync_config_id}
Delete a sync configuration.
# Force Unlock Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/force-unlock-sync-configuration
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/unlock
Force-release a distributed lock on a sync configuration.
Use this when a sync trigger returns 'skipped_already_running' but no sync
is actually executing (zombie lock from a crashed worker).
# Get Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/get-sync-configuration
get /v1/buckets/{bucket_id}/syncs/{sync_config_id}
Fetch a sync configuration.
# Get Sync Job
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/get-sync-job
get /v1/buckets/{bucket_id}/syncs/{sync_config_id}/jobs/{sync_job_id}
Get details for a specific sync job.
Returns the full job record including status, file counts,
start/completion times, and any error messages.
# Get Sync Metrics
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/get-sync-metrics
get /v1/buckets/{bucket_id}/syncs/{sync_config_id}/metrics
Operational metrics for a sync in one call.
Folds config totals + health/staleness + the DLQ failure breakdown (grouped
by normalized error reason) + the most-recent job's throughput into a single
response — so callers can see "is this sync healthy / is anything backlogged"
without stitching /jobs and /dlq or reading logs.
# List Dlq Entries
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/list-dlq-entries
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/dlq
List Dead Letter Queue entries for a sync configuration.
Returns objects that failed to sync after all retry attempts.
Each entry includes the source object ID, error details, retry count,
and timestamps. Use this to diagnose and resolve sync failures.
# List Sync Configurations
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/list-sync-configurations
post /v1/buckets/{bucket_id}/syncs/list
List all sync configurations for a bucket with optional filtering.
Returns paginated list of sync configurations with status, metrics, and settings.
Use filters to find syncs by connection, status, or active state.
# List Sync Jobs
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/list-sync-jobs
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/jobs
List sync job history for a sync configuration.
Returns paginated job records ordered by most recent first.
Each job includes status, file counts, timing, and error details.
Use the optional status filter to find failed or running jobs.
# Pause Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/pause-sync-configuration
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/pause
Pause a sync configuration.
Pauses the sync, preventing new sync jobs from executing.
Running jobs will complete but no new jobs will be scheduled.
The sync configuration and all settings are preserved.
**Note:** Use POST /resume to reactivate the sync.
# Requeue Dlq Entries
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/requeue-dlq-entries
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/dlq/requeue
Return exhausted DLQ entries to the retry path (TG-2999).
The inverse of bulk-mark-failed. Use after fixing the underlying cause (a
restored proxy URL, a storage permission, an upstream outage) to re-drive the
entries that gave up. Without this, the only way to re-drive a DLQ was to
lean on natural sync-diff and hope the objects were rediscovered.
Resets `max_retries_reached` and `retry_count`, stamps `requeued_at`, and
clears `disposal_reason` so the retry task does not immediately re-retire the
entry. Only entries that have EXHAUSTED their retries are touched; anything
still retrying is left alone rather than silently granted extra attempts.
Omit `error_types` to requeue everything exhausted for the sync config.
# Resume Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/resume-sync-configuration
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/resume
Resume a paused sync configuration.
Reactivates a paused sync, allowing new sync jobs to be scheduled.
For continuous syncs, polling will resume at the configured interval.
The next sync will be incremental (only files modified since last sync).
# Trigger Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/trigger-sync-configuration
post /v1/buckets/{bucket_id}/syncs/{sync_config_id}/trigger
Manually trigger a sync execution.
Creates a sync job and immediately dispatches it for async execution via Celery.
The sync job processes files from the storage provider and creates objects in the bucket.
**Execution Flow:**
1. Acquires distributed lock (prevents concurrent runs)
2. Iterates files from storage provider with configured filters
3. Creates objects idempotently (skips duplicates)
4. Failed objects go to Dead Letter Queue (3 retries)
5. Creates batches for collection processing (100 objects/batch)
6. Emits metrics and releases lock
**Use Cases:**
- Test sync configuration after creation
- Force sync outside of scheduled intervals
- Re-sync after updating connection credentials
- Trigger incremental sync (only modified files)
**Returns:** `sync_job_id` to track progress via GET /syncs/{id}
# Update Sync Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-syncs/update-sync-configuration
patch /v1/buckets/{bucket_id}/syncs/{sync_config_id}
Update a sync configuration.
# Batch Confirm Uploads
Source: https://docs.mixpeek.com/docs/api-reference/bucket-uploads/batch-confirm-uploads
post /v1/buckets/{bucket_identifier}/uploads/confirm/batch
Confirm multiple uploads in a single request (processed asynchronously).
Maximum 100 confirmations per batch.
All uploads must belong to the same bucket.
Returns a task_id to track progress via GET /v1/tasks/{task_id}.
# Batch Create Uploads
Source: https://docs.mixpeek.com/docs/api-reference/bucket-uploads/batch-create-uploads
post /v1/buckets/{bucket_identifier}/uploads/batch
Generate multiple presigned URLs in a single request.
All uploads belong to the same bucket (from path parameter).
Maximum 100 uploads per batch.
Shared metadata is merged with individual upload metadata (individual takes precedence).
# Confirm Upload
Source: https://docs.mixpeek.com/docs/api-reference/bucket-uploads/confirm-upload
post /v1/buckets/{bucket_identifier}/uploads/{upload_id}/confirm
Verify S3 upload completion and create bucket object.
After uploading to S3 using the presigned URL, call this endpoint to:
1. Verify the file exists in S3
2. Validate ETag and file size (if provided)
3. Create bucket object (default, unless create_object_on_confirm=false)
4. Update upload status to COMPLETED
**Sync vs Async**:
- Files < 100MB: Processed synchronously (~100ms)
- Files >= 100MB or async=true: Processed asynchronously (returns task_id)
**Duplicate Detection**:
- If file hash matches existing upload, marks as duplicate
- References original object_id if available
# Create Upload
Source: https://docs.mixpeek.com/docs/api-reference/bucket-uploads/create-upload
post /v1/buckets/{bucket_identifier}/uploads
Generate a presigned URL for direct S3 upload.
This endpoint validates all requirements BEFORE generating the presigned URL,
ensuring immediate feedback if something is wrong (bucket inactive, quota exceeded, etc.).
**Duplicate Detection (Enabled by Default)**:
- If `file_hash` provided and `skip_duplicates=true`: Checks for existing upload
- If duplicate found: Returns existing upload (200 OK) with `is_duplicate=true`
- If new file: Returns presigned URL (201 Created) with `is_duplicate=false`
**Two-Step Flow**:
1. Call this endpoint → Get presigned URL
2. PUT file to presigned URL → Upload directly to S3
3. Call confirm endpoint → Verify upload and create object
# Delete Upload
Source: https://docs.mixpeek.com/docs/api-reference/bucket-uploads/delete-upload
delete /v1/buckets/{bucket_identifier}/uploads/{upload_id}
Cancel an upload and optionally delete the S3 object.
Cannot cancel uploads with status COMPLETED.
Can cancel uploads with status: PENDING, IN_PROGRESS, FAILED.
# Get Upload
Source: https://docs.mixpeek.com/docs/api-reference/bucket-uploads/get-upload
get /v1/buckets/{bucket_identifier}/uploads/{upload_id}
Retrieve an upload by its ID.
Use this to check upload status, get S3 key, or retrieve created object_id after confirmation.
**Presigned URLs**: Set `return_presigned_urls=true` query parameter to generate fresh presigned download URLs (default: false).
The presigned URLs expire after 1 hour and allow direct download from S3.
# List Uploads
Source: https://docs.mixpeek.com/docs/api-reference/bucket-uploads/list-uploads
post /v1/buckets/{bucket_identifier}/uploads/list
List uploads in a bucket with filtering, sorting, search, and pagination.
**Filtering**: Use LogicalOperator with shorthand syntax
- Simple: `{"status": "PENDING", "metadata.campaign": "summer_2024"}`
- Complex: `{"AND": [{"field": "file_size_bytes", "operator": "gte", "value": 1000000}]}`
**Sorting**: Specify field and direction
- Example: `{"field": "created_at", "direction": "desc"}`
**Search**: Full-text search across filename and metadata
- Example: `"search": "video"`
**Pagination**: Use limit and offset
- Example: `"limit": 50, "offset": 100`
# Create Bucket
Source: https://docs.mixpeek.com/docs/api-reference/buckets/create-bucket
post /v1/buckets
This endpoint allows you to create a new bucket with a defined schema.
A bucket is a collection of objects that conform to the schema.
The schema defines the structure and validation rules for objects in the bucket.
# Delete Bucket
Source: https://docs.mixpeek.com/docs/api-reference/buckets/delete-bucket
delete /v1/buckets/{bucket_identifier}
This endpoint deletes a bucket and all its resources including:
- S3 objects and blobs
- Running Ray jobs (cancels active batch processing jobs)
- Batch processing artifacts
- Upload files
- Unique key lookups
- MongoDB metadata
The deletion is performed **asynchronously** via a background task.
Returns immediately with a task_id that can be polled via GET /v1/tasks/{task_id}.
**Response**:
- `task_id`: Use this to poll deletion status via GET /v1/tasks/{task_id}
- `status`: Initial status (PENDING)
- `bucket_id`: The bucket being deleted
- `bucket_name`: Name of the bucket
- `object_count`: Number of objects that will be deleted
**Polling**:
Poll GET /v1/tasks/{task_id} until status is COMPLETED or FAILED.
Use exponential backoff (start 1s, max 30s).
# Get Bucket
Source: https://docs.mixpeek.com/docs/api-reference/buckets/get-bucket
get /v1/buckets/{bucket_identifier}
This endpoint retrieves a bucket by its ID.
# List Buckets
Source: https://docs.mixpeek.com/docs/api-reference/buckets/list-buckets
post /v1/buckets/list
This endpoint lists buckets with pagination, sorting, and filtering options.
# Partially Update Bucket
Source: https://docs.mixpeek.com/docs/api-reference/buckets/partially-update-bucket
patch /v1/buckets/{bucket_identifier}
This endpoint allows you to partially update an existing bucket (PATCH operation).
Only provided fields will be updated. At minimum, metadata can always be updated.
Immutable fields like bucket_id and timestamps cannot be modified.
# Update Bucket
Source: https://docs.mixpeek.com/docs/api-reference/buckets/update-bucket
put /v1/buckets/{bucket_identifier}
This endpoint allows you to update an existing bucket.
You can update the bucket's name, description, and metadata.
# Delete a model
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/delete-a-model
delete /v1/namespaces/{namespace_id}/models/{model_id}
Delete a custom model from the namespace.
# Deploy model to Ray object store
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/deploy-model-to-ray-object-store
post /v1/namespaces/{namespace_id}/models/{model_id}/deploy
Pre-load model weights into the Ray object store for fast access by plugins.
This operation:
1. Downloads the model archive from S3
2. Deserializes weights based on format (safetensors, pytorch, etc.)
3. Stores weights in Ray object store for zero-copy sharing
After deployment, plugins can load the model instantly using:
```python
from engine.models.loader import load_namespace_model
weights = load_namespace_model("model_id")
```
**Note:** This is optional - models are also loaded on-demand when plugins
first request them. Use this endpoint to pre-warm the cache.
# Disable org model for namespace
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/disable-org-model-for-namespace
post /v1/namespaces/{namespace_id}/models/org/{model_id}/disable
Disable an org-level model for this namespace.
This removes the namespace-specific deployment record. The model
remains available at the organization level and can be re-enabled.
# Enable org model for namespace
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/enable-org-model-for-namespace
post /v1/namespaces/{namespace_id}/models/org/{model_id}/enable
Enable an org-level model for this namespace.
This creates a namespace-specific deployment record for the model,
allowing it to be used within this namespace. Optionally deploys
the model to Ray immediately.
**Note:** The model must first be registered at the organization level
via POST /models/uploads.
# Get model details
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/get-model-details
get /v1/namespaces/{namespace_id}/models/{model_id}
Get detailed information about a specific model.
# List available org models
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/list-available-org-models
get /v1/namespaces/{namespace_id}/models/available
List org-level models available to enable in this namespace.
Shows all models registered at the organization level and whether
they are already enabled in the target namespace.
# List models in namespace
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/list-models-in-namespace
get /v1/namespaces/{namespace_id}/models
List all custom models uploaded to the namespace.
# Upload a custom model
Source: https://docs.mixpeek.com/docs/api-reference/custom-models/upload-a-custom-model
post /v1/namespaces/{namespace_id}/models
Upload custom model weights to the namespace.
**Requirements:**
- Organization must be on Enterprise tier
**Supported Model Formats:**
- `safetensors`: SafeTensors format (recommended for transformers)
- `onnx`: ONNX Runtime format
- `pytorch`: PyTorch state_dict or TorchScript
- `huggingface`: HuggingFace model directory
**Base Images (auto-selected based on format):**
- `mixpeek/serve-gpu:latest`: For safetensors, pytorch, huggingface (includes torch, transformers)
- `mixpeek/serve-minimal:latest`: For ONNX (includes onnxruntime only)
**Important:** Models run in fixed base images. You cannot install additional pip packages.
All required frameworks (torch, transformers, onnxruntime) are pre-installed.
# List Dsar Requests
Source: https://docs.mixpeek.com/docs/api-reference/data-subject-access-requests/list-dsar-requests
get /v1/organizations/dsar/status
List all DSAR requests for this organization.
Returns the status of all pending and completed data subject
access requests (exports and deletions).
Requires ADMIN permission.
# Request Data Deletion
Source: https://docs.mixpeek.com/docs/api-reference/data-subject-access-requests/request-data-deletion
post /v1/organizations/dsar/delete
Request deletion of all organization data.
This is a DSAR deletion request per GDPR Article 17 (Right to Erasure)
and CCPA Section 1798.105. The request is logged and processed within
30 days. Once executed, this action is IRREVERSIBLE.
The actual deletion is performed by the delete_organization_resources_flow
after verification. This endpoint creates the request record.
Requires ADMIN permission.
# Request Data Export
Source: https://docs.mixpeek.com/docs/api-reference/data-subject-access-requests/request-data-export
post /v1/organizations/dsar/export
Request a full data export for this organization.
Returns a summary of all data categories that will be exported.
The export is processed asynchronously and delivered within 30 days
as required by GDPR Article 15 and CCPA Section 1798.100.
Requires ADMIN permission.
# Clone Namespace
Source: https://docs.mixpeek.com/docs/api-reference/namespace-clone/clone-namespace
post /v1/namespaces/{namespace_identifier}/clone
Clone a namespace with all its data.
**What gets cloned:**
- Namespace configuration (extractors, payload indexes)
- Buckets (metadata, references same S3 files)
- Collections (full copy of all vectors/embeddings)
- Retrievers (pipeline configuration)
**Use Cases:**
- Create staging environment from production
- Backup namespace with all data
- Fork namespace for experimentation
**For config-only copy (no data), use templates instead:**
- POST /templates/namespaces/from-namespace/{id}
- POST /templates/namespaces/{template_id}/instantiate
# Get extractor details
Source: https://docs.mixpeek.com/docs/api-reference/namespace-extractors/get-extractor-details
get /v1/namespaces/{namespace_id}/extractors/{extractor_id}
Get detailed information about a specific extractor.
Works for both builtin extractors and custom plugins.
**Parameters:**
- `extractor_id`: Extractor identifier (e.g., 'text_extractor_v1', 'my_custom_plugin_1_0_0')
**Response includes:**
- Full schema information (input, output, parameters)
- Vector index configuration
- For custom plugins: deployment status, validation status
# List all extractors available to namespace
Source: https://docs.mixpeek.com/docs/api-reference/namespace-extractors/list-all-extractors-available-to-namespace
get /v1/namespaces/{namespace_id}/extractors
List all feature extractors available for use in this namespace.
This endpoint returns a **unified view** combining:
- **Builtin extractors**: Core extractors shipped with Mixpeek (text_extractor, image_extractor, etc.)
- **Custom plugins**: User-uploaded plugins at org or namespace level (Enterprise)
Each extractor includes:
- `input_schema`: JSON schema for input data validation
- `output_schema`: JSON schema for output document structure
- `parameter_schema`: JSON schema for configurable parameters
- `required_vector_indexes`: Vector indexes produced by this extractor
- `feature_uri`: URI to reference this extractor in collections
**Use Cases:**
- Discover available extractors when creating collections
- Get schema information for SDK code generation
- Check which custom plugins are deployed
**Filtering:**
- `source=builtin`: Only builtin extractors
- `source=custom`: Only custom plugins
- `source=all` (default): All extractors
# Cancel Migration
Source: https://docs.mixpeek.com/docs/api-reference/namespace-migrations/cancel-migration
post /v1/namespaces/migrations/{migration_id}/cancel
Cancel a running migration.
Args:
request: FastAPI request
migration_id: Migration ID
cancel_request: Cancellation options
Returns:
CancelMigrationResponse with updated status
# Create Migration
Source: https://docs.mixpeek.com/docs/api-reference/namespace-migrations/create-migration
post /v1/namespaces/migrations/
Create a new namespace migration.
This endpoint creates a migration and optionally validates it.
Use start_immediately=True to begin execution immediately.
Args:
request: FastAPI request
create_request: Migration configuration
Returns:
CreateMigrationResponse with migration ID and status
# Delete Migration
Source: https://docs.mixpeek.com/docs/api-reference/namespace-migrations/delete-migration
delete /v1/namespaces/migrations/{migration_id}
Delete a migration record.
Only draft, completed, failed, or cancelled migrations can be deleted.
Args:
request: FastAPI request
migration_id: Migration ID
# Get Migration
Source: https://docs.mixpeek.com/docs/api-reference/namespace-migrations/get-migration
get /v1/namespaces/migrations/{migration_id}
Get migration details and status.
Args:
request: FastAPI request
migration_id: Migration ID
Returns:
GetMigrationResponse with full migration details
# List Migrations
Source: https://docs.mixpeek.com/docs/api-reference/namespace-migrations/list-migrations
post /v1/namespaces/migrations/list
List migrations with optional filters.
Args:
request: FastAPI request
list_request: Filter and pagination parameters
Returns:
ListMigrationsResponse with migrations list
# Start Migration
Source: https://docs.mixpeek.com/docs/api-reference/namespace-migrations/start-migration
post /v1/namespaces/migrations/{migration_id}/start
Start a migration execution.
Args:
request: FastAPI request
migration_id: Migration ID
start_request: Start options
Returns:
StartMigrationResponse with task ID
# Validate Migration
Source: https://docs.mixpeek.com/docs/api-reference/namespace-migrations/validate-migration
post /v1/namespaces/migrations/validate
Validate a migration configuration without creating it.
Use this endpoint to check if a migration configuration is valid
before actually creating and running it.
Args:
request: FastAPI request
validate_request: Configuration to validate
Returns:
ValidateMigrationResponse with validation results
# Create Namespace
Source: https://docs.mixpeek.com/docs/api-reference/namespaces/create-namespace
post /v1/namespaces
Creates a new namespace with specified feature extractors and payload indexes.
# Delete Namespace
Source: https://docs.mixpeek.com/docs/api-reference/namespaces/delete-namespace
delete /v1/namespaces/{namespace_identifier}
This endpoint deletes a namespace and ALL its resources including:
- All buckets (with S3 objects, batches, uploads)
- All collections (with Qdrant points, cache, webhooks)
- All clusters (with Ray jobs, executions, triggers, S3 artifacts)
- All retrievers (with executions, evaluations, interactions, cache)
- Remaining MongoDB collections (tasks, uploads, taxonomies, API keys, etc.)
- All S3 objects with namespace prefix
- Qdrant collection (namespace's vector database)
- All namespace cache (across all scopes)
- Analytics data (ClickHouse tables)
- Namespace metadata
The deletion is performed asynchronously. Returns a task_id that can be used
to poll for deletion progress via GET /v1/tasks/{task_id}.
⚠️ WARNING: This operation is irreversible and will delete ALL data in the namespace!
# Get Namespace
Source: https://docs.mixpeek.com/docs/api-reference/namespaces/get-namespace
get /v1/namespaces/{namespace_identifier}
Retrieve details of a specific namespace using either its name or ID
# Get Namespace Stats
Source: https://docs.mixpeek.com/docs/api-reference/namespaces/get-namespace-stats
get /v1/namespaces/{namespace_identifier}/stats
Returns document, bucket, collection, and object counts for a single namespace. Use this instead of relying on counts in the list endpoint.
# List Namespaces
Source: https://docs.mixpeek.com/docs/api-reference/namespaces/list-namespaces
post /v1/namespaces/list
List all namespaces for a user
# Partially Update Namespace
Source: https://docs.mixpeek.com/docs/api-reference/namespaces/partially-update-namespace
patch /v1/namespaces/{namespace_identifier}
Partially updates an existing namespace (PATCH operation)
# Update Namespace
Source: https://docs.mixpeek.com/docs/api-reference/namespaces/update-namespace
put /v1/namespaces/{namespace_identifier}
Fully updates an existing namespace (all fields required)
# Delete Api Key
Source: https://docs.mixpeek.com/docs/api-reference/organization-api-keys/delete-api-key
delete /v1/organizations/users/{user_email}/api-keys/{key_name}
Revoke an API key.
🔒 The "admin-key" is protected and cannot be deleted.
Refuses when a live published app still embeds this key, naming the app
(MF-413). A 2026-06-03 revocation killed 2 published apps for seven weeks
because nothing in this path looked for references and the call reported
success.
# Rotate Api Key
Source: https://docs.mixpeek.com/docs/api-reference/organization-api-keys/rotate-api-key
post /v1/organizations/users/{user_email}/api-keys/{key_name}/rotate
Rotate an API key and return the new secret.
🔒 The "admin-key" is protected and cannot be rotated.
# Get Audit Log
Source: https://docs.mixpeek.com/docs/api-reference/organization-audit/get-audit-log
get /v1/organizations/audit/logs/{audit_id}
Get a specific audit log entry by ID.
Requires ADMIN permission.
# Get Audit Settings
Source: https://docs.mixpeek.com/docs/api-reference/organization-audit/get-audit-settings
get /v1/organizations/audit/settings
Get current audit configuration for the organization.
Returns the audit settings including whether read auditing is enabled.
Requires ADMIN permission.
# List Audit Logs
Source: https://docs.mixpeek.com/docs/api-reference/organization-audit/list-audit-logs
get /v1/organizations/audit/logs
List organization audit logs with filtering and pagination.
Returns audit events for the organization, sorted by timestamp descending.
Requires ADMIN permission.
# Update Audit Settings
Source: https://docs.mixpeek.com/docs/api-reference/organization-audit/update-audit-settings
patch /v1/organizations/audit/settings
Update audit configuration for the organization.
Use this to enable or disable read auditing.
Requires ADMIN permission.
# Confirm Payment Method
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/confirm-payment-method
post /v1/organizations/billing/confirm-payment-method
Confirm payment method after frontend collects it.
After Stripe Elements confirms the SetupIntent, call this endpoint
to attach the payment method to the customer and enable auto-billing.
**Requirements:**
- Admin permission
- Must have called setup-payment-method first
**Example:**
```python
# After Stripe Elements confirms setup
response = await client.post(
"/v1/organizations/billing/confirm-payment-method",
json={"payment_method_id": "pm_1ABC2DEF3GHI"}
)
```
# Disable Auto Billing
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/disable-auto-billing
post /v1/organizations/billing/disable-auto-billing
Disable automatic monthly billing.
Disables automatic billing but keeps payment method saved.
Organization can re-enable later or pay invoices manually.
**Requirements:**
- Admin permission
**Example:**
```python
response = await client.post("/v1/organizations/billing/disable-auto-billing")
```
# Enable Auto Billing
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/enable-auto-billing
post /v1/organizations/billing/enable-auto-billing
Enable automatic monthly billing.
Re-enables automatic billing if it was previously disabled.
Payment method must already be saved.
**Requirements:**
- Admin permission
- Must have payment method saved
**Example:**
```python
response = await client.post("/v1/organizations/billing/enable-auto-billing")
```
# Generate Invoice Now
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/generate-invoice-now
post /v1/organizations/billing/generate-invoice
Manually trigger invoice generation for this organization.
Useful for testing the billing flow or generating retroactive invoices.
Requires ADMIN permission.
Line items follow the v2 (modality+features, D8) composition: flat monthly
fee + per-modality feature usage in natural units and dollars + the plan's
included-usage pool as a deduction; only usage beyond the pool is due. The
total always reconciles to the month's ledger charges (see `pricing_v2`).
**Example:**
```python
# Dry run
response = await client.post(
"/v1/organizations/billing/generate-invoice",
json={"dry_run": true}
)
# Actually generate
response = await client.post(
"/v1/organizations/billing/generate-invoice",
json={"billing_month": "2026-01"}
)
```
# Get Credit Balance
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/get-credit-balance
get /v1/organizations/billing/balance
Get current credit balance and tier information.
Returns the organization's credit balance, current tier, and usage statistics.
Useful for displaying billing status in dashboards.
**Requirements:**
- Read permission
**Example:**
```python
response = await client.get("/v1/organizations/billing/balance")
print(f"Balance: {response['credit_balance']} credits")
print(f"Tier: {response['account_tier']}")
```
# Get Current Usage
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/get-current-usage
get /v1/organizations/billing/usage/current
Get current month usage.
Returns credit consumption for the current billing period,
estimated cost, and next invoice date.
**Requirements:**
- Read permission
**Example:**
```python
response = await client.get("/v1/organizations/billing/usage/current")
print(f"Usage: {response['current_month_usage']} credits")
print(f"Estimated cost: ${response['estimated_cost_usd']}")
```
# Get Payment Method
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/get-payment-method
get /v1/organizations/billing/payment-method
Get current payment method.
Returns the saved payment method details (last 4 digits, brand)
and auto-billing status.
**Requirements:**
- Read permission
**Example:**
```python
response = await client.get("/v1/organizations/billing/payment-method")
if response["has_payment_method"]:
print(f"Card: {response['payment_method']['card_brand']} ****{response['payment_method']['card_last4']}")
```
# Get Spending Caps
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/get-spending-caps
get /v1/organizations/billing/spending-caps
Get current spending cap configuration.
Returns spending cap settings including budget limits, alert thresholds,
and current spending status.
**Requirements:**
- Read permission
**Example:**
```python
response = await client.get("/v1/organizations/billing/spending-caps")
print(f"Monthly budget: ${response['monthly_spending_budget_usd']}")
print(f"Hard cap enabled: {response['hard_cap_enabled']}")
print(f"Current spending: ${response['current_spending_usd']}")
```
# Get Usage Breakdown
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/get-usage-breakdown
get /v1/organizations/billing/usage/breakdown
Get detailed usage breakdown.
Returns usage breakdown by operation type and extractor
for the specified billing period.
**Query Parameters:**
- `billing_month`: Month to query (YYYY-MM format, defaults to current)
**Requirements:**
- Read permission
**Example:**
```python
# Current month
response = await client.get("/v1/organizations/billing/usage/breakdown")
# Specific month
response = await client.get(
"/v1/organizations/billing/usage/breakdown",
params={"billing_month": "2025-11"}
)
```
# Get Vector Backend Usage
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/get-vector-backend-usage
get /v1/organizations/billing/usage/vector-backend
Get vector backend usage breakdown.
Returns usage dimensions (vectors stored, storage, queries, writes) and
cost breakdown for the tenant's active vector backend. Works for all
backends: qdrant, mvs, and dual_write.
**Requirements:**
- Read permission
# List Invoices
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/list-invoices
get /v1/organizations/billing/invoices
List monthly invoices.
Returns paginated list of monthly invoices with links to
Stripe-hosted invoice pages.
**Query Parameters:**
- `limit`: Number of invoices (1-100, default 10)
**Requirements:**
- Read permission
**Example:**
```python
response = await client.get("/v1/organizations/billing/invoices?limit=10")
for invoice in response["invoices"]:
print(f"{invoice['billing_month']}: ${invoice['amount_paid']/100}")
```
# Setup Payment Method
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/setup-payment-method
post /v1/organizations/billing/setup-payment-method
Initialize payment method setup flow.
Creates a Stripe SetupIntent for collecting payment method without charging.
The client_secret should be used with Stripe Elements on the frontend.
**Flow:**
1. Frontend calls this endpoint
2. Backend creates Stripe Customer (if needed) and SetupIntent
3. Frontend uses client_secret with Stripe Elements
4. User enters card details
5. Frontend calls confirm-payment-method endpoint
**Requirements:**
- Admin permission (only org admins can set up payment methods)
**Example:**
```python
response = await client.post("/v1/organizations/billing/setup-payment-method")
client_secret = response["client_secret"]
# Use client_secret with Stripe Elements
```
# Update Spending Caps
Source: https://docs.mixpeek.com/docs/api-reference/organization-billing/update-spending-caps
post /v1/organizations/billing/spending-caps
Update spending cap configuration.
Configure spending limits and alert thresholds to control costs.
**Features:**
- **Soft Limit (Budget)**: Triggers alerts but doesn't block API access
- **Hard Cap**: Blocks API access when reached. Setting a positive
`hard_spending_cap` arms enforcement automatically; pass
`hard_cap_enabled: false` in the same request to set the cap without
enforcing it. Clearing the cap (null/0) disarms enforcement.
- **Alert Thresholds**: Customize when to receive spending notifications
**Requirements:**
- Admin permission
- Only applies to organizations with auto-billing enabled
**Example:**
```python
# Set $100 budget with alerts at 75% and 100%
response = await client.post(
"/v1/organizations/billing/spending-caps",
json={
"monthly_spending_budget": 10000, # $100 in cents
"spending_alert_thresholds": [75, 100],
"spending_alerts_enabled": True,
}
)
# Enable hard cap at $500
response = await client.post(
"/v1/organizations/billing/spending-caps",
json={
"hard_spending_cap": 50000, # $500 in cents
"hard_cap_enabled": True,
}
)
# Disable all spending limits
response = await client.post(
"/v1/organizations/billing/spending-caps",
json={
"monthly_spending_budget": None,
"hard_spending_cap": None,
"hard_cap_enabled": False,
}
)
```
# Create Storage Connection
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/create-storage-connection
post /v1/organizations/connections
Create a new storage provider connection.
Establishes a connection to an external storage provider (Google Drive, S3, etc.)
for use in sync operations. Credentials are validated before saving unless
test_before_save is False.
**Use Cases:**
- Connect to team Google Drive for automated file ingestion
- Link customer S3 buckets for batch processing
- Set up storage connections for sync operations
**Security:**
- Requires ADMIN permission
- Credentials are encrypted at rest
- Connection is tested before saving (unless test_before_save=False)
- Audit log entry created for compliance
**Example:**
```bash
curl -X POST "http://localhost:8000/v1/organizations/connections" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Marketing Drive",
"provider_type": "google_drive",
"provider_config": {
"credentials": {...},
"shared_drive_id": "0AH-Xabc123"
}
}'
```
# Delete Storage Connection
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/delete-storage-connection
delete /v1/organizations/connections/{connection_identifier}
Soft-delete a connection (mark archived).
Permanently retires a connection by marking it as ARCHIVED. The connection
cannot be reactivated after deletion. Credentials are preserved for audit
purposes but the connection is no longer usable.
**Example:**
```bash
curl -X DELETE "http://localhost:8000/v1/organizations/connections/conn_abc123" \
-H "Authorization: Bearer YOUR_API_KEY"
```
# Get Storage Connection
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/get-storage-connection
get /v1/organizations/connections/{connection_identifier}
Retrieve a storage connection by ID or name.
Returns connection metadata including name, provider type, status, and
health information. Credentials are automatically redacted from responses.
**Identifier Resolution:**
- If identifier starts with 'conn_', treated as connection ID
- Otherwise, treated as connection name
**Example:**
```bash
# By ID
curl -X GET "http://localhost:8000/v1/organizations/connections/conn_abc123" \
-H "Authorization: Bearer YOUR_API_KEY"
# By name
curl -X GET "http://localhost:8000/v1/organizations/connections/Marketing%20Drive" \
-H "Authorization: Bearer YOUR_API_KEY"
```
# List Google Drive Files
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/list-google-drive-files
get /v1/organizations/connections/{connection_identifier}/files
List files in Google Drive folder for preview.
Shows a preview of files in the selected folder when configuring sync operations.
Only available for Google Drive connections.
**Use Cases:**
- Preview files in a folder before selecting it for sync
- Verify folder contains expected files
- Check file types and counts
**Example:**
```bash
curl -X GET "http://localhost:8000/v1/organizations/connections/conn_abc123/files?path=/Marketing&max_results=20" \
-H "Authorization: Bearer YOUR_API_KEY"
```
# List Google Drive Folders
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/list-google-drive-folders
get /v1/organizations/connections/{connection_identifier}/folders
List folders in Google Drive for folder selection in sync configuration.
Enables users to browse and select folders when configuring sync operations.
Only available for Google Drive connections.
**Use Cases:**
- Browse available folders for sync configuration
- Select source folder for bucket sync
- Navigate nested folder structures
**Example:**
```bash
curl -X GET "http://localhost:8000/v1/organizations/connections/conn_abc123/folders?path=/Marketing" \
-H "Authorization: Bearer YOUR_API_KEY"
```
# List Storage Connections
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/list-storage-connections
post /v1/organizations/connections/list
List storage connections for the authenticated organization.
Returns paginated results with optional filters for provider type, status,
and active flag. Results are sorted by creation date (newest first).
**Use Cases:**
- List all active Google Drive connections
- Find failed connections that need attention
- Filter by provider type for sync configuration
**Example:**
```bash
curl -X POST "http://localhost:8000/v1/organizations/connections/list" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"provider_type": "google_drive",
"is_active": true
}'
```
# Test Storage Connection
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/test-storage-connection
post /v1/organizations/connections/{connection_identifier}/test
Perform a credential test against the external provider.
Validates that connection credentials are still valid and the provider
is accessible. Result is logged in audit trail.
**Use Cases:**
- Validate credentials before using in sync operations
- Diagnose connection issues
- Refresh credentials after expiration
**Example:**
```bash
curl -X POST "http://localhost:8000/v1/organizations/connections/conn_abc123/test" \
-H "Authorization: Bearer YOUR_API_KEY"
```
# Update Storage Connection
Source: https://docs.mixpeek.com/docs/api-reference/organization-connections/update-storage-connection
patch /v1/organizations/connections/{connection_identifier}
Update connection metadata or credentials.
Allows partial updates to connection metadata without changing credentials.
Credentials can be updated via provider_config. All changes are logged
in audit trail.
**What You Can Update:**
- Connection name and description
- Metadata tags
- Status (active/suspended)
- Provider credentials (via provider_config)
**Example:**
```bash
curl -X PATCH "http://localhost:8000/v1/organizations/connections/conn_abc123" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Updated Drive Name",
"status": "suspended"
}'
```
# Create Secret
Source: https://docs.mixpeek.com/docs/api-reference/organization-secrets/create-secret
post /v1/organizations/secrets
Create a new secret in organization vault.
**Security**:
- Secret value is encrypted at rest using Fernet encryption
- Encrypted using ENCRYPTION_KEY from environment
- Decrypted value is NEVER returned in API responses
- Only secret names are exposed in list operations
**Use Cases**:
- Store API keys for external services (Stripe, GitHub, etc.)
- Store authentication tokens for api_call retriever stage
- Store credentials for third-party integrations
**Important**:
- Secret names must be unique within organization
- Use update endpoint to modify existing secrets
- Delete and recreate if you forget the value
# Delete Secret
Source: https://docs.mixpeek.com/docs/api-reference/organization-secrets/delete-secret
delete /v1/organizations/secrets/{secret_name}
Delete a secret from organization vault.
**Warning**:
- Deletion is permanent and immediate
- Any api_call stages using this secret will fail
- No confirmation prompt - use with caution
**Use Cases**:
- Remove unused credentials
- Clean up after service decommissioning
- Security incident response
# List Secrets
Source: https://docs.mixpeek.com/docs/api-reference/organization-secrets/list-secrets
get /v1/organizations/secrets
List all secret names in organization vault.
**Security**:
- Returns ONLY secret names, never values
- Use for discovering which secrets are configured
- Secret values can only be retrieved by internal services
**Response**:
- List of secret names (e.g., ['stripe_api_key', 'github_token'])
- Total count of secrets
# Update Secret
Source: https://docs.mixpeek.com/docs/api-reference/organization-secrets/update-secret
put /v1/organizations/secrets/{secret_name}
Update an existing secret in organization vault.
**Security**:
- Replaces existing encrypted value with new encrypted value
- Old value is permanently overwritten
- No history or audit trail of previous values
**Use Cases**:
- Rotate API keys periodically
- Update expired tokens
- Change credentials after security incident
# Get Api Key Endpoint Breakdown
Source: https://docs.mixpeek.com/docs/api-reference/organization-usage/get-api-key-endpoint-breakdown
get /v1/organizations/api-keys/{key_id}/usage/endpoints
Return endpoint-level usage metrics for a specific API key.
# Get Api Key Usage
Source: https://docs.mixpeek.com/docs/api-reference/organization-usage/get-api-key-usage
get /v1/organizations/api-keys/{key_id}/usage
Return usage metrics for a specific API key.
# Get Org Usage
Source: https://docs.mixpeek.com/docs/api-reference/organization-usage/get-org-usage
get /v1/organizations/usage
Return aggregated usage for the organization.
# Execute Adhoc Retriever
Source: https://docs.mixpeek.com/docs/api-reference/adhoc-retrievers/execute-adhoc-retriever
post /v1/retrievers/execute
Execute a retriever ad-hoc without persisting the configuration.
This endpoint allows you to execute a retriever without saving it to the database.
Useful for one-time queries, testing configurations, or temporary searches.
Streaming Execution (stream=True):
Response uses Server-Sent Events (SSE) format with Content-Type: text/event-stream.
Each stage emits events as it executes, formatted as: data: {json}\n\n
Event Types (StreamEventType):
- stage_start: Emitted when a stage begins (includes stage_name, stage_index, total_stages)
- stage_complete: Emitted when a stage finishes (includes documents, statistics, budget_used)
- stage_error: Emitted if a stage fails (includes error message)
- execution_complete: Final event with complete results and pagination
- execution_error: Emitted if entire execution fails
StreamStageEvent Fields:
- event_type: Type of event
- execution_id: Unique execution identifier
- stage_name/stage_index/total_stages: Stage progress info
- documents: Intermediate results (stage_complete only)
- statistics: Stage metrics (duration_ms, input_count, output_count, efficiency)
- budget_used: Cumulative consumption (credits_used, time_elapsed_ms, tokens_used)
Response Headers:
- Content-Type: text/event-stream
- Cache-Control: no-cache
- Connection: keep-alive
- X-Execution-Mode: adhoc
Standard Execution (stream=False, default):
- Returns ExecuteRetrieverResponse after all stages complete
- Includes X-Execution-Mode: adhoc header
- execution_metadata.retriever_persisted = False
Use Cases:
- One-time queries without saving retriever configuration
- Testing stage configurations before persisting
- Dynamic retrieval with varying parameters
- Real-time progress tracking with streaming
# Get Adhoc Execution
Source: https://docs.mixpeek.com/docs/api-reference/adhoc-retrievers/get-adhoc-execution
get /v1/retrievers/executions/{execution_id}
Get detailed execution information for a specific ad-hoc retriever execution.
Returns comprehensive execution details including:
- Execution metadata (status, duration, credits used)
- Performance metrics (documents processed/returned, cache hit rate)
- Input data and query summary
- Stage completion information
- Collections queried
Use Cases:
- Debug specific ad-hoc executions
- Analyze performance of a particular query
- Retrieve execution inputs for reproduction
- Audit ad-hoc retriever usage
Raises:
404 NotFoundError: If execution not found or not an ad-hoc execution
# List Adhoc Executions
Source: https://docs.mixpeek.com/docs/api-reference/adhoc-retrievers/list-adhoc-executions
post /v1/retrievers/executions/list
List execution history for ad-hoc retrievers.
Returns execution history for all ad-hoc retriever executions in the namespace,
sorted by timestamp descending (most recent first).
Use Cases:
- Track ad-hoc retriever usage across the namespace
- Debug ad-hoc retriever executions
- Analyze query patterns from ad-hoc searches
- Monitor performance of ad-hoc executions
Filtering:
- Filter by status (completed, failed, etc.)
- Filter by time range (start_time, end_time)
Pagination:
- Supports offset-based pagination via query parameters
- Default limit: 20, max limit: 100
- Use ?page_size=X&page_number=Y for pagination
# Batch Queue Status
Source: https://docs.mixpeek.com/docs/api-reference/batches/batch-queue-status
get /v1/batches/queue
View the Ray job queue: active jobs, max concurrency, and waiting batches with priorities.
# Cancel Batch
Source: https://docs.mixpeek.com/docs/api-reference/batches/cancel-batch
post /v1/batches/{batch_id}/cancel
Cancel a submitted/processing batch by ID without requiring the bucket_id. Cancels all associated Ray and Celery jobs, releases queue slots, and marks the batch as CANCELLED.
# Get Batch by ID
Source: https://docs.mixpeek.com/docs/api-reference/batches/get-batch-by-id
get /v1/batches/{batch_id}
Retrieve a single batch by its ID without requiring the bucket_id. Returns full batch details including real-time progress (objects_processed, total_objects, items_per_second, eta_seconds) for in-flight batches.
# List All Batches
Source: https://docs.mixpeek.com/docs/api-reference/batches/list-all-batches
post /v1/batches/list
List batches across all buckets in the organization. Filter with `status`, `bucket_id`, `collection_id`, or `namespace_id` in the request body. NOTE: the X-Namespace header does NOT narrow this endpoint — it is organization-scoped by design. Pass `namespace_id` in the body instead.
# Cancel Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/cancel-batch
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/cancel
Cancel a submitted/processing batch: cancels Ray job via engine and marks task/batch as CANCELLED.
# Cancel Job
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/cancel-job
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/tiers/{tier_num}/jobs/{ray_job_id}/cancel
Cancel a single Ray job within a tier without affecting sibling jobs.
# Cancel Tier
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/cancel-tier
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/tiers/{tier_num}/cancel
Cancel a single tier within a batch without affecting other tiers. Cancels all Ray and Celery jobs for this tier and marks it CANCELED.
# Delete Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/delete-batch
delete /v1/buckets/{bucket_identifier}/batches/{batch_id}
Delete a batch by its ID.
# Delete Documents Produced by Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/delete-documents-produced-by-batch
delete /v1/buckets/{bucket_identifier}/batches/{batch_id}/documents
Delete all documents that were produced by this batch. Useful for cleaning up documents from batches that produced bad data (e.g. null vectors). Only works for terminal batches.
# Diagnose Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/diagnose-batch
get /v1/buckets/{bucket_identifier}/batches/{batch_id}/diagnose
Aggregate diagnostic information for a batch: status, timing, errors, infrastructure events, progress, failed documents, and recommendations.
# Get Batch Configuration
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/get-batch-configuration
get /v1/buckets/{bucket_identifier}/batches/{batch_id}
Retrieve batch configuration and historical data from MongoDB. Status is automatically synchronized from Task API. For real-time monitoring, use GET /v1/tasks/{task_id} (Redis, faster).
# Get batch health
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/get-batch-health
get /v1/buckets/{bucket_identifier}/batches/{batch_id}/health
# Get Failed Documents for Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/get-failed-documents-for-batch
get /v1/buckets/{bucket_identifier}/batches/{batch_id}/failed-documents
Retrieve failed documents for a batch, optionally filtered by tier or collection.
# Get Ray Job Logs for Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/get-ray-job-logs-for-batch
get /v1/buckets/{bucket_identifier}/batches/{batch_id}/logs
Retrieve Ray job submission logs for a batch's processing tiers. You can get logs for a specific tier or all tiers. User must have access to the batch to retrieve logs.
# Manual Self-Healing
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/heal-tier
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/tiers/{tier_num}/heal
Trigger a self-healing action on a tier: kill duplicate jobs, detect stuck jobs, sync extractor status, or check the circuit breaker.
# List Batches
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/list-batches
post /v1/buckets/{bucket_identifier}/batches/list
List batches with pagination and filtering options.
# Partially Update Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/partially-update-batch
patch /v1/buckets/{bucket_identifier}/batches/{batch_id}
This endpoint partially updates a batch (PATCH operation).
Only provided fields will be updated. At minimum, metadata can always be updated.
Immutable fields like batch_id and timestamps cannot be modified.
# Resubmit Failed Batch
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/resubmit-failed-batch
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/resubmit
Re-run a terminal (FAILED/CANCELED/COMPLETED/…) batch: resets it to DRAFT and auto-submits by default, so the verb matches the behavior. Pass auto_submit=false to only park it as DRAFT — it must then be submitted manually before the draft TTL expires.
# Retry Failed Documents
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/retry-failed-documents
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/retry
Retry failed documents in a batch with intelligent filtering by error type and tier.
# Retry Tier with Resource Overrides
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/retry-tier
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/tiers/{tier_num}/retry
Retry a failed or canceled tier, optionally with modified resource parameters. Useful for retrying OOM failures with more memory.
# Submit All Draft Batches
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/submit-all-draft-batches
post /v1/buckets/{bucket_identifier}/batches/submit-all
Submit all DRAFT batches in this bucket for processing in a single request. Bypasses per-request rate limiting. Each batch is submitted sequentially and subject to the same admission control as individual submit.
# Submit Batch for Processing
Source: https://docs.mixpeek.com/docs/api-reference/bucket-batches/submit-batch-for-processing
post /v1/buckets/{bucket_identifier}/batches/{batch_id}/submit
Submit a draft batch for asynchronous processing. The batch must be in 'DRAFT' status and contain objects.
# Aggregate Documents
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/aggregate-documents
post /v1/collections/{collection_identifier}/documents/aggregate
This endpoint performs aggregation operations on documents in a collection.
**Aggregation Framework**: Provides MongoDB-style aggregation operations:
- GROUP BY: Group documents by one or more fields
- Aggregations: COUNT, SUM, AVG, MIN, MAX, COUNT_DISTINCT, etc.
- Date Operations: Truncate or extract date parts for time-series analysis
- Filtering: Pre-aggregation filters (WHERE) and post-aggregation filters (HAVING)
- Sorting & Limiting: Control result ordering and size
**Use Cases**:
- Count documents by feature type or collection
- Calculate daily/monthly processing statistics
- Analyze feature distributions and confidence scores
- Generate reports with multiple metrics
**Note**: This endpoint works with both MongoDB and Qdrant using the same interface.
The system automatically selects the appropriate aggregation provider based on
the underlying metadata store.
# Batch Delete Documents
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/batch-delete-documents
delete /v1/collections/{collection_identifier}/documents/batch
Batch delete multiple documents by explicit IDs or filters.
Supports TWO modes:
1. Explicit IDs mode: Provide 'document_ids' array
- Deletes specific documents by ID
- Returns detailed per-document results
- Maximum 1000 documents per batch
2. Filter mode: Provide 'filters' to delete all matching documents
- Deletes ALL documents matching the filters
- Returns total count only
- Use with caution - can delete many documents
Key Features:
- Per-document success/failure reporting in explicit mode
- Validates documents exist in the specified collection
- Automatic document count update for the collection
- Efficient bulk deletion
Examples:
Explicit IDs mode:
```json
{
"document_ids": ["doc_123", "doc_456", "doc_789"]
}
```
Filter mode (logical AND/OR/NOT shape — NOT MVS-native must/key):
```json
{
"filters": {"AND": [{"field": "metadata.status", "operator": "eq", "value": "archived"}]}
}
```
# Batch Update Documents
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/batch-update-documents
post /v1/collections/{collection_identifier}/documents/batch
Batch update multiple documents by explicit IDs or filters.
Supports TWO modes:
1. Explicit IDs mode: Provide 'updates' array with document_id + update_data for each document
- Each document can have DIFFERENT update_data
- Returns detailed per-document results
2. Filter mode: Provide 'filters' + 'update_data' to update all matching documents
- All documents receive the SAME update_data
- Returns total count only
Key Features:
- Update any document field except vectors (metadata, internal_metadata, source_blobs, etc.)
- Maximum 1000 documents per batch in explicit mode
- Per-document success/failure reporting in explicit mode
- Validates documents exist in the specified collection
Examples:
Explicit IDs mode:
```json
{
"updates": [
{"document_id": "doc_123", "update_data": {"metadata": {"status": "processed"}}},
{"document_id": "doc_456", "update_data": {"metadata": {"status": "archived"}}}
]
}
```
Filter mode (logical AND/OR/NOT shape — NOT MVS-native must/key):
```json
{
"filters": {"AND": [{"field": "metadata.status", "operator": "eq", "value": "pending"}]},
"update_data": {"metadata": {"status": "processed"}}
}
```
# Bulk Update Documents
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/bulk-update-documents
patch /v1/collections/{collection_identifier}/documents/bulk
Bulk update documents matching filter conditions.
Partially updates all documents in the collection that match the provided filters.
If no filters are provided, updates all documents in the collection.
This endpoint applies the SAME update_data to ALL documents matching the filters.
For per-document updates with different values, use POST /batch endpoint instead.
# Create a document.
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/create-a-document
post /v1/collections/{collection_identifier}/documents
Create a document by ID.
# Delete a document by ID.
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/delete-a-document-by-id
delete /v1/collections/{collection_identifier}/documents/{document_id}
Delete a document by ID.
# Get a document by ID.
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/get-a-document-by-id
get /v1/collections/{collection_identifier}/documents/{document_id}
Get a document by ID.
# List documents.
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/list-documents
post /v1/collections/{collection_identifier}/documents/list
List documents with optional grouping support.
Supports two modes:
1. Regular listing: Returns flat list of documents with pagination
2. Grouped listing: When group_by is specified, returns documents grouped by field value
When using group_by:
- Requires a payload index on the specified field in Qdrant
- Pagination applies to groups, not individual documents
- Each group contains all documents sharing the same field value
# Patch Document
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/patch-document
patch /v1/collections/{collection_identifier}/documents/{document_id}
Partially update a document by ID (PATCH operation).
# Return distinct values for a single field.
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/return-distinct-values-for-a-single-field
post /v1/collections/{collection_identifier}/documents/distinct
Return the set of unique values for one document field across a collection, together with a per-value document count sorted desc. Thin wrapper over `/aggregate` for the common `SELECT DISTINCT <field>` case — use this when a UI needs a facet list (e.g. `brand_slug`) without loading every document. Null/missing values are excluded.
# Update a document's access control list (ACL).
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/update-a-documents-access-control-list-acl
patch /v1/collections/{collection_identifier}/documents/{document_id}/acl
Update a document's ACL (access control list).
Only the document owner or an org-scoped key can modify ACL.
User-scoped keys that are not the owner will receive a 403.
# Update Document
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/update-document
put /v1/collections/{collection_identifier}/documents/{document_id}
Update a document by ID.
# Validate document against collection schema
Source: https://docs.mixpeek.com/docs/api-reference/collection-documents/validate-document-schema
post /v1/collections/{collection_identifier}/documents/schema/validate
Dry-run validation: checks a document payload against the collection's document_schema without creating or modifying any data. Returns validation result with any violations found.
# Get Lifecycle Status
Source: https://docs.mixpeek.com/docs/api-reference/collection-lifecycle/get-lifecycle-status
get /v1/collections/{collection_identifier}/lifecycle
Get the storage lifecycle status of a collection including vector counts.
# Transition Lifecycle
Source: https://docs.mixpeek.com/docs/api-reference/collection-lifecycle/transition-lifecycle
patch /v1/collections/{collection_identifier}/lifecycle
Transition a collection's storage tier. 'cold' evicts from the vector store (vectors searchable via object storage). 'active' rehydrates back to the vector store. 'archived' permanently removes vectors.
# Sync Collection Schema
Source: https://docs.mixpeek.com/docs/api-reference/collection-schema/sync-collection-schema
post /v1/collections/{collection_id}/sync-schema
Sample documents from Qdrant and automatically discover new fields to add to the collection's output_schema.
This endpoint:
- Samples N documents from the collection (default: 1000)
- Discovers all fields present in actual documents
- Merges discovered fields into the collection's output_schema (additive only)
- Optionally cascades schema updates to downstream collections
- Respects debounce window (max once per 5 minutes, unless force=true)
The sync operation is additive only - it never removes or changes existing field types.
Use this endpoint to:
- Manually trigger schema discovery after data ingestion
- Force an immediate schema sync (bypassing debounce)
- Update schemas with new fields discovered in documents
# Apply Taxonomy to Existing Documents
Source: https://docs.mixpeek.com/docs/api-reference/collection-taxonomies/apply-taxonomy-to-existing-documents
post /v1/collections/{collection_identifier}/apply-taxonomy
Apply a taxonomy to all existing documents in a collection retroactively.
This endpoint triggers distributed Ray processing to enrich existing documents
with taxonomy data. Unlike automatic materialization (which happens during ingestion),
this endpoint allows you to:
1. **Backfill enrichment** for documents ingested before the taxonomy was created
2. **Re-apply taxonomy** after configuration changes
3. **Process specific subsets** using scroll_filters
⚙️ **Processing Details:**
- Uses Ray datasets with map_batches for parallel processing
- Scales horizontally across Ray cluster
- Non-blocking: Returns immediately with task_id
- Monitor progress via Tasks API
⚠️ **Prerequisites:**
- Taxonomy must exist and be valid
- Taxonomy must be in collection's taxonomy_applications list
- Collection must contain documents
📊 **Performance:**
- ~1000-5000 docs/second depending on cluster size
- Parallel processing across multiple Ray workers
- Batch size and parallelism configurable
🔍 **Use Cases:**
- Backfill: Apply new taxonomy to historical data
- Re-enrichment: Update after taxonomy changes
- Selective: Process filtered document subsets
See Collections API and Taxonomies API documentation for details.
# Clone Collection
Source: https://docs.mixpeek.com/docs/api-reference/collections/clone-collection
post /v1/collections/{collection_identifier}/clone
Clone a collection with optional modifications.
**Purpose:**
Creates a NEW collection (with new ID) based on an existing one. This is the
recommended way to iterate on collection designs when you need to modify core
configuration that PATCH doesn't allow (source, feature_extractor, field_passthrough).
**Clone vs PATCH vs Template:**
- **PATCH**: Update metadata only (enabled, metadata, taxonomy_applications)
- **Clone**: Copy and modify core configuration (source, feature_extractor)
- **Template**: Start from a pre-configured pattern (for new projects)
**Common Use Cases:**
- Change feature extractor configuration (model, parameters)
- Modify field_passthrough to include/exclude fields
- Switch to different source (bucket or collection)
- Test modifications before replacing production collection
- Create variants (e.g., different embedding models)
**How it works:**
1. Source collection is copied
2. You provide a new name (REQUIRED)
3. Optionally override any other fields
4. A new collection is created with a new ID
5. Original collection remains unchanged
# Create Collection
Source: https://docs.mixpeek.com/docs/api-reference/collections/create-collection
post /v1/collections
Create a new processing collection linked to a namespace.
A collection defines the feature extraction pipeline that runs when objects are
uploaded to a bucket. One feature extractor per collection.
**Custom plugin collections:**
When `feature_extractor_name` references a custom plugin, the collection's vector
indexes are read from the plugin's `manifest.py` `features` list. The `features`
entries **must** use these exact key names — wrong keys silently produce a collection
with no vector indexes and 0 documents will be written:
```json
{
"feature_type": "embedding",
"feature_name": "my_embedding",
"embedding_dim": 768,
"distance_metric": "cosine"
}
```
Common wrong keys (will be ignored): `type`, `name`, `dimensions`, `distance`.
**Vector schema:** Custom plugin vectors are automatically added to the namespace's
Qdrant collection the first time a batch is processed, so there is no need to recreate
the namespace when adding a plugin with new embedding types.
# Delete Collection
Source: https://docs.mixpeek.com/docs/api-reference/collections/delete-collection
delete /v1/collections/{collection_identifier}
This endpoint deletes a collection and all its resources including:
- Qdrant points (documents) with this collection_id
- Cache entries
- MongoDB collection metadata
Note: Collections are payload IDs within the namespace's Qdrant collection,
not separate Qdrant collections.
The deletion is performed synchronously and returns when complete.
# Describe collection features
Source: https://docs.mixpeek.com/docs/api-reference/collections/describe-collection-features
get /v1/collections/{collection_identifier}/features
List feature addresses and metadata available in this collection
# Export Collection
Source: https://docs.mixpeek.com/docs/api-reference/collections/export-collection
post /v1/collections/{collection_identifier}/export
Export collection documents to JSON, CSV, or Parquet format.
**Export Formats:**
- **JSON**: Line-delimited JSON (JSONL) format. Good for streaming.
- **CSV**: Comma-separated values. Best for spreadsheets.
- **PARQUET**: Columnar format (default). Best for data pipelines.
**Vector Export:**
Vectors are large and exported separately. When `include_vectors=True`,
a separate file is created for vectors with document_id mapping.
**Field Selection:**
Use `select_fields` to export only specific fields, reducing file size.
**Filtering:**
Apply filters to export a subset of documents.
**Response:**
Returns presigned download URLs valid for 1 hour.
**Limits:**
- Large exports may take time. Consider using `sample_size` for testing.
- Vector exports significantly increase processing time.
# Get Collection
Source: https://docs.mixpeek.com/docs/api-reference/collections/get-collection
get /v1/collections/{collection_identifier}
This endpoint allows you to retrieve a collection by ID or name.
# List Collections
Source: https://docs.mixpeek.com/docs/api-reference/collections/list-collections
post /v1/collections/list
This endpoint allows you to list collections.
# Trigger Collection Processing
Source: https://docs.mixpeek.com/docs/api-reference/collections/trigger-collection-processing
post /v1/collections/{collection_identifier}/trigger
Process data through a collection - works for both bucket-sourced and collection-sourced collections.
**For bucket-sourced collections:**
Discovers objects from source bucket(s), creates a batch, and submits for processing.
Use `include_buckets` to limit which source buckets to process from.
**For collection-sourced collections:**
Processes existing documents from upstream collection(s).
Use `include_collections` to limit which source collections to process from.
**Filtering:**
- `source_filters`: Field-level filters using LogicalOperator format
- Example: `{"AND": [{"field": "status", "operator": "eq", "value": "pending"}]}`
- For specific objects: `{"AND": [{"field": "object_id", "operator": "in", "value": ["obj_1", "obj_2"]}]}`
**Returns:**
- batch_id: Track progress via GET /batches/{batch_id}
- task_id: Monitor via GET /tasks/{task_id}
# Update Collection
Source: https://docs.mixpeek.com/docs/api-reference/collections/update-collection
patch /v1/collections/{collection_identifier}
Update mutable collection fields (collection_name, description, taxonomy_applications, enabled)
# Get all documents derived from an object
Source: https://docs.mixpeek.com/docs/api-reference/document-lineage/get-all-documents-derived-from-an-object
get /v1/objects/{object_id}/documents
Get all documents created from a specific root object. Useful for finding all processing outputs across multiple collections.
# Get decomposition tree visualization
Source: https://docs.mixpeek.com/docs/api-reference/document-lineage/get-decomposition-tree-visualization
get /v1/objects/{object_id}/decomposition-tree
Get a hierarchical tree structure showing all collections and documents derived from a root object. Shows the complete multi-stage processing pipeline.
# Get document lineage
Source: https://docs.mixpeek.com/docs/api-reference/document-lineage/get-document-lineage
get /v1/collections/{collection_id}/documents/{document_id}/lineage
Get the complete processing lineage for a document. Shows the full chain of transformations from the root bucket object through all collection processing stages.
# Batch get documents by IDs
Source: https://docs.mixpeek.com/docs/api-reference/documents/batch-get-documents-by-ids
post /v1/documents/batch-get
Batch retrieve multiple documents by their IDs.
Returns documents and a list of IDs that were not found.
Maximum 1000 document IDs per request.
# Get a document by ID (namespace-scoped).
Source: https://docs.mixpeek.com/docs/api-reference/documents/get-a-document-by-id-namespace-scoped
get /v1/documents/{document_id}
Get a document by ID without specifying a collection.
Searches across all collections in the namespace. Use the
collection-scoped GET /v1/collections/{collection_id}/documents/{document_id}
when you know the collection for a faster lookup.
# List documents across all collections (namespace-scoped).
Source: https://docs.mixpeek.com/docs/api-reference/documents/list-documents-across-all-collections-namespace-scoped
post /v1/documents/list
List documents across all collections in the namespace.
Use this when you don't know which collection a document belongs to,
or when you need to search across collections. Optionally filter to
specific collection_ids.
For collection-specific listing, use
POST /v1/collections/{collection_id}/documents/list instead.
# Partially update a document by ID (namespace-scoped).
Source: https://docs.mixpeek.com/docs/api-reference/documents/partially-update-a-document-by-id-namespace-scoped
patch /v1/documents/{document_id}
Partially update a document by ID without specifying a collection.
Only the fields provided are updated; all other fields are left unchanged.
# Run multiple list-documents queries concurrently.
Source: https://docs.mixpeek.com/docs/api-reference/documents/run-multiple-list-documents-queries-concurrently
post /v1/documents/list/batch
Execute up to 10 `POST /v1/documents/list` queries in parallel against the
same namespace and return their responses together.
Collapses dossier / multi-collection N+1 fan-outs into one round-trip. Each
sub-query is independent; a failure in one does not fail the batch — the
failing entry's `response` is null and `error` carries a short message.
# Update a document by ID (namespace-scoped).
Source: https://docs.mixpeek.com/docs/api-reference/documents/update-a-document-by-id-namespace-scoped
put /v1/documents/{document_id}
Replace a document's fields by ID without specifying a collection.
# Get Feature Extractor by Name
Source: https://docs.mixpeek.com/docs/api-reference/feature-extractors/get-feature-extractor-by-name
get /v1/collections/features/extractors/{feature_extractor_id}
Get detailed information about a specific feature extractor by its name
# List Feature Extractors
Source: https://docs.mixpeek.com/docs/api-reference/feature-extractors/list-feature-extractors
get /v1/collections/features/extractors
List all available feature extractors grouped by category
# List Public Retrievers
Source: https://docs.mixpeek.com/docs/api-reference/public-retriever-api/list-public-retrievers
get /v1/public/retrievers/
List all public retrievers with pagination and search.
This endpoint allows browsing and discovering all published retrievers
across all organizations. No authentication required.
**Authentication:**
- NO authentication required - completely public endpoint
- Discover retrievers created by all Mixpeek users
**Pagination:**
- Default: page=1, page_size=20
- Maximum page_size: 100
- Returns total count and total pages
**Search:**
- Search across retriever titles and descriptions
- Case-insensitive regex matching
- Combine with pagination
**Filtering:**
- By default, only active retrievers are shown
- Set `include_inactive=true` to see all retrievers
**Response includes:**
- List of public retrievers with basic info
- Pagination details (page, page_size, total_count, total_pages)
- Aggregate statistics (total active, password protected, open)
**What's NOT exposed:**
- API keys (except in individual config endpoint)
- Internal IDs or organization details
- Full retriever configuration (use template endpoint for that)
- Password values (only password_protected: true/false)
**Example:**
```bash
# List all public retrievers (first page)
curl -X GET "https://api.mixpeek.com/v1/public/retrievers/"
# Search for video-related retrievers
curl -X GET "https://api.mixpeek.com/v1/public/retrievers/?search=video&page_size=50"
# Get page 2 with custom page size
curl -X GET "https://api.mixpeek.com/v1/public/retrievers/?page=2&page_size=50"
```
**Use Cases:**
- Browse available public retrievers
- Discover search patterns and implementations
- Find retrievers to use as templates
- Explore what others have built
# Check Name Availability
Source: https://docs.mixpeek.com/docs/api-reference/published-retrievers/check-name-availability
get /v1/retrievers/{retriever_id}/publish/availability
Check if a public name is available.
Public names must be globally unique across all organizations.
Use this endpoint before publishing to ensure your desired name is available.
# Get Organization Publish Stats
Source: https://docs.mixpeek.com/docs/api-reference/published-retrievers/get-organization-publish-stats
get /v1/retrievers/publish/stats
Get organization publish statistics.
# Get Published Retriever
Source: https://docs.mixpeek.com/docs/api-reference/published-retrievers/get-published-retriever
get /v1/retrievers/{retriever_id}/publish
Get published retriever details.
Returns configuration including public URL, display settings, and rate limits.
The public API key is not returned (it was shown only once during publishing).
A retriever that simply hasn't been published yet is a normal state (the
Studio Publish tab opens this on every visit), not an error — so this
returns ``200`` with a ``null`` body rather than ``404``, which previously
logged a console error for every unpublished retriever.
# List Published Retrievers
Source: https://docs.mixpeek.com/docs/api-reference/published-retrievers/list-published-retrievers
get /v1/retrievers/published
List all published retrievers for the organization.
# Publish Retriever
Source: https://docs.mixpeek.com/docs/api-reference/published-retrievers/publish-retriever
post /v1/retrievers/{retriever_id}/publish
Publish a retriever as a public search interface.
Creates a public API endpoint and branded page for the retriever.
Returns a public API key that should be stored securely (shown only once).
**Limits:**
- Maximum 10 published retrievers per organization
- Public name must be globally unique
**Security:**
- Public API key is required for all requests
- Optional password protection via organization secrets
- Configurable rate limits per retriever
- Field masking ensures only approved fields are exposed
# Unpublish Retriever
Source: https://docs.mixpeek.com/docs/api-reference/published-retrievers/unpublish-retriever
delete /v1/retrievers/{retriever_id}/publish
Unpublish a retriever.
Removes the public API endpoint and deletes the short URL redirect.
The public API key will immediately stop working.
This action cannot be undone (you'll need to republish to get a new API key).
# Update Published Retriever
Source: https://docs.mixpeek.com/docs/api-reference/published-retrievers/update-published-retriever
patch /v1/retrievers/{retriever_id}/publish
Update published retriever configuration.
Allows updating display config, rate limits, exposed fields, and password protection.
The public API key and public name cannot be changed (unpublish and republish instead).
# Create Retriever API Key
Source: https://docs.mixpeek.com/docs/api-reference/retriever-api-keys/create-retriever-api-key
post /v1/retrievers/{retriever_id}/api-keys
Generate a scoped API key for executing this specific retriever.
**Use Cases:**
- Provide external services with execution-only access
- Embed retriever calls in customer applications
- Create separate keys for staging vs production
- Implement per-customer access keys for SaaS products
**Security:**
- Keys grant EXECUTE_RETRIEVER permission only
- Keys are scoped to single retriever (cannot access others)
- Keys inherit org's rate limits
- Keys can be revoked instantly
- Key prefix (ret_sk_abc...) shown in UI for identification
**Ownership:**
- Only the organization that owns the retriever can create keys
- Verified by matching internal_id + namespace_id
**Key Format:**
- Prefix: ret_sk_
- Length: 60 characters
- Example: ret_sk_abcdefghijklmnopqrstuvwxyz123456789...
**Response:**
- Plaintext key shown ONLY ONCE in response
- Save the key immediately - it cannot be retrieved later
- Key prefix stored for identification in UI
# List Retriever API Keys
Source: https://docs.mixpeek.com/docs/api-reference/retriever-api-keys/list-retriever-api-keys
get /v1/retrievers/{retriever_id}/api-keys
List all API keys for this retriever.
**Fields:**
- key_id: Public identifier for the key
- key_prefix: First 10 characters + "..." for identification (e.g., "ret_sk_abc...")
- name: Human-friendly label
- created_at: When the key was created
- last_used_at: When the key was last used (if ever)
- status: ACTIVE, REVOKED, or EXPIRED
- expires_at: Expiration timestamp (if set)
**Note:**
- Plaintext key is NEVER returned in list responses
- Only shown once in creation response
- Use key_prefix to identify keys in the UI
# Revoke Retriever API Key
Source: https://docs.mixpeek.com/docs/api-reference/retriever-api-keys/revoke-retriever-api-key
delete /v1/retrievers/{retriever_id}/api-keys/{key_id}
Revoke a retriever-scoped API key.
**Effect:**
- Key status is set to REVOKED
- Key can no longer be used for authentication
- Revocation is immediate (no grace period)
- Auth cache is invalidated immediately
- Cannot be undone (create a new key if needed)
**Audit:**
- Revocation is logged in the retriever's audit trail
- Includes actor user ID and timestamp
# Create benchmark
Source: https://docs.mixpeek.com/docs/api-reference/retriever-benchmarks/create-benchmark
post /v1/retrievers/benchmarks
Create a new benchmark run to compare retriever pipelines. The benchmark will replay historical sessions and measure alignment with observed user behavior.
# Delete benchmark
Source: https://docs.mixpeek.com/docs/api-reference/retriever-benchmarks/delete-benchmark
delete /v1/retrievers/benchmarks/{benchmark_id}
Delete a benchmark and its results
# Get benchmark
Source: https://docs.mixpeek.com/docs/api-reference/retriever-benchmarks/get-benchmark
get /v1/retrievers/benchmarks/{benchmark_id}
Get benchmark status and results by ID
# List benchmarks
Source: https://docs.mixpeek.com/docs/api-reference/retriever-benchmarks/list-benchmarks
get /v1/retrievers/benchmarks
List benchmarks with optional filtering
# Create evaluation dataset
Source: https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/create-evaluation-dataset
post /v1/retrievers/evaluations/datasets
Create a ground truth dataset for evaluating retrievers. Include queries with their relevant documents and optional graded relevance scores.
# Get evaluation dataset
Source: https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/get-evaluation-dataset
get /v1/retrievers/evaluations/datasets/{dataset_identifier}
Retrieve a specific dataset by ID or name
# Get evaluation results
Source: https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/get-evaluation-results
get /v1/retrievers/{retriever_id}/evaluations/{evaluation_id}
Retrieve evaluation results with all calculated metrics
# List evaluation datasets
Source: https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/list-evaluation-datasets
get /v1/retrievers/evaluations/datasets
List all evaluation datasets with pagination
# List evaluations
Source: https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/list-evaluations
get /v1/retrievers/{retriever_id}/evaluations
List all evaluations for a retriever with optional filters
# Run evaluation
Source: https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/run-evaluation
post /v1/retrievers/{retriever_id}/evaluations
Evaluate a retriever's quality using a ground truth dataset. Returns immediately with a task ID - evaluation runs asynchronously.
# Create Interaction
Source: https://docs.mixpeek.com/docs/api-reference/retriever-interactions/create-interaction
post /v1/retrievers/interactions
Record a search interaction (view, click, feedback, etc.).
Automatically computes and injects ``metadata.reward_value`` based on
the interaction types and the retriever's reward_map configuration.
This pre-computed value is used by the learned-fusion bandit aggregation
query so it doesn't need to join against the retriever config at read time.
**Deduplication (Phase 4e):** If ``(execution_id, feature_id,
interaction_type)`` has already been recorded within the last 5 minutes
the duplicate is silently dropped.
**Session cache (Phase 1b):** When ``session_id`` is present the
interaction is also written to the Redis session cache so the Thompson
Sampling bandit can incorporate it in real time (before ClickHouse
write-then-read latency settles).
# Delete Interaction
Source: https://docs.mixpeek.com/docs/api-reference/retriever-interactions/delete-interaction
delete /v1/retrievers/interactions/{interaction_id}
Delete a specific interaction (idempotent - succeeds even if already deleted).
# Get Interaction
Source: https://docs.mixpeek.com/docs/api-reference/retriever-interactions/get-interaction
get /v1/retrievers/interactions/{interaction_id}
Get a specific interaction.
# List Interactions
Source: https://docs.mixpeek.com/docs/api-reference/retriever-interactions/list-interactions
post /v1/retrievers/interactions/list
List interactions with optional filters and pagination.
Supports hybrid filtering: simple fields + advanced LogicalOperator.
# List Available Retriever Stages
Source: https://docs.mixpeek.com/docs/api-reference/retriever-stages/list-available-retriever-stages
get /v1/retrievers/stages
List all registered retriever stages with their configurations. Use this endpoint to discover available stages before creating retrievers. Each stage includes its ID, description, category, and full parameter schema. The parameter_schema field contains complete Pydantic JSON Schema with validation rules, descriptions, and examples for all stage parameters.
# Batch Execute Retriever
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/batch-execute-retriever
post /v1/retrievers/{retriever_id}/execute/batch
Execute a retriever against multiple queries in a single request. The retriever is fetched and optimized once, then executed concurrently against each query with bounded parallelism.
**Use case:** IP safety / copyright clearance — scan 20 media files against face, logo, or audio retrievers in one call instead of 60 sequential SSE requests.
**Limits:** 1-50 queries per batch, 1-20 concurrency.
Returns results keyed by query index with per-query documents and errors.
# Clone Retriever
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/clone-retriever
post /v1/retrievers/{retriever_id}/clone
Clone a retriever with optional modifications.
**Purpose:**
Creates a NEW retriever (with new ID) based on an existing one. This is the
recommended way to iterate on retriever designs when you need to modify core
logic that PATCH doesn't allow (stages, input_schema, collections).
**Clone vs PATCH vs Template:**
- **PATCH**: Update metadata only (name, description, tags, display_config)
- **Clone**: Copy and modify core logic (stages, input_schema, collections)
- **Template**: Start from a pre-configured pattern (for new projects)
**Common Use Cases:**
- Fix a typo in a stage name
- Add or remove stages
- Change target collections
- Create variants (e.g., "strict" vs "relaxed" versions)
- Test modifications before replacing production retriever
**How it works:**
1. Source retriever is copied
2. You provide a new name (REQUIRED)
3. Optionally override any other fields
4. A new retriever is created with a new ID
5. Original retriever remains unchanged
**All fields except retriever_name are OPTIONAL:**
- Omit a field to copy from source
- Provide a field to override the source value
# Create Retriever
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/create-retriever
post /v1/retrievers
Create a new retriever.
A retriever executes a series of stages to find and process documents
from one or more collections.
# Delete Retriever
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/delete-retriever
delete /v1/retrievers/{retriever_id}
Delete a retriever and all its resources comprehensively.
Deletes:
- Published retrievers
- Execution history
- Interactions (user feedback)
- Evaluations
- Cache entries
- Retriever metadata
The deletion is performed synchronously and returns when complete.
# Execute Retriever (Auto-Optimized)
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/execute-retriever-auto-optimized
post /v1/retrievers/{retriever_id}/execute
Execute a retriever and return matching documents. The pipeline is automatically optimized before execution for best performance.
**Automatic Optimization:**
Your pipeline stages are automatically transformed for optimal performance:
- Filters pushed down to reduce expensive operations
- Redundant stages merged or eliminated
- Grouping operations pushed to database layer (10-100x faster)
- Operations reordered for efficiency
**Streaming Support:**
Set stream=true in the request body to receive real-time stage updates via SSE:
- Response uses text/event-stream content type
- Each stage emits stage_start and stage_complete events
- Final event contains complete results and pagination
- Useful for progress tracking and debugging
**Response Includes (when stream=false):**
- documents: Final matching documents
- pagination: Pagination metadata
- stage_statistics: Per-stage execution metrics
- budget: Credit/time consumption
- optimization_applied: Whether optimizations were applied
- optimization_summary: Details about transformations (when applied)
**Optimization Summary Example:**
```json
{
"optimization_applied": true,
"optimization_summary": {
"original_stage_count": 5,
"optimized_stage_count": 3,
"optimization_time_ms": 8.2,
"rules_applied": ["push_down_filters", "group_by_push_down"],
"stage_reduction_pct": 40.0
}
}
```
Use the /explain endpoint to see the optimized execution plan before running.
# Explain Retriever Execution Plan
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/explain-retriever-execution-plan
post /v1/retrievers/{retriever_id}/execute/explain
Get a detailed execution plan for a retriever without actually executing it. Similar to MongoDB's explain plan or SQL's EXPLAIN command, this endpoint helps you understand performance characteristics, identify bottlenecks, estimate costs, and troubleshoot retrieval issues before running expensive queries.
**What This Returns:**
- Stage-by-stage execution plan (AFTER automatic optimizations)
- Estimated costs (credits + time per stage)
- Document flow projections (input/output counts per stage)
- Efficiency metrics (selectivity ratios, cache likelihood)
- Bottleneck identification (slowest/most expensive stages)
- Optimization details (transformations applied by the optimizer)
- Performance warnings and improvement suggestions
**Key Features:**
- **Cost Estimation**: See how many credits and milliseconds each stage will consume
- **Bottleneck Detection**: Identify which stages dominate execution time
- **Optimization Transparency**: Understand how your pipeline was optimized
- **Cache Analysis**: See which stages are likely to hit cache
- **Accuracy Troubleshooting**: Analyze stage efficiency and document flow
- **Latency Analysis**: Break down estimated duration by stage
**Important:** The execution_plan shows OPTIMIZED stages (after automatic transformations like filter push-down, stage fusion, and grouping optimization). Check optimization_details to understand what changed from your original configuration.
**Use Cases:**
- Debug slow retrievers by identifying bottleneck stages
- Estimate costs before running expensive queries
- Understand how the optimizer transformed your pipeline
- Troubleshoot accuracy issues by analyzing stage selectivity
- Compare different retriever configurations
- Plan budget allocation for production workloads
**Example Response:**
```json
{
"retriever_id": "ret_abc123",
"retriever_name": "product_search",
"execution_plan": [
{
"stage_index": 0,
"stage_name": "attribute_filter",
"stage_type": "filter",
"estimated_input": 10000,
"estimated_output": 5000,
"estimated_efficiency": 0.5,
"estimated_cost_credits": 0.01,
"estimated_duration_ms": 20,
"cache_likely": true,
"optimization_notes": ["Pushed down from stage 2"],
"warnings": []
},
{
"stage_index": 1,
"stage_name": "semantic_search",
"stage_type": "filter",
"estimated_input": 5000,
"estimated_output": 100,
"estimated_efficiency": 0.02,
"estimated_cost_credits": 0.5,
"estimated_duration_ms": 200,
"cache_likely": false,
"optimization_notes": [],
"warnings": ["High cost stage - consider reducing limit"]
}
],
"estimated_cost": {
"total_credits": 0.51,
"total_duration_ms": 220
},
"bottleneck_stages": ["semantic_search"],
"optimization_applied": true,
"optimization_details": {
"original_stage_count": 3,
"optimized_stage_count": 2,
"optimization_time_ms": 8.2,
"stage_reduction_pct": 33.3,
"decisions": [
{
"rule_type": "push_down_filters",
"applied": true,
"reason": "Moved attribute_filter before semantic_search to reduce search scope"
}
]
},
"optimization_suggestions": [
{
"type": "reduce_limit",
"stage": "semantic_search",
"message": "Consider reducing limit to improve latency"
}
]
}
```
# Get Execution
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/get-execution
get /v1/retrievers/{retriever_id}/executions/{execution_id}
Get execution details and statistics.
# Get Live Execution Status
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/get-live-execution-status
get /v1/retrievers/{retriever_id}/executions/{execution_id}/status
Fetch live execution status from the durability journal. Returns real-time progress including current stage, completed stages, and heartbeat timestamp. If the execution is completed, returns the full results. Falls back to ClickHouse for historical executions.
# Get Retriever
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/get-retriever
get /v1/retrievers/{retriever_id}
Get a retriever by ID or name.
# List Executions
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/list-executions
post /v1/retrievers/{retriever_id}/executions/list
List execution history for a retriever.
# List Retrievers
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/list-retrievers
post /v1/retrievers/list
List all retrievers in the namespace.
# Patch Retriever
Source: https://docs.mixpeek.com/docs/api-reference/retrievers/patch-retriever
patch /v1/retrievers/{retriever_id}
Update a retriever's metadata.
Editable fields:
- name, description, tags, display_config: metadata
- collection_identifiers: re-points + re-validates stage feature URIs
- stages (BACKE-1287): edit stages in place on an UNPUBLISHED retriever
(change a filter/operator/rerank without clone+repoint+delete); the full
stage list is replaced + re-validated as on create. A PUBLISHED retriever's
stages stay immutable — clone or unpublish to change them.
input_schema and budget_limits remain immutable; use POST /{retriever_id}/clone.
# Create Alert
Source: https://docs.mixpeek.com/docs/api-reference/alerts/create-alert
post /v1/alerts
Create a new alert that monitors document ingestion and sends notifications.
Alerts attach retrievers to collections. When new documents are ingested,
the alert runs the retriever and sends notifications if matches are found.
**Key Components:**
- `retriever_id`: References a retriever that defines query logic (filters, scoring, limits)
- `notification_config`: Defines where to send notifications (webhook, Slack, email)
**Note:** The retriever owns all query semantics. The alert's job is simply
to run the retriever and notify if results exist.
# Delete Alert
Source: https://docs.mixpeek.com/docs/api-reference/alerts/delete-alert
delete /v1/alerts/{alert_identifier}
Delete an alert and its execution history.
This operation:
- Removes the alert from MongoDB
- Deletes all execution history for this alert
- Does NOT affect the referenced retriever
# Get Alert
Source: https://docs.mixpeek.com/docs/api-reference/alerts/get-alert
get /v1/alerts/{alert_identifier}
Get an alert by ID (alt_...) or name.
# Get Alert Execution
Source: https://docs.mixpeek.com/docs/api-reference/alerts/get-alert-execution
get /v1/alerts/executions/{execution_id}
Get a single alert execution result by execution ID. Use this to poll for alert results without needing a webhook.
# Get Latest Alert Result
Source: https://docs.mixpeek.com/docs/api-reference/alerts/get-latest-alert-result
get /v1/alerts/{alert_identifier}/results
Get the most recent execution result for an alert. Use this endpoint to poll for alert results without a webhook. Returns the latest execution with match details, or 404 if the alert has never executed.
# List Alert Executions
Source: https://docs.mixpeek.com/docs/api-reference/alerts/list-alert-executions
get /v1/alerts/{alert_identifier}/executions
List execution history for a specific alert.
# List Alerts
Source: https://docs.mixpeek.com/docs/api-reference/alerts/list-alerts
post /v1/alerts/list
List all alerts in the namespace with optional filtering and pagination.
**Filtering:**
- Use `search` for wildcard text search across alert_id, name, description
- Use `filters` for structured queries
**Sorting:**
- Default: created_at descending (newest first)
# List All Executions
Source: https://docs.mixpeek.com/docs/api-reference/alerts/list-all-executions
get /v1/alerts/executions
List execution history for all alerts in the namespace.
# Update Alert
Source: https://docs.mixpeek.com/docs/api-reference/alerts/update-alert
patch /v1/alerts/{alert_identifier}
Partially update an alert's configuration.
**All fields are optional** - provide only what you want to update.
Unlike taxonomies, alerts can be fully updated including:
- `name`: Rename the alert
- `description`: Update documentation
- `retriever_id`: Change the retriever
- `notification_config`: Update notification channels
- `enabled`: Enable/disable the alert
- `metadata`: Update custom metadata
# Add a custom domain
Source: https://docs.mixpeek.com/docs/api-reference/apps/add-domain
post /v1/apps/{app_id}/domains
Attach a custom domain to an App.
Returns a `verification_token` and `cname_target`. To prove ownership,
add a DNS TXT record:
_mixpeek-verify.{domain} TXT "mixpeek-site-verification={verification_token}"
Then call `POST /{app_id}/domains/{domain}/verify` to start polling.
# Connect a GitHub repository
Source: https://docs.mixpeek.com/docs/api-reference/apps/connect-a-github-repository
post /v1/apps/{app_id}/connect-repo
Connect a GitHub repository for automatic deploys on push.
After connecting, pushes to the configured branch will trigger
a build + deploy via the GitHub App webhook.
# Create an App
Source: https://docs.mixpeek.com/docs/api-reference/apps/create-app
post /v1/apps
Create a new App.
Apps are full-stack web applications deployed as zip bundles to
``{slug}.mxp.co``. After creation, deploy your built frontend via the
deploy pipeline:
1. ``POST /v1/apps/{app_id}/deploy/upload-url`` — get a presigned S3 PUT URL
2. Upload your built ``dist/`` zip to that URL
3. ``POST /v1/apps/{app_id}/deploy`` — trigger the build
Only ``slug`` and ``meta`` are required. Set ``auth_config.mode`` to
control end-user authentication (``public``, ``clerk``, ``password``,
``api_key``, ``jwt``, ``sso_oidc``, ``sso_saml``).
Slugs are globally unique across all organizations.
**Deprecated fields:** ``template``, ``sections``, ``custom_html``,
``hero``, ``theme``, ``seo``, ``stats``, ``featured_gallery``, ``tabs``,
``password_secret_name`` are accepted for backward compatibility but
should not be used for new apps.
**Tier limits:** Free = 1 app, Pro = 10, Enterprise = unlimited
(``TierLimits._max_apps``). Exceeding your tier's limit returns a 400 with
the limit + current count in ``error.details``.
# Delete an App
Source: https://docs.mixpeek.com/docs/api-reference/apps/delete-app
delete /v1/apps/{app_id}
Delete an App permanently.
The slug is freed immediately. Any active custom domains will stop
resolving within seconds.
# Remove a domain
Source: https://docs.mixpeek.com/docs/api-reference/apps/delete-domain
delete /v1/apps/{app_id}/domains/{domain}
Remove a custom domain from an App.
# Deploy an App
Source: https://docs.mixpeek.com/docs/api-reference/apps/deploy-app
post /v1/apps/{app_id}/deploy
Queue a build+deploy for an App, creating a new version.
**Full deploy workflow:**
1. `POST /{app_id}/deploy/upload-url` → get a presigned S3 PUT URL
2. Upload your built dist/ zip to that URL
3. Call this endpoint with the returned `bundle_s3_key`
4. Optionally include `source_files` (path→content map) for version diffing
5. Poll `GET /{app_id}/deploys/{deploy_id}` until status is `complete`
**Edit-and-redeploy workflow:**
1. `GET /{app_id}/versions/{version}/download` → download the bundle zip
2. Edit files locally
3. Re-zip and upload via upload-url → deploy (creates a new version)
4. `GET /{app_id}/versions/{old}/diff/{new}` to review changes
Each deploy creates an immutable version record. Use `POST /{app_id}/versions/{n}/restore`
for instant rollback to any previous version.
# Diff Two Versions
Source: https://docs.mixpeek.com/docs/api-reference/apps/diff-versions
GET /v1/apps/{app_id}/versions/{from_version}/diff/{to_version}
Compare two versions — returns file-level changes and optional source diffs.
Like `git diff v1..v2`: shows added, removed, modified, and unchanged files
based on content hashes. If both versions have source_files, also returns
unified diffs of the source code.
# Disconnect GitHub repository
Source: https://docs.mixpeek.com/docs/api-reference/apps/disconnect-github-repository
delete /v1/apps/{app_id}/connect-repo
Disconnect the GitHub repository from this app.
# Download Version Bundle
Source: https://docs.mixpeek.com/docs/api-reference/apps/download-version
GET /v1/apps/{app_id}/versions/{version}/download
Get a presigned download URL for a version's original bundle zip.
Use this to download → edit → re-upload → deploy as a new version.
The returned URL expires in 1 hour.
# Get an App
Source: https://docs.mixpeek.com/docs/api-reference/apps/get-app
get /v1/apps/{app_id}
Get an App by its `app_id`.
# Get app analytics overview
Source: https://docs.mixpeek.com/docs/api-reference/apps/get-app-analytics-overview
get /v1/apps/{app_id}/analytics
Per-app analytics dashboard data: error trend, Web Vitals, recent errors, event counts.
Queries ClickHouse canvas_errors, canvas_vitals, and canvas_app_events tables.
# Get Deploy Status
Source: https://docs.mixpeek.com/docs/api-reference/apps/get-deploy-status
GET /v1/apps/{app_id}/deploys/{deploy_id}
Check the status of a specific deployment (queued, building, complete, failed).
# Get presigned upload URL
Source: https://docs.mixpeek.com/docs/api-reference/apps/get-deploy-upload-url
post /v1/apps/{app_id}/deploy/upload-url
Generate a presigned S3 PUT URL so the CLI can upload a bundle zip directly.
Pass the returned `bundle_s3_key` to the deploy endpoint.
# Get recent error rate
Source: https://docs.mixpeek.com/docs/api-reference/apps/get-recent-error-rate
get /v1/apps/{app_id}/error-rate
Query ClickHouse canvas_errors for the error count in the last N minutes.
# Get runtime or request logs
Source: https://docs.mixpeek.com/docs/api-reference/apps/get-runtime-or-request-logs
get /v1/apps/{app_id}/logs
Query ClickHouse canvas_app_logs or canvas_request_logs.
# Get Version Details
Source: https://docs.mixpeek.com/docs/api-reference/apps/get-version
GET /v1/apps/{app_id}/versions/{version}
Return full details for a specific version including asset manifest and source files.
# Invite a user
Source: https://docs.mixpeek.com/docs/api-reference/apps/invite-a-user
post /v1/apps/{app_id}/users/invite
Invite a user to this app by email via Clerk.
# List app users
Source: https://docs.mixpeek.com/docs/api-reference/apps/list-app-users
get /v1/apps/{app_id}/users
List all users in this app's Clerk organization.
# List Apps
Source: https://docs.mixpeek.com/docs/api-reference/apps/list-apps
get /v1/apps
List all Apps for this organization.
The response includes ``max_apps`` (the tier's Canvas App limit, ``null`` =
unlimited) so the UI can render usage ("X of N apps used") and gate the
create action without a second round-trip.
# List custom domains
Source: https://docs.mixpeek.com/docs/api-reference/apps/list-domains
get /v1/apps/{app_id}/domains
List all custom domains attached to an App.
# List pending invitations
Source: https://docs.mixpeek.com/docs/api-reference/apps/list-pending-invitations
get /v1/apps/{app_id}/users/invitations
List pending invitations for this app's Clerk organization.
# List version history
Source: https://docs.mixpeek.com/docs/api-reference/apps/list-versions
get /v1/apps/{app_id}/versions
Return version history for an App, most recent first.
# Promote staging to production
Source: https://docs.mixpeek.com/docs/api-reference/apps/promote-staging-to-production
post /v1/apps/{app_id}/promote
Promote the current staging deployment to production.
Copies `environments.staging` (deploy_id, asset_prefix) to
`environments.production` and updates `build_config.asset_prefix`
for backward compatibility. Also invalidates the canvas asset cache
so the new production version is served immediately.
# Publish an App
Source: https://docs.mixpeek.com/docs/api-reference/apps/publish-app
post /v1/apps/{app_id}/publish
Publish the current draft config of an App.
For deploy-based apps, code deploys go live automatically — you only
need this endpoint to publish **config changes** (auth, meta, etc.).
Creates an immutable version snapshot with a content hash.
# Remove a user
Source: https://docs.mixpeek.com/docs/api-reference/apps/remove-a-user
delete /v1/apps/{app_id}/users/{user_id}
Remove a user from this app's Clerk organization.
# Restore a Version
Source: https://docs.mixpeek.com/docs/api-reference/apps/restore-version
POST /v1/apps/{app_id}/versions/{version}/restore
Restore a previous version by pointing the environment to its assets.
This is instant — no rebuild required. Old assets are always retained in S3.
# Revoke an invitation
Source: https://docs.mixpeek.com/docs/api-reference/apps/revoke-an-invitation
delete /v1/apps/{app_id}/users/invitations/{invitation_id}
Revoke a pending invitation.
# Rollback an App
Source: https://docs.mixpeek.com/docs/api-reference/apps/rollback-app
post /v1/apps/{app_id}/rollback
Rollback to the previous published config version.
For deploy-based apps, use ``POST /v1/apps/{app_id}/versions/{version}/restore``
to roll back to any previous code deploy instead.
# Stream deploy logs (SSE)
Source: https://docs.mixpeek.com/docs/api-reference/apps/stream-deploy-logs-sse
get /v1/apps/{app_id}/deploys/{deploy_id}/logs/stream
Server-Sent Events stream of real-time deploy logs.
Sends cached logs first (catch-up), then subscribes to live log
events via Redis pub/sub until the deploy reaches a terminal state.
# Update an App
Source: https://docs.mixpeek.com/docs/api-reference/apps/update-app
patch /v1/apps/{app_id}
Partially update an App configuration (saved as draft).
All fields are optional — only the fields you provide are updated.
Updates are draft by default. Set `publish: true` to apply and publish
in a single call.
# Update user role
Source: https://docs.mixpeek.com/docs/api-reference/apps/update-user-role
patch /v1/apps/{app_id}/users/{user_id}/role
Update a user's role in this app's Clerk organization.
# Verify a presigned-URL upload landed in S3
Source: https://docs.mixpeek.com/docs/api-reference/apps/verify-a-presigned-url-upload-landed-in-s3
post /v1/apps/{app_id}/deploy/verify-upload
Confirm a bundle uploaded via a presigned PUT URL actually landed in S3.
S3 returns an empty body on a successful PUT — the ETag is only in
response headers, which browser clients often can't read due to CORS.
This endpoint does a server-side HEAD and returns {etag, size_bytes,
last_modified, exists} so clients can verify the upload before calling
`/deploy`.
# Trigger DNS verification
Source: https://docs.mixpeek.com/docs/api-reference/apps/verify-domain
post /v1/apps/{app_id}/domains/{domain}/verify
Trigger asynchronous DNS TXT record verification for a domain.
The domain status is set to `verifying` immediately. A Celery worker
polls DNS every 30 minutes for up to 72 hours. On success the status
advances to `provisioning_tls`.
# Cancel Execution
Source: https://docs.mixpeek.com/docs/api-reference/cluster-executions/cancel-execution
post /v1/clusters/{cluster_id}/executions/{run_id}/cancel
Cancel a pending or processing cluster execution.
This cancels the Ray job (if running) and marks the execution as CANCELED.
Useful for:
- Cleaning up orphaned executions where the Ray job was never submitted
- Stopping long-running clustering jobs
- Handling executions stuck in pending/processing state
Only works for executions in 'pending' or 'processing' status.
Returns an error if the execution is already 'completed', 'failed', or 'canceled'.
# Get execution artifacts
Source: https://docs.mixpeek.com/docs/api-reference/cluster-executions/get-execution-artifacts
get /v1/clusters/{cluster_id}/executions/{run_id}/artifacts
# Get Latest Cluster Execution
Source: https://docs.mixpeek.com/docs/api-reference/cluster-executions/get-latest-cluster-execution
get /v1/clusters/{cluster_id}/executions
Get the most recent execution results for a cluster.
Returns execution metadata including:
- Execution status (pending, processing, completed, failed)
- Clustering metrics (silhouette score, Davies-Bouldin index, etc.)
- Number of clusters found and documents processed
- Centroid information with labels and summaries
- Execution timestamps
Useful for:
- Displaying cluster statistics in dashboards
- Showing cluster quality metrics to users
- Rendering cluster labels and summaries in the UI
- Tracking execution status and errors
# Get Specific Cluster Execution
Source: https://docs.mixpeek.com/docs/api-reference/cluster-executions/get-specific-cluster-execution
get /v1/clusters/{cluster_id}/executions/{run_id}
Get a specific execution by run ID.
Returns detailed execution information for a particular clustering run,
allowing you to review historical executions and compare results over time.
# List Cluster Execution History
Source: https://docs.mixpeek.com/docs/api-reference/cluster-executions/list-cluster-execution-history
post /v1/clusters/{cluster_id}/executions/list
List execution history for a cluster with pagination, filtering, sorting, and search.
Returns all historical executions for the specified cluster, including:
- Execution status (pending, processing, completed, failed)
- Clustering metrics (silhouette score, Davies-Bouldin index, etc.)
- Number of clusters found and documents processed
- Execution timestamps and duration
- Centroid information
Supports:
- **Filtering**: Filter by status, date range, metrics, etc.
- **Sorting**: Sort by created_at, execution time, metrics
- **Search**: Full-text search across execution metadata
- **Pagination**: Limit and offset for large result sets
Use cases:
- View all past executions for a cluster
- Compare metrics across runs
- Track execution history over time
- Debug failed executions
- Analyze clustering performance trends
# Stream execution data
Source: https://docs.mixpeek.com/docs/api-reference/cluster-executions/stream-execution-data
post /v1/clusters/{cluster_id}/executions/{run_id}/data
# Create Cluster Trigger
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/create-cluster-trigger
post /v1/clusters/triggers
Create a new trigger for automated cluster execution.
Supports multiple trigger types:
- **cron**: Execute at specific times using cron expressions
- **interval**: Execute at fixed intervals
- **event**: Execute when specific events occur (e.g., documents added)
- **conditional**: Execute when conditions are met (e.g., drift threshold)
# Delete Cluster Trigger
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/delete-cluster-trigger
delete /v1/clusters/triggers/{trigger_id}
Delete a cluster trigger (soft delete).
# Get Cluster Trigger
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/get-cluster-trigger
get /v1/clusters/triggers/{trigger_id}
Get a cluster trigger by ID.
# Get Trigger Execution History
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/get-trigger-execution-history
post /v1/clusters/triggers/{trigger_id}/history
Get execution history for a trigger with pagination.
# List Cluster Triggers
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/list-cluster-triggers
post /v1/clusters/triggers/list
List cluster triggers with filters and pagination.
# Pause Cluster Trigger
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/pause-cluster-trigger
post /v1/clusters/triggers/{trigger_id}/pause
Pause trigger execution. Paused triggers retain configuration but do not execute.
# Resume Cluster Trigger
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/resume-cluster-trigger
post /v1/clusters/triggers/{trigger_id}/resume
Resume paused trigger. Next execution time is recalculated from current time.
# Update Cluster Trigger
Source: https://docs.mixpeek.com/docs/api-reference/cluster-triggers/update-cluster-trigger
patch /v1/clusters/triggers/{trigger_id}
Update a cluster trigger.
Allowed updates:
- schedule_config: Modify trigger schedule
- description: Update description
- status: Change status (use pause/resume endpoints instead)
Not allowed:
- trigger_type: Must delete and recreate
- cluster_id: Immutable
- execution_config: Immutable
# Apply Cluster Enrichment
Source: https://docs.mixpeek.com/docs/api-reference/clusters/apply-cluster-enrichment
post /v1/clusters/enrich
Apply clustering enrichments to a collection via engine.
# Create Cluster
Source: https://docs.mixpeek.com/docs/api-reference/clusters/create-cluster
post /v1/clusters
Create a new cluster configuration and output collection.
This endpoint:
1. Creates cluster metadata
2. Creates output collection for cluster documents
3. Returns cluster metadata with output_collection_id
The cluster can then be executed via POST /v1/clusters/{id}/execute
# Delete Cluster
Source: https://docs.mixpeek.com/docs/api-reference/clusters/delete-cluster
delete /v1/clusters/{cluster_id}
This endpoint deletes a cluster and all its resources including:
- Running Ray jobs (cancels active jobs)
- Cluster triggers
- Execution history (clustering_results)
- S3 artifacts (parquet files, documents, members)
- Related tasks
- Clustering jobs
- MongoDB cluster metadata
- Output collections created by the cluster (by default)
By default, output collections created by the cluster are cascade-deleted
along with the cluster. Pass `cascade_output_collections=false` to preserve
them (useful when the cluster output documents were enriched / promoted
into user-managed data).
The deletion is performed synchronously and returns when complete.
# Execute Clustering
Source: https://docs.mixpeek.com/docs/api-reference/clusters/execute-clustering
post /v1/clusters/{cluster_id}/execute
Execute clustering on a specific cluster.
This endpoint:
1. Validates the cluster exists
2. Queues clustering job via Celery
3. Returns task_id immediately (non-blocking)
4. Celery prepares data and submits to Engine
5. Monitor progress via GET /v1/tasks/{task_id}
Flow:
- API: Receives request
- Celery: Fetches documents, creates parquet, uploads to S3
- Engine: Runs Ray job on parquet data
- Status: Automatically updates cluster when complete
Use GET /v1/clusters/{id}/executions to retrieve results.
Optional body (`ExecuteClusterByIdRequest`) accepts a `filters` override
that applies to this execution only and is not persisted on the cluster.
# Get Cluster
Source: https://docs.mixpeek.com/docs/api-reference/clusters/get-cluster
get /v1/clusters/{cluster_identifier}
Retrieve a cluster by ID or name.
Returns cluster metadata including:
- Configuration (cluster_type, algorithm, parameters)
- Output collection information (output_collection_id, output_collection_name)
- Execution results (num_clusters, num_documents_clustered, status)
- Timestamps and metadata
# Get cluster artifacts
Source: https://docs.mixpeek.com/docs/api-reference/clusters/get-cluster-artifacts
get /v1/clusters/{cluster_id}/artifacts
# List Cluster Groups
Source: https://docs.mixpeek.com/docs/api-reference/clusters/list-cluster-groups
get /v1/clusters/{cluster_id}/groups
Get all cluster groups with labels, summaries, and document counts from the latest execution.
# List Clusters
Source: https://docs.mixpeek.com/docs/api-reference/clusters/list-clusters
post /v1/clusters/list
This endpoint allows you to list clusters.
# Partially Update Cluster
Source: https://docs.mixpeek.com/docs/api-reference/clusters/partially-update-cluster
patch /v1/clusters/{cluster_identifier}
This endpoint partially updates a cluster (PATCH operation).
Only provided fields will be updated. At minimum, metadata can always be updated.
Immutable fields like cluster_id, status, and computed fields cannot be modified.
# Stream cluster data
Source: https://docs.mixpeek.com/docs/api-reference/clusters/stream-cluster-data
post /v1/clusters/{cluster_id}/data
# Create Trigger
Source: https://docs.mixpeek.com/docs/api-reference/create-trigger
post /v1/triggers
Create a new trigger for scheduled job execution.
**Action Types:**
- `cluster`: Execute clustering on a cluster definition
- `taxonomy_enrichment`: Apply taxonomy enrichment to a collection
**Schedule Types:**
- `cron`: Execute at specific times using cron expressions (e.g., "0 2 * * *" for daily at 2am)
- `interval`: Execute at fixed intervals (e.g., every 6 hours)
- `event`: Execute when specific events occur (e.g., after 100 documents added)
- `conditional`: Execute when conditions are met (e.g., drift threshold exceeded)
**Examples:**
Cluster trigger (daily at 2am):
```json
{
"action_type": "cluster",
"action_config": {"cluster_id": "clust_abc123"},
"trigger_type": "cron",
"schedule_config": {"cron_expression": "0 2 * * *", "timezone": "UTC"},
"description": "Daily clustering at 2am"
}
```
Taxonomy enrichment trigger (every 6 hours):
```json
{
"action_type": "taxonomy_enrichment",
"action_config": {
"taxonomy_id": "tax_products",
"collection_id": "col_inventory",
"batch_size": 1000
},
"trigger_type": "interval",
"schedule_config": {"interval_seconds": 21600},
"description": "Re-enrich products every 6 hours"
}
```
# Delete Trigger
Source: https://docs.mixpeek.com/docs/api-reference/delete-trigger
delete /v1/triggers/{trigger_id}
Delete a trigger (soft delete - sets status to disabled).
# Execute Trigger Now
Source: https://docs.mixpeek.com/docs/api-reference/execute-trigger-now
post /v1/triggers/{trigger_id}/execute
Manually execute a trigger immediately.
This bypasses the schedule and executes the trigger's action right away.
Useful for testing trigger configuration or forcing immediate execution.
Returns a task response that can be used to monitor execution progress.
# Get Trigger
Source: https://docs.mixpeek.com/docs/api-reference/get-trigger
get /v1/triggers/{trigger_id}
Get a trigger by ID.
# Get Trigger Execution History
Source: https://docs.mixpeek.com/docs/api-reference/get-trigger-execution-history
get /v1/triggers/{trigger_id}/history
Get execution history for a trigger with pagination.
# List Triggers
Source: https://docs.mixpeek.com/docs/api-reference/list-triggers
post /v1/triggers/list
List triggers with filters and pagination.
**Filters:**
- `action_type`: Filter by action type (cluster, taxonomy_enrichment)
- `trigger_type`: Filter by trigger type (cron, interval, event, conditional)
- `status`: Filter by status (active, paused, disabled, failed)
- `resource_id`: Filter by resource ID (cluster_id or taxonomy_id)
# Pause Trigger
Source: https://docs.mixpeek.com/docs/api-reference/pause-trigger
post /v1/triggers/{trigger_id}/pause
Pause trigger execution. Paused triggers retain configuration but do not execute.
# Get public App config
Source: https://docs.mixpeek.com/docs/api-reference/public-apps-api/get-app-config
get /v1/public/apps/{slug}/config
Fetch the full configuration for a public App — no authentication required.
Used by the `canvas/` host runtime and the `@mixpeek/canvas-sdk`
`useApp()` hook to render the app. Returns tabs, hero, stats, sections,
theme, and SEO settings.
Serves the `published_config` snapshot when available, so draft edits
made via `PATCH /v1/apps/{app_id}` are not visible until the app is
published.
# Execute an App tab search
Source: https://docs.mixpeek.com/docs/api-reference/public-apps-api/search-app
post /v1/public/apps/{slug}/search
Execute a search on a specific tab of a public App — no authentication required.
Routes `tab_id` → retriever → results. Each tab can be backed by either:
- An **internal retriever** (`retriever_id`) — org-scoped
- A **marketplace catalog entry** (`public_name`) — proxied via public API
The `inputs` dict is passed directly to the retriever's `input_schema`.
Extra top-level fields are automatically merged into `inputs`.
**Example:**
```json
{
"tab_id": "search",
"inputs": { "query": "red sneakers" },
"settings": { "limit": 20 }
}
```
# Execute Public Retriever
Source: https://docs.mixpeek.com/docs/api-reference/public-retriever-api/execute-public-retriever
post /v1/public/retrievers/{public_name}/execute
Execute a published retriever (public endpoint).
**Authentication:**
- API key is OPTIONAL for public retrievers
- Supports: no key, prk_ keys (deprecated), or ret_sk_ keys
- If password-protected, requires `X-Retriever-Password` header
**Rate Limiting:**
- Subject to per-retriever rate limits (per minute/hour/day)
- May also have IP-based rate limits
**Response:**
- Only returns fields specified in `exposed_fields` configuration
- Internal metadata is stripped from results
- Includes `execution_id` for interaction tracking
- Presigned URLs returned by default (return_presigned_urls=true) for media rendering
**Example (no API key - recommended for public access):**
```bash
curl -X POST "https://api.mixpeek.com/v1/public/retrievers/video-search/execute" \
-H "Content-Type: application/json" \
-d '{
"inputs": {"query": "red car"},
"pagination": {"method": "offset", "page_number": 1, "page_size": 10}
}'
```
**Example with ret_sk_ key (for SDK/programmatic access):**
```bash
curl -X POST "https://api.mixpeek.com/v1/public/retrievers/video-search/execute" \
-H "X-Public-API-Key: ret_sk_abc123..." \
-H "Content-Type: application/json" \
-d '{
"inputs": {"query": "red car"},
"pagination": {"method": "offset", "page_number": 1, "page_size": 10}
}'
```
# Get Public Retriever Config
Source: https://docs.mixpeek.com/docs/api-reference/public-retriever-api/get-public-retriever-config
get /v1/public/retrievers/{public_name}/config
Get display configuration for public page rendering.
⚠️ **DEPRECATED**: Use `/v1/marketplace/catalog/{public_name}/config` instead.
This endpoint is maintained for backwards compatibility but may be removed in the future.
Returns the UI configuration needed to render the public search interface.
Used by the frontend app at mxp.co to dynamically build the UI.
**Authentication:**
- NO authentication required - this endpoint is public
- Anyone can access the config if they know the public_name
- The config includes the public_api_key needed for execute/interact endpoints
**Response includes:**
- Display config (logo, theme, components, field rendering)
- Title and description
- Password protection status
- Public API key for subsequent authenticated requests
**Example (deprecated):**
```bash
curl -X GET "https://api.mixpeek.com/v1/public/retrievers/video-search/config"
```
**Example (recommended):**
```bash
curl -X GET "https://api.mixpeek.com/v1/marketplace/catalog/video-search/config"
```
# Get Public Retriever Template
Source: https://docs.mixpeek.com/docs/api-reference/public-retriever-api/get-public-retriever-template
get /v1/public/retrievers/{public_name}/template
Get retriever configuration as a reusable template.
Returns the published retriever's configuration in a format that can be
directly used to create your own retriever. This is perfect for discovering
patterns and adapting them to your own data.
**Authentication:**
- NO authentication required - this endpoint is completely public
- Anyone can get the template if they know the public_name
**Use Case:**
1. Browse public retrievers to find patterns you like
2. GET this endpoint to get the full configuration
3. Copy the config and modify for your needs (especially `collection_identifiers`)
4. POST to `/v1/retrievers` to create your own retriever
5. Optionally publish it with the same display_config
**What's included:**
- Retriever configuration (stages, input_schema, budget_limits)
- Display configuration (for publishing with similar UI)
- Original metadata for reference
**What you need to change:**
- `collection_identifiers`: Replace with your own collection IDs
- `retriever_name`: Give it a unique name
- Optionally modify stages, inputs, display_config as needed
**Example:**
```bash
# 1. Get the template
curl -X GET "https://api.mixpeek.com/v1/public/retrievers/video-search/template"
# 2. Modify the response and create your own retriever
curl -X POST "https://api.mixpeek.com/v1/retrievers" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"retriever_name": "my_video_search",
"collection_identifiers": ["my_videos"],
"stages": [...], # From template
"input_schema": {...}, # From template
"budget_limits": {...}, # From template
"display_config": {...} # From template (optional)
}'
```
**Response includes:**
- All retriever configuration fields
- Display config for publishing (optional to use)
- Source reference (where this template came from)
# Track Interaction
Source: https://docs.mixpeek.com/docs/api-reference/public-retriever-api/track-interaction
post /v1/public/retrievers/{public_name}/interactions
Track user interaction with search results.
Records user engagement (clicks, views, etc.) for analytics and
potential search optimization (Learning to Rank).
**Authentication:**
- API key is OPTIONAL (same as execute endpoint)
- Password NOT required (tracking should work even without auth)
**Recommended Headers:**
- `X-Session-ID`: Session identifier for tracking user journey
**Interaction Types:**
- `VIEW`: Result was visible in viewport
- `CLICK`: User clicked on result
- `POSITIVE_FEEDBACK`: User explicitly liked result
- `NEGATIVE_FEEDBACK`: User explicitly disliked result
- `PURCHASE`: User purchased/converted
- `ADD_TO_CART`: User added to cart
- `WISHLIST`: User added to wishlist
- `LONG_VIEW`: User spent significant time viewing
- `SHARE`: User shared result
- `BOOKMARK`: User bookmarked result
**Example:**
```bash
curl -X POST "https://api.mixpeek.com/v1/public/retrievers/video-search/interactions" \
-H "X-Session-ID: sess_xyz..." \
-H "Content-Type: application/json" \
-d '{
"document_id": "doc_123",
"interaction_type": ["CLICK"],
"position": 2,
"execution_id": "exec_abc",
"query_snapshot": {"query": "red car"}
}'
```
# Track Interaction Batch
Source: https://docs.mixpeek.com/docs/api-reference/public-retriever-api/track-interaction-batch
post /v1/public/retrievers/{public_name}/interactions/batch
Track multiple interactions in a single request (batching).
More efficient than sending individual interaction requests.
Use this for batching viewport visibility, bulk actions, etc.
**Authentication:**
- API key is OPTIONAL (same as execute endpoint)
- Password NOT required (tracking should work even without auth)
**Recommended Headers:**
- `X-Session-ID`: Applied to all interactions in the batch
**Limits:**
- Maximum 100 interactions per batch
**Example:**
```bash
curl -X POST "https://api.mixpeek.com/v1/public/retrievers/video-search/interactions/batch" \
-H "X-Session-ID: sess_xyz..." \
-H "Content-Type: application/json" \
-d '{
"interactions": [
{
"document_id": "doc_123",
"interaction_type": ["VIEW"],
"position": 0,
"execution_id": "exec_abc"
},
{
"document_id": "doc_456",
"interaction_type": ["VIEW"],
"position": 1,
"execution_id": "exec_abc"
}
]
}'
```
# Verify Password
Source: https://docs.mixpeek.com/docs/api-reference/public-retriever-api/verify-password
post /v1/public/retrievers/{public_name}/verify
Verify password for a password-protected retriever.
Allows the frontend to check if a password is valid before attempting to execute
a password-protected retriever. Returns the public API key if the password is valid.
**Authentication:**
- NO authentication required - this endpoint is public
- The password is verified against the retriever's configured password
**Use Case:**
1. Frontend detects that a retriever is password-protected (from /config endpoint)
2. User enters password in the UI
3. Frontend calls this endpoint to verify the password
4. If valid, frontend receives the public_api_key to use for subsequent requests
**Response:**
- `valid`: Whether the password is correct
- `public_api_key`: The API key to use for execute/interact endpoints (only if valid)
**Example:**
```bash
curl -X POST "https://api.mixpeek.com/v1/public/retrievers/private-search/verify" \
-H "Content-Type: application/json" \
-d '{"password": "secret123"}'
```
**Response if valid:**
```json
{
"valid": true,
"public_api_key": "prk_abc123..."
}
```
**Response if invalid:**
```json
{
"valid": false,
"public_api_key": null
}
```
# Resume Trigger
Source: https://docs.mixpeek.com/docs/api-reference/resume-trigger
post /v1/triggers/{trigger_id}/resume
Resume a paused trigger. Next execution time is recalculated from current time.
# Clone Taxonomy
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/clone-taxonomy
post /v1/taxonomies/{taxonomy_identifier}/clone
Clone a taxonomy with optional modifications.
**Purpose:**
Creates a NEW taxonomy (with new ID) based on an existing one. This is the
recommended way to iterate on taxonomy designs when you need to modify core
logic that PATCH doesn't allow (config, retriever_id, input_mappings).
**Clone vs PATCH vs Template:**
- **PATCH**: Update metadata only (name, description, metadata)
- **Clone**: Copy and modify core logic (config, retriever, collections)
- **Template**: Start from a pre-configured pattern (for new projects)
**Common Use Cases:**
- Fix configuration errors without losing join history
- Change retriever or input mappings
- Modify enrichment fields or collection configuration
- Test modifications before replacing production taxonomy
- Create variants for different datasets
**How it works:**
1. Source taxonomy is copied
2. You provide a new name (REQUIRED)
3. Optionally override any other fields (description, config)
4. A new taxonomy is created with a new ID
5. Original taxonomy remains unchanged
# Create Taxonomy
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/create-taxonomy
post /v1/taxonomies
Create a taxonomy and return the created resource.
# Create Taxonomy Version
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/create-taxonomy-version
post /v1/taxonomies/{taxonomy_id}/versions
Create a new version for a taxonomy with a new config snapshot.
# Delete Taxonomy
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/delete-taxonomy
delete /v1/taxonomies/{taxonomy_identifier}
This endpoint deletes a taxonomy and all its resources including:
- Taxonomy versions (version snapshots)
- Taxonomy metadata from MongoDB
The deletion is performed synchronously and returns when complete.
# Get Taxonomy
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/get-taxonomy
get /v1/taxonomies/{taxonomy_identifier}
Get a taxonomy by ID or name.
# List Taxonomies
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/list-taxonomies
post /v1/taxonomies/list
List taxonomies with optional filters and pagination.
# List Taxonomy Versions
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/list-taxonomy-versions
get /v1/taxonomies/{taxonomy_id}/versions
List all versions for a taxonomy (head included as latest).
# Partially Update Taxonomy
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/partially-update-taxonomy
patch /v1/taxonomies/{taxonomy_identifier}
Update a taxonomy's metadata.
**Metadata Only Updates:**
This endpoint allows updating ONLY metadata fields. Core taxonomy logic is immutable
to ensure consistency for join history and dependent resources.
**Fields You CAN Update:**
- taxonomy_name: Rename the taxonomy
- description: Update documentation
- metadata: Update custom metadata
**Fields You CANNOT Update:**
- config: Taxonomy configuration (retriever_id, input_mappings, collections)
- taxonomy_type: Type (flat vs hierarchical)
**Need to Modify Core Logic?**
Use POST /{taxonomy_identifier}/clone instead to modify configuration,
retriever_id, input_mappings, or collections.
# Test taxonomy configuration (validation only) — DEPRECATED path
Source: https://docs.mixpeek.com/docs/api-reference/taxonomies/test-taxonomy-configuration-validation-only
post /v1/taxonomies/execute/{taxonomy_identifier}
⚠️ VALIDATION ENDPOINT ONLY - Not for production enrichment!
DEPRECATED path shape — prefer POST /taxonomies/{id}/execute (id-first,
consistent with retrievers). This verb-first path still works.
This endpoint validates taxonomy configuration with 1-5 sample documents.
Results are returned immediately and NOT persisted to any collection.
❌ DO NOT USE FOR:
- Enriching entire collections (use taxonomy_applications instead)
- Batch processing documents (automatic during ingestion)
- Persisting enriched documents (use retriever pipelines instead)
✅ USE THIS FOR:
- Testing taxonomy configuration is correct
- Validating retriever finds matching taxonomy nodes
- Checking enrichment fields are properly applied
- Development/debugging taxonomy setup
📚 FOR PRODUCTION ENRICHMENT:
Automatic (during ingestion):
1. Create taxonomy: POST /taxonomies
2. Attach to collection: PUT /collections/{id} with taxonomy_applications field
3. Ingest documents: Documents are automatically enriched by engine
On-the-fly (during retrieval):
1. Add taxonomy_join stage to retriever pipeline
2. Execute retriever: GET /retrievers/{id}/execute
3. Results include enriched documents (not persisted)
See API documentation for Collections and Retrievers for details.
# Analyze multi-step transition paths
Source: https://docs.mixpeek.com/docs/api-reference/taxonomy-analytics/analyze-multi-step-transition-paths
post /v1/taxonomies/{taxonomy_id}/analytics/paths
Discover the most common multi-step paths documents take between two taxonomy steps.
Unlike the `/transitions` endpoint which only analyzes direct A→B transitions,
this endpoint reveals the intermediate steps documents actually take.
## Use Cases
**Email Thread Analysis:**
- Question: What paths do emails take from "inquiry" to "closed_won"?
- Discover: Some go inquiry → followup → proposal → closed_won
- Discover: Others skip steps: inquiry → proposal → closed_won
- Discover: Fast track: inquiry → closed_won
**Content Editorial Paths:**
- Question: Common paths from "draft" to "published"?
- Discover: draft → review → edit → review → published
- Discover: draft → review → published (expedited)
- Discover: Paths that loop back (draft → review → draft → review)
**Compliance Resolution Paths:**
- Question: How do violations get resolved?
- Discover: violation → investigated → remediated → resolved
- Discover: violation → false_positive → closed
- Discover: Escalation paths: violation → escalated → legal_review → resolved
## Requirements
- Taxonomy must have `step_analytics` configured
- Collection must contain documents with timestamp and sequence_id fields
## Returns
**Completion Metrics:**
- `total_sequences`: Sequences starting at from_step
- `completed_sequences`: Number reaching to_step
- `completion_rate`: Percentage that completed
**Paths (sorted by frequency):**
- `path`: Ordered sequence of steps
- `count`: Number of sequences following this path
- `percentage`: Percentage of completing sequences
- `avg_duration_sec`: Average time for this path
## Example Request
```json
{
"collection_id": "col_emails",
"taxonomy_id": "tax_sales_stages",
"from_step": "inquiry",
"to_step": "closed_won",
"max_path_length": 10,
"min_support": 5
}
```
## Example Response
```json
{
"from_step": "inquiry",
"to_step": "closed_won",
"total_sequences": 1000,
"completed_sequences": 350,
"completion_rate": 0.35,
"paths": [
{
"path": ["inquiry", "followup", "proposal", "closed_won"],
"count": 120,
"percentage": 34.3,
"avg_duration_sec": 604800.0
},
{
"path": ["inquiry", "proposal", "closed_won"],
"count": 90,
"percentage": 25.7,
"avg_duration_sec": 432000.0
},
{
"path": ["inquiry", "closed_won"],
"count": 70,
"percentage": 20.0,
"avg_duration_sec": 172800.0
}
]
}
```
## Path Interpretation
**Length Analysis:**
- Shorter paths indicate efficient progression
- Longer paths may indicate complexity or bottlenecks
- Loops (repeated steps) indicate rework or revisions
**Duration Analysis:**
- Compare avg_duration_sec across paths
- Shorter paths may not always be faster
- Identify optimization opportunities
**Frequency Analysis:**
- High-percentage paths are "happy paths"
- Low-percentage paths may be edge cases or exceptions
- Missing expected paths indicate drop-off points
# Compute step transition analytics
Source: https://docs.mixpeek.com/docs/api-reference/taxonomy-analytics/compute-step-transition-analytics
post /v1/taxonomies/{taxonomy_id}/analytics/transitions
Analyze how documents progress from one taxonomy step to another.
This endpoint computes conversion rates, duration statistics, and predictor lifts
for documents transitioning between taxonomy labels.
## Use Cases
**Email Thread Analysis:**
- Question: How long from "inquiry" to "closed_won"?
- Question: What % of inquiries result in sales?
- Question: Which sender domains have highest conversion?
**Content Workflow Tracking:**
- Question: Conversion rate from "draft" to "published"?
- Question: How long does content stay in review?
- Question: Which authors publish fastest?
**Safety Compliance Monitoring:**
- Question: Time from violation detection to resolution?
- Question: Success rate for remediation efforts?
## Requirements
- Taxonomy must have `step_analytics` configured (or provide `override_step_analytics`)
- Collection must contain documents enriched with this taxonomy
- Documents must have timestamp and sequence grouping fields configured
## Returns
**Conversion Metrics:**
- `count`: Total sequences starting at from_step
- `converted`: Number reaching to_step
- `conversion_rate`: Percentage that converted
**Duration Statistics (if converted > 0):**
- `mean`, `median`: Average and middle duration
- `p90`, `p95`: 90th and 95th percentile durations
- `std_dev`, `min`, `max`: Distribution statistics
**Top Predictors:**
- Covariates with highest impact on conversion
- Lift values (>1.0 = increases conversion, <1.0 = decreases)
- Statistical significance via minimum support threshold
## Example Request
```json
{
"collection_id": "col_emails",
"taxonomy_id": "tax_sales_stages",
"from_step": "inquiry",
"to_step": "closed_won",
"max_window_days": 90,
"min_support": 10
}
```
## Example Response
```json
{
"from_step": "inquiry",
"to_step": "closed_won",
"count": 1000,
"converted": 350,
"conversion_rate": 0.35,
"durations_sec": {
"mean": 432000.0,
"median": 345600.0,
"p50": 345600.0,
"p90": 691200.0,
"p95": 864000.0
},
"top_predictors": [
{
"field": "Sender Domain",
"value": "enterprise.com",
"count": 150,
"conversion_rate": 0.75,
"lift": 2.14
}
]
}
```
# Get Available Steps
Source: https://docs.mixpeek.com/docs/api-reference/taxonomy-analytics/get-available-steps
get /v1/taxonomies/{taxonomy_id}/analytics/available-steps
Get all available steps for a taxonomy and collection.
This endpoint discovers what steps exist in your analytics data by querying
the ClickHouse taxonomy_events table. Use this before querying transitions
or paths to understand what step values you can use.
**Use Cases:**
- Discover available steps before querying analytics
- Validate step names (avoid typos in from_step/to_step)
- See which steps have the most events
- Check data freshness (first_seen/last_seen timestamps)
**Example Usage:**
```python
# 1. Get available steps
GET /v1/taxonomies/tax_sales/analytics/available-steps?collection_id=col_emails
# Response:
{
"taxonomy_id": "tax_sales",
"collection_id": "col_emails",
"total_events": 5432,
"total_sequences": 1000,
"steps": [
{"step_key": "inquiry", "event_count": 1000, ...},
{"step_key": "followup", "event_count": 450, ...},
{"step_key": "closed_won", "event_count": 350, ...}
]
}
# 2. Use discovered steps in transition query
POST /v1/taxonomies/tax_sales/analytics/transitions
{
"collection_id": "col_emails",
"from_step": "inquiry", # From available steps
"to_step": "closed_won" # From available steps
}
```
Args:
request: FastAPI request object (contains tenant context)
taxonomy_id: Taxonomy ID to query
collection_id: Collection ID for filtering events
Returns:
AvailableStepsResponse with all steps sorted by event count (descending)
Raises:
NotFoundError: If taxonomy not found
ValidationError: If unable to query ClickHouse
# Update Trigger
Source: https://docs.mixpeek.com/docs/api-reference/update-trigger
patch /v1/triggers/{trigger_id}
Update a trigger.
**Allowed updates:**
- `schedule_config`: Modify trigger schedule
- `description`: Update description
- `status`: Change status (prefer using pause/resume endpoints)
**Not allowed:**
- `action_type`: Must delete and recreate
- `trigger_type`: Must delete and recreate
- `action_config`: Must delete and recreate
# Create Webhook
Source: https://docs.mixpeek.com/docs/api-reference/webhooks/create-webhook
post /v1/organizations/webhooks
Create a new webhook for the user's organization.
# Delete Webhook
Source: https://docs.mixpeek.com/docs/api-reference/webhooks/delete-webhook
delete /v1/organizations/webhooks/{identifier}
Delete a webhook (idempotent - succeeds even if already deleted).
# Get Webhook
Source: https://docs.mixpeek.com/docs/api-reference/webhooks/get-webhook
get /v1/organizations/webhooks/{identifier}
Get a single webhook by its ID.
# List Webhooks
Source: https://docs.mixpeek.com/docs/api-reference/webhooks/list-webhooks
post /v1/organizations/webhooks/list
List all webhooks for the user's organization.
# Update Webhook
Source: https://docs.mixpeek.com/docs/api-reference/webhooks/update-webhook
put /v1/organizations/webhooks/{identifier}
Update an existing webhook.
# Create Session
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/create-session
post /v1/agents/sessions
Create a new agent session.
A session represents a stateful conversation with an AI agent that can
call tools to search data, filter results, and perform multi-step reasoning.
Args:
request: FastAPI request with tenant context
payload: Session creation request
Returns:
CreateSessionResponse with session metadata
Example:
```bash
curl -X POST http://localhost:8000/v1/agents/sessions \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{
"agent_config": {
"model": "claude-3-5-sonnet-20241022",
"temperature": 0.7,
"available_tools": ["search_retrievers", "execute_retriever"]
},
"quotas": {
"max_messages": 100,
"max_tokens_total": 100000
}
}'
```
# Detect Intent
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/detect-intent
post /v1/agents/sessions/intent/detect
Detect user intent from natural language request.
This endpoint analyzes a user's request to determine whether they want to:
- Execute queries on existing data (execution mode)
- Create new resources/infrastructure (setup mode)
- Or if the request is ambiguous and needs clarification
It performs keyword analysis and checks existing collections to provide
intelligent classification and recommendations.
Args:
request: FastAPI request with tenant context
payload: Intent detection request with user's input
Returns:
IntentClassification with detected intent and recommendations
Example:
```bash
curl -X POST http://localhost:8000/v1/agents/intent/detect \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{
"user_request": "I want to search videos by faces",
"include_collection_analysis": true
}'
```
# Get History
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/get-history
get /v1/agents/sessions/{session_id}/history
Get conversation history for a session.
Returns messages in chronological order (oldest first).
Args:
request: FastAPI request with tenant context
session_id: Session identifier
limit: Maximum messages to return (default: 50, max: 200)
offset: Pagination offset (default: 0)
Returns:
GetHistoryResponse with message history
Raises:
NotFoundError: If session not found
Example:
```bash
curl -X GET "http://localhost:8000/v1/agents/sessions/ses_abc123/history?limit=20&offset=0" \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}"
```
# Get Session
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/get-session
get /v1/agents/sessions/{session_id}
Get session metadata by ID.
Args:
request: FastAPI request with tenant context
session_id: Session identifier
Returns:
GetSessionResponse with session metadata
Raises:
NotFoundError: If session not found
Example:
```bash
curl -X GET http://localhost:8000/v1/agents/sessions/ses_abc123 \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}"
```
# List Sessions
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/list-sessions
post /v1/agents/sessions/list
List agent sessions in the namespace.
Args:
request: FastAPI request with tenant context
list_request: Optional filters and sorting
pagination: Pagination parameters
Returns:
ListSessionsResponse with session list
Example:
```bash
curl -X POST http://localhost:8000/v1/agents/sessions/list \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{"status": "active"}'
```
# List Tools
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/list-tools
get /v1/agents/sessions/tools
List all available agent tools.
Use this endpoint to discover available tools before creating a session.
Tool Categories:
- search: Tools for searching data (execute_retriever)
- read: Tools for reading resources (list_*, get_*)
- create: Tools for creating resources (create_*) - requires confirmation
- update: Tools for updating resources (update_*) - requires confirmation
- delete: Tools for deleting resources (delete_*) - requires confirmation
- upload: Tools for file uploads (upload_object)
Note: Write operations (create, update, delete) require user confirmation
via the /confirmations endpoint before execution.
Args:
request: FastAPI request with tenant context
category: Optional filter by tool category
Returns:
ListToolsResponse with available tools
Example:
```bash
# List all tools
curl -X GET http://localhost:8000/v1/agents/tools \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}"
# List only search tools
curl -X GET "http://localhost:8000/v1/agents/tools?category=search" \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}"
```
# Patch Session
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/patch-session
patch /v1/agents/sessions/{session_id}
Update session metadata.
Only user_memory can be updated. To change agent configuration,
create a new session.
Args:
request: FastAPI request with tenant context
session_id: Session identifier
payload: Update request
Returns:
PatchSessionResponse with update timestamp
Raises:
NotFoundError: If session not found
Example:
```bash
curl -X PATCH http://localhost:8000/v1/agents/sessions/ses_abc123 \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{
"user_memory": {
"preferences": {"language": "en", "domain": "tech"}
}
}'
```
# Respond To Confirmation
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/respond-to-confirmation
post /v1/agents/sessions/{session_id}/confirmations/{confirmation_id}
Respond to a pending confirmation for a write operation.
When the agent requests a write operation (create, update, delete),
the stream pauses and emits a `confirmation_required` event. The user
must call this endpoint to approve or deny the action.
After responding, this endpoint returns a new SSE stream that continues
the agent's execution from where it paused.
## Confirmation Workflow
1. User sends message via POST /sessions/{id}/messages
2. Agent proposes a write operation
3. Stream emits `confirmation_required` event with `confirmation_id`
4. User calls this endpoint with `approved: true/false`
5. This endpoint returns SSE stream continuing the agent's work
6. If approved, agent executes the tool and continues
7. If denied, agent acknowledges and continues without executing
## Confirmation Expiration
Confirmations expire after 5 minutes. Attempting to respond to an
expired confirmation returns a 400 error.
Args:
request: FastAPI request with tenant context
session_id: Session identifier
confirmation_id: Confirmation identifier from `confirmation_required` event
payload: Approval/denial decision
Returns:
StreamingResponse with SSE events (continuation of agent execution)
Raises:
NotFoundError: If session or confirmation not found
ValidationError: If confirmation already processed or expired
Example:
```bash
# Approve a pending action
curl -N -X POST http://localhost:8000/v1/agents/sessions/ses_abc/confirmations/conf_xyz \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{"approved": true}'
# SSE Output (continuation):
event: tool_result
data: {"tool_name": "delete_collection", "success": true, "result": {...}}
event: token
data: {"content": "I've deleted the collection as requested."}
event: done
data: {}
```
# Send Message
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/send-message
post /v1/agents/sessions/{session_id}/messages
Send a message to the agent and stream the response.
This endpoint streams Server-Sent Events (SSE) back to the client as
the agent processes the message through its workflow.
## SSE Event Types
**Core Events:**
- `intent`: Intent classification result (emitted first)
- `{intent, confidence, category, reasoning, context_scope}`
- `thinking`: Agent is analyzing/planning
- `{step, message}`
- `tool_call`: Agent is calling a tool
- `{tool_name, tool_call_id, inputs}`
- `tool_result`: Tool execution completed
- `{tool_name, tool_call_id, success, output, latency_ms}`
- `token`: Response token (streaming)
- `{content}`
- `message`: Final response content
- `{content, message_id, is_final}`
- `session_name`: Auto-generated session name (first message only)
- `{session_name}`
- `done`: Processing complete
- `{latency_ms, tool_calls_made, message_id, retriever_summary, data_accessed_via_retriever}`
- `error`: Error occurred
- `{message, recoverable}`
**Retriever Events (IMPORTANT - Primary Data Pathway):**
- `retriever_execution`: Retriever was used for data access
- `{tool_name, execution_id, retriever_id, is_adhoc, documents_returned, latency_ms, message}`
- Emitted whenever data is accessed via retriever (saved or ad-hoc)
- `pipeline_config`: Ad-hoc retriever configuration
- `{tool_name, config, message}`
- Contains the exact pipeline config users can save as a named retriever
**Retriever Summary in `done` Event:**
```json
{
"retriever_summary": {
"used_retrievers": true,
"retriever_count": 2,
"saved_retrievers": 1,
"adhoc_retrievers": 1,
"total_documents": 25,
"executions": [...]
},
"data_accessed_via_retriever": true
}
```
Args:
request: FastAPI request with tenant context
session_id: Session identifier
payload: Message request
Returns:
StreamingResponse with SSE events
Raises:
NotFoundError: If session not found
Example:
```bash
curl -N -X POST http://localhost:8000/v1/agents/sessions/ses_abc123/messages \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{
"content": "Find videos about machine learning",
"stream": true
}'
# SSE Output:
event: intent
data: {"intent": "retriever_search", "confidence": 0.92, "category": "retriever"}
event: thinking
data: {"step": "processing", "message": "Analyzing your request..."}
event: tool_call
data: {"tool_name": "execute_retriever", "tool_call_id": "run_abc", "inputs": {...}}
event: tool_result
data: {"tool_name": "execute_retriever", "success": true, "output": {...}}
event: retriever_execution
data: {"tool_name": "execute_retriever", "is_adhoc": false, "documents_returned": 5}
event: message
data: {"content": "I found 5 videos about machine learning...", "is_final": true}
event: done
data: {"latency_ms": 1250.5, "data_accessed_via_retriever": true, "retriever_summary": {...}}
```
# Submit Feedback
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/submit-feedback
post /v1/agents/sessions/{session_id}/feedback
Submit feedback on an assistant message.
When positive feedback is received, the conversation exchange is stored
to memory for future context. When negative feedback is received, the
exchange is NOT stored. This enables learning from quality interactions.
Args:
request: FastAPI request with tenant context
session_id: Session identifier
payload: Feedback request
Returns:
SubmitFeedbackResponse with feedback status
Raises:
NotFoundError: If session or message not found
Example:
```bash
curl -X POST http://localhost:8000/v1/agents/sessions/ses_abc123/feedback \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}" \
-H "Content-Type: application/json" \
-d '{
"message_id": "msg_xyz789",
"rating": "positive",
"feedback_text": "Very helpful response!"
}'
```
# Terminate Session
Source: https://docs.mixpeek.com/docs/api-reference/agent-sessions/terminate-session
delete /v1/agents/sessions/{session_id}
Terminate a session and kill its actor.
This permanently ends the session and releases all associated resources.
Args:
request: FastAPI request with tenant context
session_id: Session identifier
Returns:
TerminateSessionResponse with termination timestamp
Raises:
NotFoundError: If session not found
Example:
```bash
curl -X DELETE http://localhost:8000/v1/agents/sessions/ses_abc123 \
-H "Authorization: Bearer {api_key}" \
-H "X-Namespace: {namespace_id}"
```
# Analyze Bottlenecks
Source: https://docs.mixpeek.com/docs/api-reference/analytics/analyze-bottlenecks
get /v1/analytics/performance/engine/bottlenecks
Identify performance bottlenecks.
Analyzes profiling data to identify the biggest bottlenecks,
ranked by total time spent.
**Use Cases:**
- Find what's slowing down your pipelines
- Prioritize optimization efforts
- Monitor bottleneck trends
**Ranking:**
- Sorted by total time spent (sum across all executions)
- Shows percentage of total execution time
- Includes execution count and average time
**Example:**
```bash
GET /v1/analytics/performance/engine/bottlenecks?hours=24&limit=10
```
# Analyze For Tuning
Source: https://docs.mixpeek.com/docs/api-reference/analytics/analyze-for-tuning
post /v1/analytics/retrievers/{retriever_id}/analyze-tuning
Analyze retriever and generate tuning recommendations.
Performs comprehensive analysis and generates actionable recommendations:
- Parameter tuning suggestions
- Cache optimization opportunities
- Performance improvement estimates
**Recommendations Include:**
- Increase/decrease k value
- Adjust reranking thresholds
- Enable/optimize caching
- Stage reordering suggestions
**Use Cases:**
- Initial retriever configuration
- Periodic performance optimization
- A/B testing parameter changes
- Cost optimization
**Example:**
```bash
curl -X POST "https://api.mixpeek.com/v1/analytics/retrievers/ret_abc123/analyze-tuning?days=7" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Assignment Metrics
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-assignment-metrics
get /v1/analytics/taxonomies/{taxonomy_id}/assignments
Get taxonomy assignment metrics over time.
Tracks assignment counts including:
- Assignment volume over time
- Average confidence scores
- Unique labels assigned
**Use Cases:**
- Monitor taxonomy usage
- Track assignment trends
- Identify popular labels
# Get Batch Diagnostics
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-batch-diagnostics
get /v1/analytics/performance/batches/{batch_id}/diagnostics
Get comprehensive diagnostics for a batch.
Combines batch status, task progress, collection info, performance metrics,
and actionable insights into a single response for easy frontend rendering.
**Use Cases:**
- Monitor batch processing in real-time
- Debug failed batches
- View performance breakdown after completion
- Get actionable next steps
**Response includes:**
- Overall batch status and progress
- Per-tier task details with Ray job links
- Collection document counts
- Performance insights and bottlenecks (if completed)
- Error details (if failed)
- Recommended next actions
**Example:**
```bash
GET /v1/analytics/performance/batches/{batch_id}/diagnostics
```
**Perfect for:**
- Real-time progress tracking UI
- Batch monitoring dashboards
- Debugging failed extractions
- Performance optimization
# Get Bucket Health
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-bucket-health
get /v1/analytics/buckets/{bucket_id}/health
Get bucket health monitoring metrics.
Analyzes bucket health including:
- Error breakdown by type
- Sync health per configuration
- Stuck/failing syncs
- Overall health status
**Use Cases:**
- Monitor bucket health
- Identify failing syncs
- Debug errors by type
- Alert on unhealthy buckets
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/buckets/bkt_abc123/health?hours=24" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Bucket Storage
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-bucket-storage
get /v1/analytics/buckets/{bucket_id}/storage
Get storage growth trends over time.
Analyzes bucket storage metrics including:
- Total storage size over time
- Object count trends
- Growth rates
**Use Cases:**
- Monitor storage capacity planning
- Identify storage growth patterns
- Track object accumulation
- Alert on unexpected growth
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/buckets/bkt_abc123/storage?group_by=day" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Bucket Usage
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-bucket-usage
get /v1/analytics/buckets/{bucket_id}/usage
Get usage and cost metrics.
Analyzes bucket usage and costs including:
- Storage costs (GB-hours)
- Upload operation costs
- Sync operation costs
- Cost breakdown by category
**Use Cases:**
- Track bucket costs
- Optimize spending
- Forecast future costs
- Cost attribution
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/buckets/bkt_abc123/usage?group_by=day" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Cache Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-cache-performance
get /v1/analytics/retrievers/{retriever_id}/cache-performance
Get cache performance metrics.
Analyzes cache effectiveness including:
- Hit/miss rates
- Latency comparison (cache vs full search)
- Hourly cache performance trends
**Use Cases:**
- Evaluate cache effectiveness
- Optimize cache TTL settings
- Monitor cache performance
- Identify cache warming opportunities
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/retrievers/ret_abc123/cache-performance?hours=168" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Collection Overview
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-collection-overview
get /v1/analytics/collections/{collection_id}/overview
Get high-level collection health and status metrics.
Provides collection health including:
- Total document count and recent growth
- Processing performance and success rates
- Active enrichments (taxonomies, clusters)
**Use Cases:**
- Monitor collection health
- Quick status check
- Identify collections needing attention
# Get Compound Index Patterns
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-compound-index-patterns
get /v1/analytics/namespaces/indexes/compound-patterns
Identify compound index opportunities.
Finds metadata fields commonly used together in filters, suggesting
opportunities for compound (multi-field) indexes.
**Use Cases:**
- Optimize multi-field queries
- Create compound indexes
- Understand query complexity
- Improve complex filter performance
**Response Includes:**
- Field combinations used together
- Frequency of combination usage
- Average and P95 latency
- Sorted by combination frequency
**Compound Index Example:**
If `brand + status` appears frequently, create:
```javascript
db.documents.createIndex({"metadata.brand": 1, "metadata.status": 1})
```
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/namespaces/indexes/compound-patterns" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Confidence Distribution
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-confidence-distribution
get /v1/analytics/taxonomies/{taxonomy_id}/confidence
Get confidence score distribution.
Analyzes assignment confidence including:
- Distribution across confidence ranges
- Low-confidence assignment alerts
- Average confidence trends
**Use Cases:**
- Monitor taxonomy quality
- Identify low-confidence assignments
- Track confidence improvements
# Get Document Growth
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-document-growth
get /v1/analytics/collections/{collection_id}/growth
Get document growth trends over time.
Tracks document additions over time, useful for understanding
batch processing patterns and indexing velocity.
# Get Engine Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-engine-performance
get /v1/analytics/performance/engine
Get engine performance metrics over time.
Query profiling data logged by the engine's profiling infrastructure.
All queries are automatically filtered to your namespace.
**Time Range:**
- Specify `hours` for recent history (e.g., `hours=24` for last 24 hours)
- OR specify `start_date` and `end_date` for custom range
- Defaults to last 24 hours if neither provided
**Grouping:**
- `minute`: High-resolution (for short time ranges)
- `hour`: Standard resolution (default)
- `day`: For longer time ranges
- `week`, `month`: For historical trends
**Response:**
- Time-series metrics (avg, p50, p95, p99 latencies)
- Summary statistics across the entire time range
- All latencies in milliseconds
**Example:**
```bash
GET /v1/analytics/performance/engine?hours=24&group_by=hour
```
# Get Engine Stage Breakdown
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-engine-stage-breakdown
get /v1/analytics/performance/engine/stages
Get stage-level performance breakdown.
Breaks down engine performance by individual profiled stages
(e.g., pipeline_run, generate_input_dataset, gcs_batch_upload).
**Use Cases:**
- Identify which stages are slowest
- See percentage of total time per stage
- Optimize specific bottlenecks
**Response:**
- Per-stage metrics (count, avg, p95, max latencies)
- Total time spent in each stage
- Percentage of total execution time
- Sorted by total time (worst bottlenecks first)
**Example:**
```bash
GET /v1/analytics/performance/engine/stages?hours=24
```
# Get Enrichment History
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-enrichment-history
get /v1/analytics/taxonomies/{taxonomy_id}/enrichments
Get enrichment execution history.
Tracks enrichment operations including:
- Enrichment success rates
- Average latency
- Volume trends
**Use Cases:**
- Monitor enrichment performance
- Track success rates
- Identify performance issues
# Get Execution History
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-execution-history
get /v1/analytics/clusters/{cluster_id}/execution-history
Get cluster execution history with metrics.
Provides execution timeline including:
- Execution duration and document counts
- Algorithm and cluster counts
- Success/failure status
**Use Cases:**
- Monitor clustering performance
- Track execution patterns
- Identify execution issues
# Get Extractor Breakdown
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-extractor-breakdown
get /v1/analytics/performance/engine/extractors
Get extractor performance breakdown.
Shows performance metrics for each extractor and its stages.
**Use Cases:**
- Compare performance across extractors
- Identify slow extractor stages
- Monitor specific extractor performance
**Response:**
- Per-extractor, per-stage metrics
- Execution counts
- Latency statistics (avg, p95, max)
**Example:**
```bash
GET /v1/analytics/performance/engine/extractors?hours=24
```
# Get Extractor Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-extractor-performance
get /v1/analytics/extractors/performance
Get feature extraction performance metrics.
Returns per-extractor execution metrics from the ``extraction_events``
ClickHouse table over the last ``hours`` (namespace-scoped), one row per
(extractor_name, version):
- Total executions per extractor
- Average duration + P95/P99 latencies
- Success/failure counts and success rate
Returns an empty array when no extraction has run in the window or analytics
is unavailable.
**Example:**
```bash
GET /v1/analytics/extractors/performance?hours=24
GET /v1/analytics/extractors/performance?extractor_name=text_extractor&hours=168
```
# Get Extractor Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-extractor-performance-1
get /v1/analytics/collections/{collection_id}/extractors
Get feature extractor performance breakdown.
Analyzes extractor execution metrics including:
- Execution counts and success/failure rates
- Latency percentiles (P95, P99)
- Total processing time
**Use Cases:**
- Identify slow extractors for optimization
- Monitor extraction success rates
- Understanding processing bottlenecks
# Get Failure Analysis
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-failure-analysis
get /v1/analytics/collections/{collection_id}/failures
Get failure analysis metrics.
Debug processing failures including:
- Error distribution by type
- Recent error messages
- Failure patterns
**Use Cases:**
- Debug processing failures
- Identify common error patterns
- Track error trends
# Get Failure Analysis
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-failure-analysis-1
get /v1/analytics/clusters/{cluster_id}/failures
Get cluster failure analysis.
Analyzes clustering failures including:
- Error messages and types
- Failure timestamps
- Failure patterns
**Use Cases:**
- Debug clustering failures
- Identify common error patterns
- Monitor cluster health
# Get Field Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-field-performance
get /v1/analytics/namespaces/fields/performance
Analyze field performance correlation.
Shows which metadata fields correlate with slow queries, helping identify
fields that would benefit most from indexing.
**Use Cases:**
- Identify fields causing performance issues
- Quantify indexing impact potential
- Prioritize index creation
- Monitor field usage patterns
**Response Includes:**
- Field usage count
- Latency statistics (avg, P50, P95, P99, max)
- Index priority score (usage × latency)
- Sorted by priority score
**Index Priority Score:**
Higher scores indicate fields where indexing would have greatest impact.
Score = (query count) × (average latency)
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/namespaces/fields/performance?days=30" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Field Usage
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-field-usage
get /v1/analytics/indexes/usage
Get usage statistics for all filtered fields.
# Get Index Recommendations
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-index-recommendations
get /v1/analytics/namespaces/indexes/recommendations
Get comprehensive MongoDB index recommendations.
Analyzes query patterns and generates prioritized index recommendations
with ready-to-use MongoDB commands.
**Use Cases:**
- Get actionable index suggestions
- Prioritize database optimization
- Copy/paste index creation commands
- Track optimization opportunities
**Recommendation Levels:**
- **HIGH PRIORITY**: >100 queries, >300ms avg OR >10 very slow queries
- **MEDIUM PRIORITY**: >50 queries, >200ms avg OR >20 slow queries
- **LOW PRIORITY**: >10 queries but acceptable performance
- **NO ACTION**: Low usage, no optimization needed
**Response Includes:**
- Prioritized recommendations
- Usage and latency statistics
- MongoDB index creation commands
- Summary by priority level
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/namespaces/indexes/recommendations" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Index Suggestions
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-index-suggestions
get /v1/analytics/indexes/suggestions
Get index suggestions based on filter usage patterns.
# Get Inference Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-inference-performance
get /v1/analytics/inference/performance
Get inference performance metrics.
TODO: Implement inference performance query logic.
# Get Label Distribution
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-label-distribution
get /v1/analytics/taxonomies/{taxonomy_id}/labels
Get label distribution.
Analyzes label usage including:
- Top labels by assignment count
- Label popularity percentages
- Confidence by label
**Use Cases:**
- Understand label usage
- Identify most common categories
- Balance taxonomy coverage
# Get Latency Metrics
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-latency-metrics
get /v1/analytics/collections/{collection_id}/latency
Get processing latency distribution.
Analyzes document processing latency including:
- Latency percentiles over time
- Slowest document operations
- Performance trends
# Get Most Queried Fields
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-most-queried-fields
get /v1/analytics/namespaces/fields/most-queried
Get most frequently queried metadata fields.
Identifies which metadata fields are accessed most often in retriever queries,
helping prioritize index creation and understand query patterns.
**Use Cases:**
- Identify fields that need indexing
- Understand common query patterns
- Prioritize optimization efforts
- Plan database schema improvements
**Response Includes:**
- Field name and usage frequency
- Average and P95 latency metrics
- Total unique fields analyzed
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/namespaces/fields/most-queried?days=30&limit=20" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Namespace Summary
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-namespace-summary
get /v1/analytics/namespaces/summary
Get comprehensive namespace optimization summary.
Provides a complete overview of namespace performance including:
- Top index recommendations
- Most queried fields
- Slowest fields
- Compound index opportunities
- Summary statistics
**Use Cases:**
- Get full optimization picture
- Regular performance reviews
- Database health checks
- Planning optimization work
**Response Includes:**
- Summary statistics (field counts, priority levels)
- Top 10 index recommendations
- Top 10 most queried fields
- Top 10 slowest fields
- Top 10 compound index opportunities
**Recommended Workflow:**
1. Call this endpoint for overview
2. Use specific endpoints for details
3. Implement high-priority recommendations
4. Monitor improvement over time
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/namespaces/summary?days=30" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Retriever Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-retriever-performance
get /v1/analytics/retrievers/{retriever_id}/performance
Get retriever performance metrics for tuning.
Retrieves time-series performance data including:
- Query latency (P50, P95, P99)
- Query counts
- Result counts
- Latency trends
**Use Cases:**
- Monitor retriever performance over time
- Identify performance degradations
- Compare performance across time periods
- Establish performance baselines
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/retrievers/ret_abc123/performance?hours=24&group_by=hour" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Retriever Signals
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-retriever-signals
get /v1/analytics/retrievers/{retriever_id}/signals
Get retriever signals for interaction tuning.
Retrieves fine-grained signals about retriever behavior:
- Cache hits/misses
- Reranking scores
- Filter effectiveness
- Query expansion results
**Signal Types:**
- `cache_hit`: Successful cache lookups
- `cache_miss`: Cache misses requiring full search
- `rerank_scores`: Reranking effectiveness metrics
- `filter_reduction`: Pre-filter document reduction
- `expansion_results`: Query expansion impact
**Use Cases:**
- Fine-tune retrieval parameters
- Analyze query patterns
- Optimize cache strategies
- Validate optimization improvements
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/retrievers/ret_abc123/signals?signal_type=rerank_scores&limit=50" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Slow Queries
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-slow-queries
get /v1/analytics/namespaces/queries/slow
Get slow queries and their filter patterns.
Identifies queries exceeding a latency threshold and shows which metadata
fields they're filtering on, helping pinpoint optimization opportunities.
**Use Cases:**
- Troubleshoot slow queries
- Identify unindexed fields causing slowdowns
- Debug performance issues
- Optimize query patterns
**Response Includes:**
- Query details (retriever, inputs, latency)
- Results count
- Metadata fields being filtered
- Full query context
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/namespaces/queries/slow?latency_threshold_ms=1000" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Slowest Operations
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-slowest-operations
get /v1/analytics/performance/engine/slow-operations
Get slowest individual operations.
Returns the slowest profiled operations, useful for debugging
specific slow executions.
**Use Cases:**
- Troubleshoot specific slow operations
- Identify outliers
- Deep dive into problematic executions
**Filtering:**
- `threshold_ms`: Only show operations slower than this
- Default: 1000ms (1 second)
**Response:**
- Timestamp of slow operation
- Stage name and component
- Latency in milliseconds
- Full metadata context
**Example:**
```bash
GET /v1/analytics/performance/engine/slow-operations?hours=24&threshold_ms=5000
```
# Get Stage Breakdown
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-stage-breakdown
get /v1/analytics/retrievers/{retriever_id}/stages
Get stage-level performance breakdown.
Analyzes individual stage performance to identify bottlenecks:
- Stage execution times
- Document flow (in/out)
- Stage-level latency distribution
**Use Cases:**
- Identify slow stages in retrieval pipeline
- Optimize stage ordering
- Debug pipeline bottlenecks
- Understand document reduction rates
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/retrievers/ret_abc123/stages?hours=24" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Sync Comparison
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-sync-comparison
get /v1/analytics/buckets/{bucket_id}/sync-comparison
Compare performance across sync configurations.
Compares sync configurations by:
- Average duration and throughput
- Success rates
- Total files and bytes synced
- Provider performance
**Use Cases:**
- Compare sync providers (S3 vs GCS)
- Optimize sync configurations
- Identify best-performing syncs
- Benchmark sync strategies
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/buckets/bkt_abc123/sync-comparison?hours=168" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Sync Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-sync-performance
get /v1/analytics/buckets/{bucket_id}/sync-performance
Get sync performance metrics.
Analyzes sync job execution including:
- Files synced/failed
- Sync duration and throughput
- Success rates by provider
**Use Cases:**
- Monitor sync reliability
- Compare sync configurations
- Identify slow syncs
- Debug sync failures
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/buckets/bkt_abc123/sync-performance?hours=168" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Upload Performance
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-upload-performance
get /v1/analytics/buckets/{bucket_id}/upload-performance
Get upload performance metrics.
Analyzes upload operations including:
- Upload latency (P50, P95, P99)
- Throughput (MB/s)
- Error rates
**Use Cases:**
- Monitor upload performance
- Identify performance degradations
- Optimize upload strategies
- Debug upload issues
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/buckets/bkt_abc123/upload-performance?hours=24" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# Get Usage Summary
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-usage-summary
get /v1/analytics/usage/summary
Get usage summary for billing.
Aggregates the organization's usage_records — the SAME billing source of
truth that feeds daily_burn_rate and monthly invoicing (TG-3003: this was
a stub returning ``usage=[]``/``total_cost=0.0``, so Studio's costs view
showed $0 while the burn meter recorded millions of credits/day).
Returns per-operation credit totals for the window with USD derived at the
canonical $0.001/credit rate (CREDIT_RATE_USD):
- ``usage``: one row per operation_type — ``{operation_type, credits,
cost_usd}`` — biggest spend first. Refund records are negative and net
automatically (SP-187).
- ``total_credits`` / ``total_cost``: window totals across operations.
Scoped to the ORGANIZATION (matching the billing meter). Records are only
partially namespace-attributed (async engine work carries no namespace),
so filtering by namespace would silently under-report; ``namespace_id`` is
echoed for request context only.
**Time Range:**
- If both `start_date` and `end_date` are provided, uses that range
- If neither provided, defaults to last 30 days
- If only one provided, defaults to now as the other bound
**Example:**
```bash
GET /v1/analytics/usage/summary
GET /v1/analytics/usage/summary?start_date=2025-01-01T00:00:00Z&end_date=2025-01-31T23:59:59Z
```
# Annotation Stats
Source: https://docs.mixpeek.com/docs/api-reference/annotations/annotation-stats
get /v1/annotations/stats
Aggregate annotation counts grouped by label.
# Bulk Annotations
Source: https://docs.mixpeek.com/docs/api-reference/annotations/bulk-annotations
post /v1/annotations/bulk
Create, update, and/or delete annotations in a single call.
Each operation is independent — a failure in one does not roll back the others.
Maximum 1000 operations per type (creates, updates, deletes).
# Create Annotation
Source: https://docs.mixpeek.com/docs/api-reference/annotations/create-annotation
post /v1/annotations
Record a human decision on a document.
# Delete Annotation
Source: https://docs.mixpeek.com/docs/api-reference/annotations/delete-annotation
delete /v1/annotations/{annotation_id}
Delete an annotation.
# Get Annotation
Source: https://docs.mixpeek.com/docs/api-reference/annotations/get-annotation
get /v1/annotations/{annotation_id}
Get a single annotation by ID.
# List Annotations
Source: https://docs.mixpeek.com/docs/api-reference/annotations/list-annotations
post /v1/annotations/list
Query annotations with optional filters.
# Patch Annotation
Source: https://docs.mixpeek.com/docs/api-reference/annotations/update-annotation
patch /v1/annotations/{annotation_id}
Update an existing annotation (label, confidence, reasoning, payload).
# Get All Discovery Information
Source: https://docs.mixpeek.com/docs/api-reference/discovery/get-all-discovery-information
get /v1/discovery
Returns combined discovery information including extractors, stages, and manifest schema in a single request. Use this for comprehensive capability discovery.
# Get Manifest Schema for Agent Configuration
Source: https://docs.mixpeek.com/docs/api-reference/discovery/get-manifest-schema-for-agent-configuration
get /v1/discovery/schema
Returns the complete manifest schema including all resource types, their JSON schemas, dependency graph, and YAML examples. Use this for programmatic manifest generation and validation.
# List Available Feature Extractors
Source: https://docs.mixpeek.com/docs/api-reference/discovery/list-available-feature-extractors
get /v1/discovery/extractors
Discover all available feature extractors with their capabilities, supported modalities, output features, and example usage. Use this to understand what extractors are available when configuring namespaces and collections in manifests.
# List Available Retriever Stages with Examples
Source: https://docs.mixpeek.com/docs/api-reference/discovery/list-available-retriever-stages-with-examples
get /v1/discovery/stages
Discover all available retriever stages with extended information including example configurations, common use cases, and cost tiers. This endpoint provides more context than /v1/retrievers/stages for agent-driven configuration.
# Apply Manifest
Source: https://docs.mixpeek.com/docs/api-reference/manifest/apply-manifest
post /v1/manifest/apply
Apply a YAML manifest to create resources.
Creates all resources defined in the manifest file in dependency order.
Fails if any resource already exists (create-only mode).
Performs automatic rollback if any resource creation fails.
**Features:**
- Topological sorting ensures resources are created in correct dependency order
- Secret references (`${{ secrets.NAME }}`) are resolved from organization secrets
- Atomic operation: rolls back all created resources if any creation fails
- Dry run mode validates the manifest without making changes
**Example:**
```bash
curl -X POST /v1/manifest/apply \
-H "Authorization: Bearer $API_KEY" \
-H "X-Namespace: ns_xxx" \
-F "manifest_file=@mixpeek.yaml"
```
**Example manifest:**
```yaml
version: "1.0"
metadata:
name: "my-environment"
namespaces:
- name: video_search
feature_extractors:
- name: multimodal_extractor
version: v1
buckets:
- name: raw_videos
namespace: video_search
schema:
properties:
video: { type: video }
```
# Diff Manifest
Source: https://docs.mixpeek.com/docs/api-reference/manifest/diff-manifest
post /v1/manifest/diff
Compare a manifest file with current state.
Shows resources that would be:
- **Created**: In manifest but not in system
- **In system only**: In system but not in manifest
- **Different**: In both but with configuration differences
This is useful for understanding what changes would occur
before applying a manifest.
**Example:**
```bash
curl -X POST /v1/manifest/diff \
-H "Authorization: Bearer $API_KEY" \
-F "manifest_file=@mixpeek.yaml"
```
# Export Manifest Get
Source: https://docs.mixpeek.com/docs/api-reference/manifest/export-manifest
get /v1/manifest/export
Export current resources to a YAML manifest.
Exports all resources (or a specific namespace) to a YAML file
that can be version-controlled and re-applied to another environment.
**Features:**
- Resource IDs are converted to human-readable names
- Secret values are replaced with placeholder references (`${{ secrets.NAME }}`)
- Output is formatted for readability and git-friendliness
**Note:** You must configure actual secrets before applying the exported
manifest to a new environment.
**Example:**
```bash
# Export all resources
curl /v1/manifest/export \
-H "Authorization: Bearer $API_KEY" \
-o mixpeek.yaml
# Export specific namespace
curl "/v1/manifest/export?namespace_id=ns_abc123" \
-H "Authorization: Bearer $API_KEY" \
-o namespace.yaml
```
# Generate Manifest
Source: https://docs.mixpeek.com/docs/api-reference/manifest/generate-manifest
post /v1/manifest/generate
Generate a manifest from natural language description.
Uses AI to create a valid YAML manifest from a natural language description
of desired resources. The generated manifest can be reviewed and applied.
**Example:**
```bash
curl -X POST /v1/manifest/generate \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"description": "I need a bucket of images that feeds into a multimodal extractor and a retriever",
"manifest_name": "image-search-setup"
}'
```
**Response:**
```json
{
"manifest": "version: '1.0'
metadata:
name: image-search-setup
...",
"format": "yaml",
"manifest_name": "image-search-setup",
"description": "I need a bucket of images..."
}
```
**Next Steps:**
1. Review the generated manifest
2. Apply it using POST /v1/manifest/apply
# Lint Manifest
Source: https://docs.mixpeek.com/docs/api-reference/manifest/lint-manifest
post /v1/manifest/lint
Lint a YAML manifest for best practices and potential issues.
Goes beyond basic validation to provide actionable suggestions for
improving your manifest configuration. This endpoint is designed for
AI agents and developers who want to optimize their Mixpeek setup.
**Lint Rules:**
- `UNUSED_EXTRACTOR`: Feature extractor defined but not used by any collection
- `UNUSED_COLLECTION`: Collection not referenced by any retriever
- `MISSING_INPUT_SCHEMA`: Retriever uses templates but has no input_schema
- `MISSING_CACHE_CONFIG`: Retriever without caching (especially with LLM stages)
- `SUBOPTIMAL_STAGE_ORDER`: Filter stages after expensive operations
- `DUPLICATE_FEATURE_URI`: Same feature searched multiple times
- `MISSING_DESCRIPTION`: Resources without descriptions
- `NO_SEARCH_STAGE`: Retriever with no search stages
- `EXTRACTOR_NOT_IN_NAMESPACE`: Collection uses extractor not in namespace
- `MISSING_SECRET`: Secret reference not configured
**Severity Levels:**
- `error`: Must be fixed before applying
- `warning`: Best practice violation, should be fixed
- `info`: Suggestion for improvement
**Example:**
```bash
curl -X POST /v1/manifest/lint \
-H "Authorization: Bearer $API_KEY" \
-F "manifest_file=@mixpeek.yaml"
```
**Response includes actionable suggestions:**
```json
{
"valid": true,
"results": [
{
"code": "MISSING_CACHE_CONFIG",
"severity": "warning",
"message": "Retriever 'product_search' has no cache configuration",
"location": "retrievers[0]",
"suggestion": "Add cache_config to improve performance",
"fix_example": "cache_config:\n enabled: true\n ttl_seconds: 3600"
}
],
"summary": {"error": 0, "warning": 1, "info": 0}
}
```
# Validate Manifest
Source: https://docs.mixpeek.com/docs/api-reference/manifest/validate-manifest
post /v1/manifest/validate
Validate a YAML manifest without applying.
Checks:
- YAML syntax validity
- Schema validation against manifest models
- Cross-resource reference validation
- Dependency resolution (no circular dependencies)
- Secret reference existence
Returns detailed validation results including:
- Resource counts by type
- Missing secrets that need to be configured
- Validation errors and warnings
**Example:**
```bash
curl -X POST /v1/manifest/validate \
-H "Authorization: Bearer $API_KEY" \
-F "manifest_file=@mixpeek.yaml"
```
# Get App featured gallery
Source: https://docs.mixpeek.com/docs/api-reference/public-apps-api/get-app-gallery
get /v1/public/apps/{slug}/gallery
Fetch featured gallery results for a public App — no authentication required.
Executes the app's `featured_gallery` retriever using `default_inputs`
defined in the gallery configuration. Returns an empty result set if the
gallery is disabled or not configured.
# Get taxonomy node tree for an App
Source: https://docs.mixpeek.com/docs/api-reference/public-apps-api/get-app-taxonomies
get /v1/public/apps/{slug}/taxonomies/{taxonomy_id}/tree
Fetch the node tree for a taxonomy associated with a public App — no authentication required.
Looks up the taxonomy by ``taxonomy_id`` within the namespace that owns
the app identified by ``slug``. Returns the full ``nodes`` array as stored,
which can be used to render faceted navigation or classification trees in
the app UI.
# Track an App interaction event
Source: https://docs.mixpeek.com/docs/api-reference/public-apps-api/track-app-interaction
post /v1/public/apps/{slug}/interactions
Record a user interaction event for a public App — no authentication required.
Interaction events are written to the ``app_interactions`` collection for
analytics and relevance tuning. Write errors are swallowed server-side so
that a transient storage failure never surfaces as an error to the caller.
**Example:**
```json
{
"event_type": "click",
"document_id": "doc_abc123",
"position": 2,
"query": "red sneakers",
"session_id": "sess_xyz"
}
```
# Get Public Namespace Template
Source: https://docs.mixpeek.com/docs/api-reference/public-templates-api/get-public-namespace-template
get /v1/public/templates/namespaces/{template_id}
Get public namespace template details (no authentication required).
Returns template only if marked as public.
# Get Public Retriever Template
Source: https://docs.mixpeek.com/docs/api-reference/public-templates-api/get-public-retriever-template
get /v1/public/templates/retrievers/{template_id}
Get public retriever template details (no authentication required).
Returns template only if marked as public.
# Get Scaffold
Source: https://docs.mixpeek.com/docs/api-reference/public-templates-api/get-scaffold
get /v1/public/templates/scaffolds/{template_id}
Get scaffold template details (public, no auth required).
Returns the complete configuration including:
- Namespace: feature extractors, payload indexes
- Bucket: name, description, schema
- Collection: feature extractor config
- Retriever: stages, input schema
**To instantiate (requires auth):**
```
POST /v1/templates/scaffolds/{template_id}/instantiate
{"namespace_name": "my_app"}
```
# List Public Namespace Templates
Source: https://docs.mixpeek.com/docs/api-reference/public-templates-api/list-public-namespace-templates
get /v1/public/templates/namespaces
List public namespace templates (no authentication required).
Returns only templates marked as public (is_public=True).
# List Public Retriever Templates
Source: https://docs.mixpeek.com/docs/api-reference/public-templates-api/list-public-retriever-templates
get /v1/public/templates/retrievers
List public retriever templates (no authentication required).
Returns only templates marked as public (is_public=True).
# List Scaffolds
Source: https://docs.mixpeek.com/docs/api-reference/public-templates-api/list-scaffolds
get /v1/public/templates/scaffolds
List available scaffold templates (public, no auth required).
Scaffolds are pre-configured templates that create complete infrastructure:
- Namespace with feature extractors
- Bucket with data schema
- Collection with processing config
- Retriever with search pipeline
**Categories:**
- `media` - Video, image, podcast search
- `documents` - Document Q&A, RAG
- `ecommerce` - Product catalog search
**To instantiate (requires auth):**
```
POST /v1/templates/scaffolds/{template_id}/instantiate
{"namespace_name": "my_app"}
```
# Create Bucket Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/create-bucket-template
post /v1/templates/buckets/from-bucket/{bucket_id}
Create template from existing bucket.
Supports three template scopes:
- **organization**: Available to all users in your organization (default)
- **user**: Available only to you
- **system**: Available to all organizations (requires Mixpeek admin email)
# Create Cluster Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/create-cluster-template
post /v1/templates/clusters/from-cluster/{cluster_id}
Create template from existing cluster.
# Create Collection Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/create-collection-template
post /v1/templates/collections/from-collection/{collection_id}
Create template from existing collection.
Supports three template scopes:
- **organization**: Available to all users in your organization (default)
- **user**: Available only to you
- **system**: Available to all organizations (requires Mixpeek admin email)
# Create Namespace Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/create-namespace-template
post /v1/templates/namespaces/from-namespace/{namespace_id}
Create template from existing namespace.
Supports three template scopes:
- **organization**: Available to all users in your organization (default)
- **user**: Available only to you
- **system**: Available to all organizations (requires Mixpeek admin email)
# Create Retriever Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/create-retriever-template
post /v1/templates/retrievers/from-retriever/{retriever_id}
Create template from existing retriever.
Supports three template scopes:
- **organization**: Available to all users in your organization (default)
- **user**: Available only to you
- **system**: Available to all organizations (requires Mixpeek admin email)
# Create Taxonomy Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/create-taxonomy-template
post /v1/templates/taxonomies/from-taxonomy/{taxonomy_id}
Create template from existing taxonomy.
Supports three template scopes:
- **organization**: Available to all users in your organization (default)
- **user**: Available only to you
- **system**: Available to all organizations (requires Mixpeek admin email)
# Get Bucket Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/get-bucket-template
get /v1/templates/buckets/{template_id}
Get bucket template details.
# Get Cluster Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/get-cluster-template
get /v1/templates/clusters/{template_id}
Get cluster template details.
# Get Collection Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/get-collection-template
get /v1/templates/collections/{template_id}
Get collection template details.
# Get Namespace Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/get-namespace-template
get /v1/templates/namespaces/{template_id}
Get namespace template details.
# Get Retriever Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/get-retriever-template
get /v1/templates/retrievers/{template_id}
Get retriever template details.
# Get Taxonomy Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/get-taxonomy-template
get /v1/templates/taxonomies/{template_id}
Get taxonomy template details.
# Instantiate Bucket Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/instantiate-bucket-template
post /v1/templates/buckets/{template_id}/instantiate
Instantiate bucket template.
# Instantiate Cluster Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/instantiate-cluster-template
post /v1/templates/clusters/{template_id}/instantiate
Instantiate cluster template.
# Instantiate Collection Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/instantiate-collection-template
post /v1/templates/collections/{template_id}/instantiate
Instantiate collection template.
# Instantiate Namespace Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/instantiate-namespace-template
post /v1/templates/namespaces/{template_id}/instantiate
Instantiate namespace template.
# Instantiate Retriever Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/instantiate-retriever-template
post /v1/templates/retrievers/{template_id}/instantiate
Instantiate retriever template.
# Instantiate Taxonomy Template
Source: https://docs.mixpeek.com/docs/api-reference/templates/instantiate-taxonomy-template
post /v1/templates/taxonomies/{template_id}/instantiate
Instantiate taxonomy template.
# List Bucket Templates
Source: https://docs.mixpeek.com/docs/api-reference/templates/list-bucket-templates
post /v1/templates/buckets
List bucket templates (system + organization + user).
Supports filtering, sorting, and search like other list operations.
# List Cluster Templates
Source: https://docs.mixpeek.com/docs/api-reference/templates/list-cluster-templates
post /v1/templates/clusters
List cluster templates (system + organization + user).
Supports filtering, sorting, and search like other list operations.
# List Collection Templates
Source: https://docs.mixpeek.com/docs/api-reference/templates/list-collection-templates
post /v1/templates/collections
List collection templates (system + organization + user).
Supports filtering, sorting, and search like other list operations.
# List Namespace Templates
Source: https://docs.mixpeek.com/docs/api-reference/templates/list-namespace-templates
get /v1/templates/namespaces
List namespace templates (system + organization + user).
# List Retriever Templates
Source: https://docs.mixpeek.com/docs/api-reference/templates/list-retriever-templates
post /v1/templates/retrievers
List retriever templates (system + organization + user).
Supports filtering, sorting, and search like other list operations.
**Request Body (optional):**
- `filters`: Attribute-based filters `{"AND": [{"field": "category", "operator": "eq", "value": "semantic_search"}]}`
- `sort`: Sort options `{"field": "name", "direction": "asc"}`
- `search`: Wildcard search across template_id, name, description, tags
- `scope`: Filter by scope (system, organization, user)
- `category`: Filter by category
- `is_active`: Show only active templates (default: true)
- `tags`: Filter by tags (templates must have ALL specified tags)
# List Taxonomy Templates
Source: https://docs.mixpeek.com/docs/api-reference/templates/list-taxonomy-templates
post /v1/templates/taxonomies
List taxonomy templates (system + organization + user).
Supports filtering, sorting, and search like other list operations.
# Get Slowest Queries
Source: https://docs.mixpeek.com/docs/api-reference/analytics/get-slowest-queries
get /v1/analytics/retrievers/{retriever_id}/slow-queries
Get slowest queries for troubleshooting.
Identifies slowest-performing queries for optimization:
- Query text
- Execution time
- Result counts
- Stage breakdown
**Use Cases:**
- Identify problematic queries
- Debug performance issues
- Optimize query patterns
- User experience improvements
**Example:**
```bash
curl -X GET "https://api.mixpeek.com/v1/analytics/retrievers/ret_abc123/slow-queries?limit=10&hours=24" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Namespace: your-namespace"
```
# List change feed events
Source: https://docs.mixpeek.com/docs/api-reference/events/list-events
get /v1/events
List this organization's change feed, ordered and resumable by cursor.
A consumer can be killed mid-stream and resume from its last stored
`next_cursor` without re-reading or missing anything, **as long as it
resumes within the 90-day retention window** — events older than that
are permanently expired, not archived. A consumer that has been down
longer than 90 days must resync current state instead of resuming.
A cursor is only meaningful for a FIXED filter set: it encodes a
position in this organization's overall sequence, not a position
within any particular `namespace_id`/`event_type` filter. Changing
either filter mid-stream while reusing an old cursor silently skips
whatever the previous filter combination would have matched in
between — start a fresh cursor (or none) whenever the filters change.
# Execute Raw Inference
Source: https://docs.mixpeek.com/docs/api-reference/inference/execute-raw-inference
post /v1/inference
Execute raw inference with provider+model or custom plugin.
This endpoint provides direct access to inference services without
the retriever framework overhead. Supports two modes:
1. **Provider + Model**: Use standard providers (openai, google, anthropic)
2. **Custom Plugin**: Use your custom inference plugins by inference_name
## Supported Providers
- **openai**: GPT models, embeddings, Whisper transcription
- **google**: Gemini models, Vertex multimodal embeddings (1408D)
- **anthropic**: Claude models
## Examples
### Custom Plugin (by inference_name)
```json
{
"inference_name": "my_text_embedder_1_0_0",
"inputs": {"text": "hello world"},
"parameters": {}
}
```
### Custom Plugin (by feature_uri)
```json
{
"feature_uri": "mixpeek://my_custom_embedder@1.0.0/embedding",
"inputs": {"text": "hello world"},
"parameters": {}
}
```
### Builtin Embedder (by feature_uri)
```json
{
"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
"inputs": {"text": "hello world"},
"parameters": {}
}
```
### Chat Completion
```json
{
"provider": "openai",
"model": "gpt-4o-mini",
"inputs": {"prompts": ["What is AI?"]},
"parameters": {"temperature": 0.7, "max_tokens": 500}
}
```
### Text Embedding (OpenAI)
```json
{
"provider": "openai",
"model": "text-embedding-3-large",
"inputs": {"text": "machine learning"},
"parameters": {}
}
```
### Text Embedding (Google Vertex Multimodal - 1408D)
```json
{
"provider": "google",
"model": "multimodalembedding",
"inputs": {"text": "machine learning"},
"parameters": {}
}
```
### Image Embedding (Google Vertex Multimodal - 1408D)
```json
{
"provider": "google",
"model": "multimodalembedding",
"inputs": {"image_url": "https://example.com/image.jpg"},
"parameters": {}
}
```
### Image Embedding from Base64
```json
{
"provider": "google",
"model": "multimodalembedding",
"inputs": {"image_base64": "<base64-encoded-image>"},
"parameters": {}
}
```
### Video Embedding (Google Vertex Multimodal - 1408D)
```json
{
"provider": "google",
"model": "multimodalembedding",
"inputs": {"video_url": "https://example.com/video.mp4"},
"parameters": {}
}
```
### Video Embedding from Base64
```json
{
"provider": "google",
"model": "multimodalembedding",
"inputs": {"video_base64": "<base64-encoded-video>"},
"parameters": {}
}
```
### Audio Transcription
```json
{
"provider": "openai",
"model": "whisper-1",
"inputs": {"audio_url": "https://example.com/audio.mp3"},
"parameters": {}
}
```
### Vision (Multimodal LLM)
```json
{
"provider": "openai",
"model": "gpt-4o",
"inputs": {
"prompts": ["Describe this image"],
"image_url": "https://example.com/image.jpg"
},
"parameters": {"temperature": 0.5}
}
```
Args:
request: FastAPI request object (populated by middleware)
payload: Raw inference request
Returns:
Inference response with results and metadata
Raises:
400 Bad Request: Invalid provider, model, or inputs
401 Unauthorized: Missing or invalid API key
429 Too Many Requests: Rate limit exceeded
500 Internal Server Error: Inference execution failed
# Advance Funnel Stage
Source: https://docs.mixpeek.com/docs/api-reference/notifications/advance-funnel-stage
post /v1/notifications/funnel/advance
Advance the funnel stage from the client side.
Studio calls this when key milestones happen in-browser that the server
cannot observe directly (e.g. first_search executed in the test tab).
Accepts both resource-creation stages and activation funnel stages.
# Delete Notification
Source: https://docs.mixpeek.com/docs/api-reference/notifications/delete-notification
delete /v1/notifications/{notification_id}
Delete a notification (idempotent - succeeds even if already deleted).
# Get Funnel State
Source: https://docs.mixpeek.com/docs/api-reference/notifications/get-funnel-state
get /v1/notifications/funnel/state
Get the current funnel state for the organization.
Returns the user's position in the onboarding funnel, including:
- Current stage
- When each stage was reached
- Nudges that have been sent
- Last activity timestamp
# Get Notification
Source: https://docs.mixpeek.com/docs/api-reference/notifications/get-notification
get /v1/notifications/{notification_id}
Get a single notification by its ID.
# Get Preferences
Source: https://docs.mixpeek.com/docs/api-reference/notifications/get-preferences
get /v1/notifications/preferences
Get notification preferences for the organization.
# Get Recent Email Sends
Source: https://docs.mixpeek.com/docs/api-reference/notifications/get-recent-email-sends
get /v1/notifications/emails/recent
Get recent emails sent via Resend with delivery status.
Returns a list of sent emails enriched with Resend delivery events
(delivered, bounced, complained, etc.) for visibility into the email pipeline.
# Get Reminder Preferences
Source: https://docs.mixpeek.com/docs/api-reference/notifications/get-reminder-preferences
get /v1/notifications/preferences/reminders
Get reminder/nudge email preferences for the organization.
These preferences control funnel-based onboarding nudges,
re-engagement emails, and feature tips.
# Get Unread Count
Source: https://docs.mixpeek.com/docs/api-reference/notifications/get-unread-count
get /v1/notifications/unread/count
Get count of unread notifications.
# List Notifications
Source: https://docs.mixpeek.com/docs/api-reference/notifications/list-notifications
post /v1/notifications/list
List all notifications for the user's organization.
# Mark All As Read
Source: https://docs.mixpeek.com/docs/api-reference/notifications/mark-all-as-read
post /v1/notifications/read/all
Mark all notifications as read for a user.
# Mark As Read
Source: https://docs.mixpeek.com/docs/api-reference/notifications/mark-as-read
post /v1/notifications/{notification_id}/read
Mark a notification as read.
# Update Preferences
Source: https://docs.mixpeek.com/docs/api-reference/notifications/update-preferences
put /v1/notifications/preferences
Update notification preferences for the organization.
# Update Reminder Preferences
Source: https://docs.mixpeek.com/docs/api-reference/notifications/update-reminder-preferences
put /v1/notifications/preferences/reminders
Update reminder/nudge email preferences for the organization.
You can update individual preferences without affecting others:
- enabled: Master toggle for all reminder emails
- onboarding_nudges: Funnel progression nudges
- engagement_reminders: Re-engagement emails for inactivity
- feature_tips: Helpful tips and inspiration
- quiet_hours_start/end: Hours (0-23 UTC) when no emails are sent
# Get the org's signup-time suggested namespace
Source: https://docs.mixpeek.com/docs/api-reference/onboarding/get-suggested-config
get /v1/onboarding/suggested-config
Returns the namespace configuration that was suggested from Mixpeek's company research at signup, if one was provisioned. Studio uses this to badge the suggested namespace.
# Infer onboarding configuration
Source: https://docs.mixpeek.com/docs/api-reference/onboarding/infer-onboarding-configuration
post /v1/onboarding/infer-config
Given a free-text use-case description, uses an LLM to recommend the best scaffold template, extractors, retriever stages, and sample queries for the user's project.
# Get Pricing Configuration
Source: https://docs.mixpeek.com/docs/api-reference/pricing/get-pricing-configuration
get /v1/billing/pricing
Public pricing rate card. Price = how much content of each file type × (base rate + the search-by features enabled). Returns per-modality rates, tier usage pools, and plan definitions. No authentication required.
# Search Resources
Source: https://docs.mixpeek.com/docs/api-reference/resource-search/search-resources
post /v1/resources/search
Search across all resource names and IDs within your namespace.
This endpoint performs a case-insensitive search across:
- Buckets (bucket_name, bucket_id)
- Collections (collection_name, collection_id)
- Retrievers (retriever_name, retriever_id)
- Taxonomies (taxonomy_name, taxonomy_id)
- Clusters (cluster_name, cluster_id)
- Namespaces (namespace_name, namespace_id)
Results are sorted by relevance (exact matches first) and creation time (newest first).
Use the resource_types parameter to filter searches to specific resource types.
Pagination is supported via limit and offset parameters.
# Get Task Information
Source: https://docs.mixpeek.com/docs/api-reference/tasks/get-task-information
get /v1/tasks/{task_id}
Retrieve a task by its ID.
A task may have an expiration time, after which it will still be returned but marked as expired.
This allows tracking of historical tasks while indicating their current validity state.
# Kill Task
Source: https://docs.mixpeek.com/docs/api-reference/tasks/kill-task
delete /v1/tasks/{task_id}
Kill a task (idempotent - succeeds even if task doesn't exist or is already killed).
# List Tasks
Source: https://docs.mixpeek.com/docs/api-reference/tasks/list-tasks
post /v1/tasks/list
List tasks with optional filtering, sorting, and pagination.
**Filter Options**:
- `status`: Filter by specific status (PENDING, IN_PROGRESS, COMPLETED, FAILED, etc.)
- `task_type`: Filter by task type
**Examples**:
- All tasks: `{}`
- Failed tasks only: `{"status": "FAILED"}`
- Pending batches: `{"status": "PENDING", "task_type": "API_BUCKETS_UPLOADS_BATCH_CONFIRM"}`
- In-progress tasks: `{"status": "IN_PROGRESS"}`