> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Autonomous Namespaces
> Describe your data and get a namespace manifest, improve it with a manifest loop scored on held-out data, and bring your own code as plugin images
Describe your data and what you want to find, and Mixpeek writes the namespace for you. An agent, the manifest loop, can then try other models, settings and retrievers, score each try on labelled examples, and keep what works. When the catalog is not enough, you add your own extractor code as a container image.
```mermaid theme={null}
flowchart LR
A["Describe your data
and your goal"] --> B["POST /v1/manifest/generate
a starting manifest"]
B --> C["Manifest loop
tries changes, scores
each on tuning labels"]
C --> D["Finalists scored once
on held-out labels"]
D --> E["Apply the winner
POST /v1/manifest/apply"]
P["Plugin images
your own extractor code"] -.-> C
```
**1. Generate a manifest**
```bash theme={null}
curl -X POST https://api.mixpeek.com/v1/manifest/generate \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "Content-Type: application/json" \
-d '{"description": "Hiking clips in an S3 bucket. Find dead logs and dry brush in each frame, and search the frames by text.", "manifest_name": "hikes"}'
```
**2. Check it and apply it**
Read `validation`. When it says `valid: true`, send `manifest` to `POST /v1/manifest/apply`.
**3. Improve it with the manifest loop**
Give the loop your data and a few labelled questions. It returns up to three finalists, each scored on examples it never saw.
## Generate a manifest
`POST /v1/manifest/generate` takes a plain-language `description` and returns a complete manifest. The model that writes it reads the live extractor and stage catalogs. The answer goes through the same checks as `POST /v1/manifest/validate`. When those checks find errors, the model gets them back and fixes its answer, up to two times.
**manifest** `string`
The manifest as YAML. Apply it as it is, or edit it first.
**validation** `object`
`valid`, `errors` and `warnings`. When `valid` is false, the errors stay listed so you can fix them by hand.
**repair\_rounds** `integer`
How many times the model fixed its own answer.
**cost\_estimate** `object`
The manifest priced at list rates: per 1,000 items, and per month.
**suggested** `object`
The templates and catalog models closest to your description: `{"templates": [...], "models": [...]}`.
**quote** `object`
Only when you send `inventory` (files per bucket): what indexing those files costs, the same quote `POST /v1/organizations/billing/estimate` returns.
For every manifest field, see the [manifest reference](https://docs.mixpeek.com/docs/operations/manifests.md).
## Improve it with the manifest loop
The loop takes a starting manifest and tries changes. It scores each change on labelled questions and keeps the best. An AI agent, the driver, decides what to try. It writes down what it expects before it sees each score.
> [!NOTE]
>
> The loop runs from a script in the Mixpeek repository (`server/scripts/manifest_loop`). There is no hosted endpoint for it. Each change it tries is an ordinary manifest applied through the API, so a finalist deploys like any other manifest.
### What the driver can change
```mermaid theme={null}
flowchart TB
subgraph cheap ["Seconds, free"]
R1["Search weights and fusion"]
R2["How many results to keep"]
R3["Group, de-duplicate, threshold"]
end
subgraph costly ["Minutes per clip: re-process the data"]
X1["A different catalog model"]
X2["Extractor settings,
for example tiles of each frame"]
X3["A second model on the first
one's output (a tier)"]
X4["Your own code as a plugin image"]
end
cheap --> T["Try these first"]
costly --> U["Spend the extraction budget
only when a retriever change
cannot test the idea"]
```
A retriever change re-ranks what is already indexed, so it costs seconds. A model or extractor change re-processes the data. The loop gives those a fixed budget of extraction minutes per run.
### Why you can trust the score
```mermaid theme={null}
flowchart LR
L["All labelled examples"] --> T["Tuning half
the driver scores on this"]
L --> H["Held-out half
locked away"]
T --> P["Driver picks
up to 3 finalists"]
P --> S["Finalists rebuilt and
scored once on held-out"]
H --> S
```
The examples are split by source file, so near-identical frames of one video never land on both sides. Every reported number is a held-out score with a 95% bootstrap interval. A change that only wins on the tuning half usually loses on held-out.
### What it found
| Benchmark | Metric | generate | Manifest loop | Hand-built | What the loop found |
| - | - | - | - | - | - |
| Hiking video | fuel F2 | 0.591 | **0.686** | 0.670 | Tiles of each frame, ranked by the best tile |
| Hiking video, tiling turned off | fuel F2 | **0.571** | 0.518 | 0.665 | A CLIP model on the frames, which won only on one tuning clip |
| Fashion search | F1 | 0.717 | **0.743** | 0.737 | A general model beat the fashion-specific one |
| Product photos | F1 | **0.629** | 0.624 | 0.382 | Nothing beat the starting manifest |
With every tool available, the loop matched the best hand-built manifest on these benchmarks. Its lead over generate was within the noise of the held-out intervals. With tiling turned off, the loop picked a model that won on one tuning clip, and it scored below generate on held-out.
### Run it
```bash Start theme={null}
cd server
export MLOOP_TASK=path/to/task.json # data source, labelled questions, budget
.venv/bin/python -m scripts.manifest_loop init
.venv/bin/python -m scripts.manifest_loop status
```
```bash Try changes theme={null}
PY=.venv/bin/python
$PY -m scripts.manifest_loop log "Try CLIP B/16 on the frames; expect better dry brush"
$PY -m scripts.manifest_loop apply-variant clip --manifest clip.yaml
$PY -m scripts.manifest_loop retriever-variant fused --variant clip \
--collections frames,frames-clip --stages @fused.stages.json
$PY -m scripts.manifest_loop evaluate fused
```
```bash Finish theme={null}
$PY -m scripts.manifest_loop finalists fused,clip.frame-search --why "..."
$PY -m scripts.manifest_loop teardown --keep clip
```
| Command | What it does | Cost |
| - | - | - |
| `log "..."` | Record what you will try and why. Required before every result. | free |
| `apply-variant NAME --manifest FILE` | Build a scratch namespace from a manifest and process the tuning data | extraction minutes |
| `apply-variant V --manifest FILE --into V` | Add a tier to an existing variant | extraction minutes |
| `retriever-variant NAME --variant V ...` | A new retriever over existing collections | seconds |
| `plugin-variant NAME --dir DIR` | Build your plugin folder into an image, apply it and score it | extraction minutes |
| `evaluate RETRIEVER` | Score a retriever on the tuning labels | seconds |
| `finalists A,B,C --why "..."` | Pick up to three for held-out scoring | free |
The operator runs held-out scoring (`heldout.py`) and the data split (`prepare.py`). The driver never sees the held-out labels.
### Iterate with your own agent
You can run the same loop without the script. Any coding agent that can call the API (Claude Code, Codex, Cursor) can drive it. It needs three things from you:
* **Labelled questions.** Each question names the files or documents that should come back for it. More questions give a narrower interval. With the fashion benchmark's 41 questions, the loop's 0.026 lead over generate stayed inside the noise.
* **A held-out split.** Split by source file before the agent starts, so near-identical frames of one video never land on both sides. Keep the held-out labels away from the agent until the end.
* **A budget.** Extraction minutes for model and extractor changes. Retriever changes cost seconds.
**Mixpeek resources the agent uses**
| Step | Resource |
| - | - |
| Starting manifest | `POST /v1/manifest/generate` |
| Build a variant | `POST /v1/manifest/apply`, one namespace per variant |
| Labelled questions | `POST /v1/retrievers/evaluations/datasets` (each query has a `query_id`, `query_input` and `relevant_documents`) |
| Labels from real usage | `POST /v1/retrievers/{retriever_id}/evaluations/generate-from-interactions` |
| Score a retriever | `POST /v1/retrievers/{retriever_id}/evaluations`: precision, recall, `f1`, `f2`, `map`, `ndcg` and `mrr` at each of `k_values`, or at a multiple of each query's relevant count with `k_multiples_of_relevant` |
| Score your own rankings | `POST /v1/evaluations/score`: precision, recall, `f1` and `f2`, with an optional cutoff `k`, per item and averaged |
| List the evaluation endpoints | `GET /v1/evaluations` |
Pick the metric for the cost of a miss. `f2` weighs recall above precision, so use it when missing a relevant document costs more than an extra one, as with the fire-fuel classes above. Use `ndcg` or `mrr` when order matters most.
**Check your setup on a public benchmark first.** Text-to-image retrieval on COCO 2017 (the 5,000-image Karpathy test split) or Flickr30k (the 1,000-image Karpathy test split) has published recall\@1 numbers to compare against. CLIP ViT-B/16 at 224 px is published at 33.1 on COCO and 62.1 on Flickr30k. A setup that lands near those numbers loses nothing in indexing or retrieval, so any gap on your own data comes from the manifest.
**A prompt to start the agent**
```text theme={null}
You are improving a Mixpeek namespace for this use case: .
Data: . Labelled questions: , each with a
question and the files that should be returned. Held-out labels exist, but you will not see them.
Use the Mixpeek API at https://api.mixpeek.com with the key in $MIXPEEK_API_KEY.
1. Generate a starting manifest with POST /v1/manifest/generate from the use-case description, and
apply it to a new namespace with POST /v1/manifest/apply. This is the baseline.
2. Upload the tuning questions as an evaluation dataset (POST /v1/retrievers/evaluations/datasets)
and score the baseline retriever (POST /v1/retrievers/{retriever_id}/evaluations).
Primary metric: , at k = .
3. Try changes, cheapest first: retriever weights, fusion, limits, thresholds and grouping; then
query templates; then a second model fused with the first; then a different catalog model or
extractor settings. Build each model or extractor change as its own namespace from a manifest.
Stay under extraction minutes.
4. Before each score, write down in log.md what you are trying and what you expect. Record the score
after it. Do not change a variant after seeing its score and then report it as untouched.
5. Stop when three changes in a row fail to beat the best variant by more than the noise, or when the
budget is spent. Pick at most three finalists and say why.
6. Hand me the finalists' manifests and the log. I will score them once on the held-out labels with a
95% bootstrap interval over queries, and compare each against the baseline as a paired difference.
7. Delete every scratch namespace you created except the finalists.
```
The held-out score is the one to trust. A change that wins only on the tuning half usually loses on held-out, as the hiking run with tiling turned off showed.
## Models, plugins and templates
```mermaid theme={null}
flowchart LR
M1["Model
weights, by reference"] --> P1["Plugin
an image: code and packages,
plus the models it declares"]
M2["Model"] --> P1
P1 --> T["Template
a manifest that composes
plugins into collections
and retrievers"]
P2["Built-in plugin
for example image_extractor"] --> T
```
* A **model** is weights. You name it by reference: `hf://org/name@revision`, `mixpeek://...` or `catalog://`.
* A **plugin** is a container image with your code and packages, plus the models it declares. An extractor is the kind of plugin that fills a collection. The built-in extractors are plugins of the same shape.
* A **template** is a manifest that composes plugins into collections and retrievers.
## Plugins as images
### Where they run
| Deployment | Approved registry extractors | Your own plugin images |
| - | - | - |
| Shared cloud (`api.mixpeek.com`) | Yes | No, refused by name |
| Enterprise single-tenant deployment | Yes | Yes, when turned on |
| Standalone container | Yes | Yes, when turned on |
```mermaid theme={null}
flowchart LR
subgraph shared ["Shared cloud"]
S1["Approved registry only"]
end
subgraph tenant ["Your single-tenant deployment"]
T1["Your own project and cluster"] --> T2["Your plugin images"]
end
subgraph local ["Standalone"]
L1["Stack container"] --- L2["Plugin worker container
runs your image"]
end
```
A single-tenant deployment has its own project, cluster, storage and credentials. Your code there can reach only your data. Custom images are off by default everywhere. Set `CUSTOM_PLUGIN_IMAGES_ENABLED=1` on a deployment that may run them. Apply checks this, and the engine checks it again when it starts each batch job.
> [!NOTE]
>
> Plugin images on an Enterprise single-tenant deployment are set up with your account team. The batch path there runs your image as the job's image. The end-to-end runs so far are on the standalone container.
### Declare it in a manifest
```yaml theme={null}
extractors:
- name: plugin_tiles
namespace: my-namespace
source:
image: registry.example.com/team/plugin-tiles@sha256:<64 hex digest>
serve:
- model: "hf://google/siglip-base-patch16-224@7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed"
min_replicas: 0
max_replicas: 2
```
**source.image** `string` *required*
The image, pinned by digest: `/@sha256:<64 hex>`. A tag is refused at apply and again at job start, because a tag can point to new code later.
**serve** `list`
The models your plugin uses. Each gets its own deployment for this extractor alone, with its own `min_replicas` and `max_replicas`. No other extractor calls it.
**serve\[].min\_reason** `string`
Required when `min_replicas` is above 0. Say why the model stays warm when idle.
Every declared model needs a catalog record whose licence allows commercial use. A model with no catalog record is refused, because there is no licence on record to check. Pick models from `GET /v1/discovery/models`.
### Scale to zero and cold starts
A model at `min_replicas: 0` costs nothing when idle and wakes on the next request.
* **Batch:** the job waits for the model to wake.
* **Search:** a query that reaches a sleeping model waits for it, up to 45 seconds by default. The first query after an idle spell is slow; the next ones are fast.
* **Never wait:** set `min_replicas: 1` with a `min_reason`. That keeps one replica running, which you pay for while idle.
### Build the image
A plugin builds `FROM` the stock image of the deployment it runs on. Your code goes where the engine finds extractors. The image holds code and packages; models come from `serve`.
```dockerfile theme={null}
FROM mixpeek/standalone:
ENV ENGINE_LOCAL_PLUGINS=1
COPY . /app/engine/extractors/_local/plugin_tiles/v1/
```
The worked example below runs image\_extractor's pipeline with a fixed 2x2 grid of tiles, embedding through its own SigLIP deployment. It ran end to end on the standalone container.
> [tab] definition.py
>
> ```python theme={null}
> from engine.extractors.image_extractor.v1.definition import (
> VECTOR_INDEXES,
> )
> from engine.extractors.image_extractor.v1.definition import (
> definition as image_definition,
> )
> from shared.collection.features.extractors.models import ComputeProfile
>
> NAME = "plugin_tiles"
> VERSION = "v1"
> # The reference the manifest lists under this extractor's serve entries.
> MODEL_REF = (
> "hf://google/siglip-base-patch16-224@7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed"
> )
>
> vector_index = VECTOR_INDEXES[0].model_copy(
> deep=True,
> update={
> "name": "siglip",
> "index": VECTOR_INDEXES[0].index.model_copy(
> update={
> "name": f"{NAME}_{VERSION}_embedding",
> "inference_service_id": MODEL_REF,
> "inference_name": MODEL_REF,
> }
> ),
> },
> )
>
> definition = image_definition.model_copy(
> deep=True,
> update={
> "feature_extractor_name": NAME,
> "version": VERSION,
> "description": "image_extractor with every frame also indexed as 2x2 tiles.",
> "required_vector_indexes": [vector_index],
> "compute_profile": ComputeProfile(
> resource_type="cpu", serve_app=None, actor_memory_gb=2.0, batch_size=8
> ),
> "extractor_function_path": f"engine.extractors._local.{NAME}.{VERSION}.pipeline.extract",
> },
> )
>
> metadata = {
> "name": NAME,
> "version": VERSION,
> "description": definition.description,
> "category": "image",
> "icon": "image",
> }
> ```
> [tab] pipeline.py
>
> ```python theme={null}
> from engine.extractors.image_extractor.v1 import pipeline as base
> from engine.extractors.plugin_sdk import embedding_through
> from engine.pipelines.models import PipelineStepsAndPrepare
> from shared.collection.features.extractors.models import ExtractorDefinition
>
> from .definition import MODEL_REF, NAME, VERSION
>
> TILE_GRID = {"cols": 2, "rows": 2}
>
>
> def build_steps(
> extractor_request,
> container=None,
> base_steps=None,
> dataset_size=None,
> content_flags=None,
> ):
> # image_extractor reads the grid from the request's parameters.
> config = extractor_request.extractor_config
> config.parameters = {**(config.parameters or {}), "tile_grid": TILE_GRID}
> with embedding_through(MODEL_REF):
> return base.build_steps(
> extractor_request,
> container,
> base_steps,
> dataset_size,
> content_flags,
> )
>
>
> def extract(extractor_request, base_steps=None, dataset_size=None, content_flags=None):
> result = build_steps(
> extractor_request,
> base_steps=base_steps,
> dataset_size=dataset_size,
> content_flags=content_flags,
> )
> return PipelineStepsAndPrepare(steps=result["steps"], prepare=result["prepare"])
>
>
> extractor_definition = ExtractorDefinition(
> name=NAME,
> version=VERSION,
> extractor_function_path=extract,
> requires_file_resolution=True,
> )
> ```
> [tab] **init**.py
>
> ```python theme={null}
> from .definition import definition, metadata
>
> __all__ = ["definition", "metadata"]
> ```
`engine.extractors.plugin_sdk.model_app(ref)` returns the deployment name of a declared model, for code that calls it directly. Custom serving code inside the image is not supported yet. Your models are served by the platform's serving path, from the references you declare.
### Test it locally
The standalone container has no container runtime inside it. A plugin image runs as a second container that shares the stack's network and runs a small Ray of its own.
**1. Start the stack with custom images on**
```bash theme={null}
docker run -d --name mixpeek --stop-timeout 600 -p 8000:8000 \
-v mixpeek-data:/data -e CUSTOM_PLUGIN_IMAGES_ENABLED=1 \
mixpeek/standalone:
```
**2. Build the image and get its digest from a local registry**
```bash theme={null}
docker run -d --name registry -p 127.0.0.1:5000:5000 registry:2
docker build -t localhost:5000/my-tiles:dev .
docker push localhost:5000/my-tiles:dev
docker inspect --format '{{index .RepoDigests 0}}' localhost:5000/my-tiles:dev
```
**3. Start a plugin worker for that digest**
```bash theme={null}
IMAGE=localhost:5000/my-tiles@sha256:
docker inspect mixpeek --format '{{range .Config.Env}}{{println .}}{{end}}' > stack.env
docker run -d --name my-tiles-worker \
--network container:mixpeek --volumes-from mixpeek \
--memory 6g --cpus 4 --env-file stack.env \
-e MIXPEEK_PLUGIN_IMAGE=$IMAGE \
$IMAGE /app/standalone/entrypoint.sh plugin-worker
```
The worker listens on a port derived from the digest, so the engine finds it from the digest alone. A job refuses to run if the worker holds a different image.
**4. Apply a manifest that names the digest**
Put the digest in `source.image` and apply the manifest. Apply deploys your models, runs a short job in your image to read its definition, and creates the extractor. Batches on its collections then run in your image.
The manifest loop does steps 2 to 4 for you with `plugin-variant NAME --dir DIR`.
> [accordion] What a plugin gets in its environment
>
> No platform-wide credential is added to a plugin job. Provider keys, proxies and the engine's own tokens are removed. On a single-tenant deployment the pod's secret belongs to that tenant alone.
> [accordion] One image per batch job
>
> A batch job runs one image. Re-applying an extractor with a new digest replaces its old image.
> [accordion] Limits
>
> Models must have a catalog record with a commercial licence. Custom serving code in the image is not supported yet. The first build on a new machine copies the whole base image, which takes several minutes. Later builds take seconds.
> [card] [Manifest reference](https://docs.mixpeek.com/docs/operations/manifests.md)
>
> Every manifest field, validate, plan and apply.
> [card] [Custom extractors](https://docs.mixpeek.com/docs/processing/custom-extractors.md)
>
> Registry-backed extractors and submissions on the shared API.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.