Autonomous Namespaces

Describe your data and what you want to find, and Mixpeek writes the namespace for you. An agent, the manifest loop, can then try other models, settings and retrievers, score each try on labelled examples, and keep what works. When the catalog is not enough, you add your own extractor code as a container image.

Generate a manifest

curl -X POST https://api.mixpeek.com/v1/manifest/generate \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"description": "Hiking clips in an S3 bucket. Find dead logs and dry brush in each frame, and search the frames by text.", "manifest_name": "hikes"}'

Check it and apply it

Read validation. When it says valid: true, send manifest to POST /v1/manifest/apply.

Improve it with the manifest loop

Give the loop your data and a few labelled questions. It returns up to three finalists, each scored on examples it never saw.

Generate a manifest

POST /v1/manifest/generate takes a plain-language description and returns a complete manifest. The model that writes it reads the live extractor and stage catalogs. The answer goes through the same checks as POST /v1/manifest/validate. When those checks find errors, the model gets them back and fixes its answer, up to two times.

string

The manifest as YAML. Apply it as it is, or edit it first.

object

valid, errors and warnings. When valid is false, the errors stay listed so you can fix them by hand.

integer

How many times the model fixed its own answer.

object

The manifest priced at list rates: per 1,000 items, and per month.

object

The templates and catalog models closest to your description: {"templates": [...], "models": [...]}.

object

Only when you send inventory (files per bucket): what indexing those files costs, the same quote POST /v1/organizations/billing/estimate returns.

For every manifest field, see the manifest reference.

Improve it with the manifest loop

The loop takes a starting manifest and tries changes. It scores each change on labelled questions and keeps the best. An AI agent, the driver, decides what to try. It writes down what it expects before it sees each score.

The loop runs from a script in the Mixpeek repository (server/scripts/manifest_loop). There is no hosted endpoint for it. Each change it tries is an ordinary manifest applied through the API, so a finalist deploys like any other manifest.

What the driver can change

A retriever change re-ranks what is already indexed, so it costs seconds. A model or extractor change re-processes the data. The loop gives those a fixed budget of extraction minutes per run.

Why you can trust the score

The examples are split by source file, so near-identical frames of one video never land on both sides. Every reported number is a held-out score with a 95% bootstrap interval. A change that only wins on the tuning half usually loses on held-out.

What it found

Held-out scores of generate, the manifest loop and a hand-built manifest on four benchmarks

Benchmark	Metric	generate	Manifest loop	Hand-built	What the loop found
Hiking video	fuel F2	0.591	0.686	0.670	Tiles of each frame, ranked by the best tile
Hiking video, tiling turned off	fuel F2	0.571	0.518	0.665	A CLIP model on the frames, which won only on one tuning clip
Fashion search	F1	0.717	0.743	0.737	A general model beat the fashion-specific one
Product photos	F1	0.629	0.624	0.382	Nothing beat the starting manifest

With every tool available, the loop matched the best hand-built manifest on these benchmarks. Its lead over generate was within the noise of the held-out intervals. With tiling turned off, the loop picked a model that won on one tuning clip, and it scored below generate on held-out.

Run it

cd server
export MLOOP_TASK=path/to/task.json   # data source, labelled questions, budget
.venv/bin/python -m scripts.manifest_loop init
.venv/bin/python -m scripts.manifest_loop status

PY=.venv/bin/python
$PY -m scripts.manifest_loop log "Try CLIP B/16 on the frames; expect better dry brush"
$PY -m scripts.manifest_loop apply-variant clip --manifest clip.yaml
$PY -m scripts.manifest_loop retriever-variant fused --variant clip \
    --collections frames,frames-clip --stages @fused.stages.json
$PY -m scripts.manifest_loop evaluate fused

$PY -m scripts.manifest_loop finalists fused,clip.frame-search --why "..."
$PY -m scripts.manifest_loop teardown --keep clip

Command	What it does	Cost
`log "..."`	Record what you will try and why. Required before every result.	free
`apply-variant NAME --manifest FILE`	Build a scratch namespace from a manifest and process the tuning data	extraction minutes
`apply-variant V --manifest FILE --into V`	Add a tier to an existing variant	extraction minutes
`retriever-variant NAME --variant V ...`	A new retriever over existing collections	seconds
`plugin-variant NAME --dir DIR`	Build your plugin folder into an image, apply it and score it	extraction minutes
`evaluate RETRIEVER`	Score a retriever on the tuning labels	seconds
`finalists A,B,C --why "..."`	Pick up to three for held-out scoring	free

The operator runs held-out scoring (heldout.py) and the data split (prepare.py). The driver never sees the held-out labels.

Iterate with your own agent

You can run the same loop without the script. Any coding agent that can call the API (Claude Code, Codex, Cursor) can drive it. It needs three things from you:

Labelled questions. Each question names the files or documents that should come back for it. More questions give a narrower interval. With the fashion benchmark’s 41 questions, the loop’s 0.026 lead over generate stayed inside the noise.
A held-out split. Split by source file before the agent starts, so near-identical frames of one video never land on both sides. Keep the held-out labels away from the agent until the end.
A budget. Extraction minutes for model and extractor changes. Retriever changes cost seconds.

Mixpeek resources the agent uses

Step	Resource
Starting manifest	`POST /v1/manifest/generate`
Build a variant	`POST /v1/manifest/apply`, one namespace per variant
Labelled questions	`POST /v1/retrievers/evaluations/datasets` (each query has a `query_id`, `query_input` and `relevant_documents`)
Labels from real usage	`POST /v1/retrievers/{retriever_id}/evaluations/generate-from-interactions`
Score a retriever	`POST /v1/retrievers/{retriever_id}/evaluations`: precision, recall, `f1`, `f2`, `map`, `ndcg` and `mrr` at each of `k_values`, or at a multiple of each query’s relevant count with `k_multiples_of_relevant`
Score your own rankings	`POST /v1/evaluations/score`: precision, recall, `f1` and `f2`, with an optional cutoff `k`, per item and averaged
List the evaluation endpoints	`GET /v1/evaluations`

Pick the metric for the cost of a miss. f2 weighs recall above precision, so use it when missing a relevant document costs more than an extra one, as with the fire-fuel classes above. Use ndcg or mrr when order matters most. Check your setup on a public benchmark first. Text-to-image retrieval on COCO 2017 (the 5,000-image Karpathy test split) or Flickr30k (the 1,000-image Karpathy test split) has published recall@1 numbers to compare against. CLIP ViT-B/16 at 224 px is published at 33.1 on COCO and 62.1 on Flickr30k. A setup that lands near those numbers loses nothing in indexing or retrieval, so any gap on your own data comes from the manifest. A prompt to start the agent

You are improving a Mixpeek namespace for this use case: <describe the data and what people search for>.

Data: <bucket or storage connection>. Labelled questions: <path to tuning_labels.json>, each with a
question and the files that should be returned. Held-out labels exist, but you will not see them.

Use the Mixpeek API at https://api.mixpeek.com with the key in $MIXPEEK_API_KEY.

1. Generate a starting manifest with POST /v1/manifest/generate from the use-case description, and
   apply it to a new namespace with POST /v1/manifest/apply. This is the baseline.
2. Upload the tuning questions as an evaluation dataset (POST /v1/retrievers/evaluations/datasets)
   and score the baseline retriever (POST /v1/retrievers/{retriever_id}/evaluations).
   Primary metric: <f1 | f2 | ndcg>, at k = <k>.
3. Try changes, cheapest first: retriever weights, fusion, limits, thresholds and grouping; then
   query templates; then a second model fused with the first; then a different catalog model or
   extractor settings. Build each model or extractor change as its own namespace from a manifest.
   Stay under <N> extraction minutes.
4. Before each score, write down in log.md what you are trying and what you expect. Record the score
   after it. Do not change a variant after seeing its score and then report it as untouched.
5. Stop when three changes in a row fail to beat the best variant by more than the noise, or when the
   budget is spent. Pick at most three finalists and say why.
6. Hand me the finalists' manifests and the log. I will score them once on the held-out labels with a
   95% bootstrap interval over queries, and compare each against the baseline as a paired difference.
7. Delete every scratch namespace you created except the finalists.

The held-out score is the one to trust. A change that wins only on the tuning half usually loses on held-out, as the hiking run with tiling turned off showed.

Models, plugins and templates

A model is weights. You name it by reference: hf://org/name@revision, mixpeek://... or catalog://<id>.
A plugin is a container image with your code and packages, plus the models it declares. An extractor is the kind of plugin that fills a collection. The built-in extractors are plugins of the same shape.
A template is a manifest that composes plugins into collections and retrievers.

Plugins as images

Where they run

Deployment	Approved registry extractors	Your own plugin images
Shared cloud (`api.mixpeek.com`)	Yes	No, refused by name
Enterprise single-tenant deployment	Yes	Yes, when turned on
Standalone container	Yes	Yes, when turned on

A single-tenant deployment has its own project, cluster, storage and credentials. Your code there can reach only your data. Custom images are off by default everywhere. Set CUSTOM_PLUGIN_IMAGES_ENABLED=1 on a deployment that may run them. Apply checks this, and the engine checks it again when it starts each batch job.

Plugin images on an Enterprise single-tenant deployment are set up with your account team. The batch path there runs your image as the job’s image. The end-to-end runs so far are on the standalone container.

Declare it in a manifest

extractors:
  - name: plugin_tiles
    namespace: my-namespace
    source:
      image: registry.example.com/team/plugin-tiles@sha256:<64 hex digest>
    serve:
      - model: "hf://google/siglip-base-patch16-224@7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed"
        min_replicas: 0
        max_replicas: 2

string

required

The image, pinned by digest: <registry>/<name>@sha256:<64 hex>. A tag is refused at apply and again at job start, because a tag can point to new code later.

list

The models your plugin uses. Each gets its own deployment for this extractor alone, with its own min_replicas and max_replicas. No other extractor calls it.

string

Required when min_replicas is above 0. Say why the model stays warm when idle.

Every declared model needs a catalog record whose licence allows commercial use. A model with no catalog record is refused, because there is no licence on record to check. Pick models from GET /v1/discovery/models.

Scale to zero and cold starts

A model at min_replicas: 0 costs nothing when idle and wakes on the next request.

Batch: the job waits for the model to wake.
Search: a query that reaches a sleeping model waits for it, up to 45 seconds by default. The first query after an idle spell is slow; the next ones are fast.
Never wait: set min_replicas: 1 with a min_reason. That keeps one replica running, which you pay for while idle.

Build the image

A plugin builds FROM the stock image of the deployment it runs on. Your code goes where the engine finds extractors. The image holds code and packages; models come from serve.

FROM mixpeek/standalone:<tag>
ENV ENGINE_LOCAL_PLUGINS=1
COPY . /app/engine/extractors/_local/plugin_tiles/v1/

The worked example below runs image_extractor’s pipeline with a fixed 2x2 grid of tiles, embedding through its own SigLIP deployment. It ran end to end on the standalone container.

definition.py
pipeline.py
init.py

from engine.extractors.image_extractor.v1.definition import (
    VECTOR_INDEXES,
)
from engine.extractors.image_extractor.v1.definition import (
    definition as image_definition,
)
from shared.collection.features.extractors.models import ComputeProfile

NAME = "plugin_tiles"
VERSION = "v1"
# The reference the manifest lists under this extractor's serve entries.
MODEL_REF = (
    "hf://google/siglip-base-patch16-224@7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed"
)

vector_index = VECTOR_INDEXES[0].model_copy(
    deep=True,
    update={
        "name": "siglip",
        "index": VECTOR_INDEXES[0].index.model_copy(
            update={
                "name": f"{NAME}_{VERSION}_embedding",
                "inference_service_id": MODEL_REF,
                "inference_name": MODEL_REF,
            }
        ),
    },
)

definition = image_definition.model_copy(
    deep=True,
    update={
        "feature_extractor_name": NAME,
        "version": VERSION,
        "description": "image_extractor with every frame also indexed as 2x2 tiles.",
        "required_vector_indexes": [vector_index],
        "compute_profile": ComputeProfile(
            resource_type="cpu", serve_app=None, actor_memory_gb=2.0, batch_size=8
        ),
        "extractor_function_path": f"engine.extractors._local.{NAME}.{VERSION}.pipeline.extract",
    },
)

metadata = {
    "name": NAME,
    "version": VERSION,
    "description": definition.description,
    "category": "image",
    "icon": "image",
}

from engine.extractors.image_extractor.v1 import pipeline as base
from engine.extractors.plugin_sdk import embedding_through
from engine.pipelines.models import PipelineStepsAndPrepare
from shared.collection.features.extractors.models import ExtractorDefinition

from .definition import MODEL_REF, NAME, VERSION

TILE_GRID = {"cols": 2, "rows": 2}


def build_steps(
    extractor_request,
    container=None,
    base_steps=None,
    dataset_size=None,
    content_flags=None,
):
    # image_extractor reads the grid from the request's parameters.
    config = extractor_request.extractor_config
    config.parameters = {**(config.parameters or {}), "tile_grid": TILE_GRID}
    with embedding_through(MODEL_REF):
        return base.build_steps(
            extractor_request,
            container,
            base_steps,
            dataset_size,
            content_flags,
        )


def extract(extractor_request, base_steps=None, dataset_size=None, content_flags=None):
    result = build_steps(
        extractor_request,
        base_steps=base_steps,
        dataset_size=dataset_size,
        content_flags=content_flags,
    )
    return PipelineStepsAndPrepare(steps=result["steps"], prepare=result["prepare"])


extractor_definition = ExtractorDefinition(
    name=NAME,
    version=VERSION,
    extractor_function_path=extract,
    requires_file_resolution=True,
)

from .definition import definition, metadata

__all__ = ["definition", "metadata"]

engine.extractors.plugin_sdk.model_app(ref) returns the deployment name of a declared model, for code that calls it directly. Custom serving code inside the image is not supported yet. Your models are served by the platform’s serving path, from the references you declare.

Test it locally

The standalone container has no container runtime inside it. A plugin image runs as a second container that shares the stack’s network and runs a small Ray of its own.

Start the stack with custom images on

docker run -d --name mixpeek --stop-timeout 600 -p 8000:8000 \
  -v mixpeek-data:/data -e CUSTOM_PLUGIN_IMAGES_ENABLED=1 \
  mixpeek/standalone:<tag>

Build the image and get its digest from a local registry

docker run -d --name registry -p 127.0.0.1:5000:5000 registry:2
docker build -t localhost:5000/my-tiles:dev .
docker push localhost:5000/my-tiles:dev
docker inspect --format '{{index .RepoDigests 0}}' localhost:5000/my-tiles:dev

Start a plugin worker for that digest

IMAGE=localhost:5000/my-tiles@sha256:<digest>
docker inspect mixpeek --format '{{range .Config.Env}}{{println .}}{{end}}' > stack.env
docker run -d --name my-tiles-worker \
  --network container:mixpeek --volumes-from mixpeek \
  --memory 6g --cpus 4 --env-file stack.env \
  -e MIXPEEK_PLUGIN_IMAGE=$IMAGE \
  $IMAGE /app/standalone/entrypoint.sh plugin-worker

The worker listens on a port derived from the digest, so the engine finds it from the digest alone. A job refuses to run if the worker holds a different image.

Apply a manifest that names the digest

Put the digest in source.image and apply the manifest. Apply deploys your models, runs a short job in your image to read its definition, and creates the extractor. Batches on its collections then run in your image.

The manifest loop does steps 2 to 4 for you with plugin-variant NAME --dir DIR.

What a plugin gets in its environment

No platform-wide credential is added to a plugin job. Provider keys, proxies and the engine’s own tokens are removed. On a single-tenant deployment the pod’s secret belongs to that tenant alone.

One image per batch job

A batch job runs one image. Re-applying an extractor with a new digest replaces its old image.

Limits

Models must have a catalog record with a commercial licence. Custom serving code in the image is not supported yet. The first build on a new machine copies the whole base image, which takes several minutes. Later builds take seconds.

Manifest reference

Every manifest field, validate, plan and apply.

Custom extractors

Registry-backed extractors and submissions on the shared API.

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

Autonomous Namespaces

Generate a manifest

Improve it with the manifest loop

What the driver can change

Why you can trust the score

What it found

Run it

Iterate with your own agent

Models, plugins and templates

Plugins as images

Where they run

Declare it in a manifest

Scale to zero and cold starts

Build the image

Test it locally

Manifest reference

Custom extractors

​Generate a manifest

​Improve it with the manifest loop

​What the driver can change

​Why you can trust the score

​What it found

​Run it

​Iterate with your own agent

​Models, plugins and templates

​Plugins as images

​Where they run

​Declare it in a manifest

​Scale to zero and cold starts

​Build the image

​Test it locally

Manifest reference

Custom extractors

Generate a manifest

Improve it with the manifest loop

What the driver can change

Why you can trust the score

What it found

Run it

Iterate with your own agent

Models, plugins and templates

Plugins as images

Where they run

Declare it in a manifest

Scale to zero and cold starts

Build the image

Test it locally