The Short Answer
A managed search platform can let you run your own model in four distinct places, and they are not interchangeable:
At ingest time (custom extraction). Your code runs on every object as it is indexed, producing embeddings or structured attributes. This is the most common extension point and the easiest to support, because it is batch work with no latency budget.
At query time (custom inference). Your code runs inside the request path, usually to embed the incoming query with the same model you indexed with. This is the hardest to support well, because it puts your container on the critical path with a latency budget measured in milliseconds.
At rerank time. Your model reorders an already-retrieved candidate set. Latency-sensitive but bounded, because you see tens of candidates rather than the whole corpus.
At enrichment time. Your model runs after retrieval to annotate or transform results, off the critical path.
The distinction that actually matters: a platform that supports only ingest-time extraction cannot support a custom embedding model. If you embed documents with your fine-tuned encoder but the platform embeds queries with its own, your query vector and your document vectors live in different spaces and the similarity scores are meaningless. Custom embedding requires ingest and query time together, running the identical model artifact. That single requirement disqualifies most "bring your own model" claims.
Why This Is the Question People Actually Ask
Teams evaluating a managed platform almost always arrive with a model they already trust: a fine-tuned CLIP for their product photography, a domain encoder trained on their own legal corpus, a classifier that encodes a policy no general model knows. The question is never "can this platform embed things." It is "can it embed things *my way*, and what do I give up to get that."
The honest answer depends on architecture, not on a feature checkbox, so this guide is about what to look for.
The Four Extension Points, and What Each Costs the Platform
| Extension point | Runs when | Latency budget | What the platform must provide | Typical availability |
| Ingest-time extraction | During indexing | Minutes; batch | Scheduling, GPU allocation, retries, checkpointing | Common |
| Query-time inference | Inside the search request | Tens of milliseconds | Warm containers, autoscaling, cold-start avoidance | Rare |
| Reranking | After candidate retrieval | ~100ms on tens of items | Same as query-time, smaller batch | Uncommon |
| Post-retrieval enrichment | After results are returned | Seconds; async | Ordinary async execution | Common |
The Trap: Ingest and Query Must Load the Same Model
This is the failure that costs a full re-index, so it is worth being precise.
Retrieval works because the query vector and the document vectors are produced by the same function. Cosine similarity between a vector from model A and a vector from model B is a real number, and it means nothing. There is no error, no exception, no warning. Recall simply collapses to something slightly better than random, and because the system returns *plausible-looking* results ranked in a confident order, it can take a long time to notice.
So when you evaluate a platform's custom-model support, the question to ask is not "can I bring my own embedding model." It is: "when I issue a text query against an index built with my model, what embeds the query?" If the answer is anything other than "the same artifact you uploaded," custom embedding is not supported no matter what the marketing page says.
A platform that gets this right exposes the same extractor in both places, and the query-time path is an explicit stage in the retriever rather than an implicit default. See Multi-Stage Retrieval for how those stages compose, and Late Interaction Retrieval for what changes when your model emits many vectors per document instead of one.
The Packaging Contract
Every platform that runs your code defines a contract. The shape is fairly consistent across vendors:
The output schema is the part people underestimate. Your extractor usually produces more than one thing (an embedding, plus a few attributes you want to filter on), and each output needs a stable name because retrievers, filters, taxonomies and clusters all reference it by that name. Rename an output and you have silently broken every retriever pointing at it.
Versioning Against a Corpus That Already Exists
Once your extractor has written vectors, its version is part of your data, not just your code.
Ship v2 with a different architecture or dimension and you now have two incompatible vector populations in one index. Three ways out, in increasing order of cost:
1. Additive outputs. v2 emits a new named output alongside the old one. Nothing breaks, storage grows, and you cut over retrievers when you are ready. 2. Shadow index. Build the v2 space separately, evaluate it against the v1 space on real queries, then swap. This is the safe default and it is what evaluating multimodal retrieval is for. 3. Full re-extraction. Correct, and it means paying GPU cost for your entire corpus again. Budget it explicitly; see what it costs to make a media library searchable.
If the change is only the embedding model and not the upstream feature extraction, you may not need to re-run the expensive part at all. Migrating embedding models without re-embedding covers when that is possible.
Cold Starts Decide Whether Query-Time Inference Is Usable
A custom model in the query path is only as good as its warm-start behavior. A 2 GB checkpoint that loads in eight seconds is fine for batch and unusable for search, so the platform has to keep at least one replica resident. That has three consequences worth checking before you commit:
Ask what the p99 looks like immediately after a deploy, not in steady state. That is where the honest answer lives.
How This Works on Mixpeek
Mixpeek runs custom extractors on the same Ray cluster as the built-in ones. A single extractor archive supplies both sides: a batch entry point used during decomposition, and a
realtime.py that exposes an HTTP endpoint retriever stages call at query time. Because it is one artifact, the feature_search stage embeds the query with exactly the model that embedded the corpus, which is the trap described above closed by construction.Outputs land in the vector store and MongoDB under the same feature URI scheme as built-in pipelines (
mixpeek://[email protected]/my_embedding), so retrievers, taxonomies and clusters reference custom outputs the same way they reference native ones. Pricing derives from the compute profile your extractor declares, and you can quote it before running.The limitation, stated plainly: the upload, deploy and real-time inference lifecycle runs on dedicated infrastructure. It is *not* exposed on the shared public API, where those endpoints return 404 or 405 by design. On the shared API you can discover extractors, and you can author, lint and test locally with no API key. To get an extractor into production you either provision a dedicated deployment or submit it for review to be merged into the built-in catalog, which is the right path when the extractor is not proprietary.
What To Ask Before You Commit
1. When I query an index built with my model, what embeds the query? (The disqualifying question.) 2. Is query-time custom inference on the shared plan, or does it require dedicated infrastructure? 3. What happens to already-indexed vectors when I ship a new version of my extractor? 4. Can one extractor emit several named outputs, and can filters and taxonomies reference them? 5. What is p99 latency right after a deploy? 6. If I stop paying, can I get my vectors out, or only my source files?
Question 6 is the one people skip. Custom models make your index harder to reproduce elsewhere, because reproducing it requires both your model artifact and the platform's chunking and preprocessing behavior. Running the store on object storage you already own is the usual hedge.