> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# End-to-End Data Flow
> One object, from arrival to a moderation decision: what runs where, what it costs, and what leaves your account.
One object entering the system, through every layer, to a decision and a queryable index.
Written extractor-agnostic: the extractor is a variable, and this page shows where that
variable is resolved and what it changes.
The primary walk is customer-hosted (BYOC), because that is the shape where "what runs
where" matters. Mixpeek Cloud is the same pipeline with every plane in Mixpeek's account.
```mermaid theme={null}
flowchart TB
A["① ARRIVAL
presigned PUT or sync connector"] --> B["② OBJECT
bucket row + blobs + schema validation"]
B --> C["③ BATCH
work unit over N objects, deduped"]
C --> D["④ SCHEDULE
Celery → profile resolve → sizing"]
D --> E["⑤ EXECUTE
ephemeral RayJob, tier 0..N"]
E -->|"model calls"| S["Ray Serve
long-lived, autoscaled"]
E --> F["⑥ PERSIST
SyncVectorWriter → MVS → WAL → object storage"]
F --> G["⑦ ENRICH / CLASSIFY
taxonomy · clusters"]
G --> H["⑧ DECIDE
retriever + threshold + trigger"]
F --> I["QUERY
retriever stages"]
H -.->|"pass / flag / hold"| OUT["your destination
object storage or webhook"]
```
Nodes are execution environments and artifacts are what runs on them, so this view
answers "what is deployed where" rather than "what happens next".
PlantUML source, so the next editor regenerates rather than redraws:
```plantuml theme={null}
@startuml
node "Mixpeek account" as MXP {
artifact "API gateway · auth · billing · plan gates" as GW
}
node "Your Kubernetes cluster" as K8S {
node "Mixpeek services" as SVC {
artifact "API pods" as API
artifact "Celery workers · 7 queue shards" as CEL
artifact "Redis · broker, locks" as REDIS
}
node "Ray on KubeRay" as RAY {
artifact "RayJob · one per batch\nephemeral · holds no models" as JOB
artifact "RayService · Serve\nlong-lived · holds the models" as SERVE
}
node "Stateful services" as STATE {
artifact "MVS shards · Rust" as MVS
artifact "MongoDB" as MONGO
artifact "ClickHouse" as CH
}
}
node "Your object storage" as OS {
artifact "source objects" as OBJ
artifact "sealed WAL" as WAL
artifact "MVS snapshots" as SNAP
}
node "Hosted LLM provider" as LLM {
artifact "OpenAI · Anthropic · Vertex" as PROV
}
GW ..> API : "https (managed control plane only)"
JOB ..> OBJ : "reads blobs"
JOB ..> SERVE : "inference RPC"
MVS ..> WAL : "ships sealed WAL"
MVS ..> SNAP : "ships snapshots"
SERVE ..> PROV : "egress, only if a stage uses a hosted model"
@enduml
```
## The boundary, first
BYOC splits on one line: you provide the substrate, Mixpeek provides and operates the
software.
**Where the control plane sits is an install-time choice.** The diagrams above draw the
managed shape.
Six collections are control-plane by definition: API keys, organizations, users, tenant
routing, the Canvas app registry, and an internal repair queue. On a **managed** install
those resolve to a Mixpeek-operated database and your namespace egresses to it, which is
the egress the install kit's network policy opens. On a **fully self-hosted** install the
control-plane database is left unset, the same six resolve to the database in your own
namespace, and no pipeline stage needs outbound to Mixpeek.
The eight stages below are identical in both shapes. Two things outside the pipeline
differ, and both are worth knowing before you design around self-hosted:
* **Canvas apps.** The build worker, the bundle store and the app runtime all operate on
one shared registry. A registry sitting in your own namespace is not reachable by them,
so Canvas apps are unavailable on that shape.
* **Usage reporting.** Your organization record and your usage records both sit in your
namespace, so nothing reports consumption back to Mixpeek. Metering becomes a
commercial arrangement rather than something the platform does for you.
Which shape you get is set at install time, so tell us which one your environment
requires and it is configured that way from the first deploy.
The eight stages below each answer the same four questions: input, work, output, resources.
## Stage by stage
### ① Arrival
| | |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | Bytes, from one of two doors |
| **Work** | Door A: `POST /v1/buckets/{id}/uploads` returns `upload_id` and `presigned_url`; the client PUTs straight to object storage, so bytes never transit the API. Door B: a sync pulls from a source you already have. |
| **Output** | An object in your bucket, at the storage class the bucket's `storage_tier` names |
| **Resources** | Object storage write. No compute. No Mixpeek pod touches the payload. |
| **Where** | Your cloud. The API only mints and validates the URL. |
Sync providers today: `google_drive`, `s3`, `snowflake`, `sharepoint`, `tigris`, `iconik`.
Modes are `initial_only` or `continuous` with a `polling_interval_seconds`. **Continuous
sync is polling, not an event subscription**, which sets your latency expectation for "a
video arrives".
### ② Object
| | |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | The stored bytes plus caller metadata |
| **Work** | Create the bucket row: `object_id`, `key_prefix`, `blobs[]`, `source_details[]`. Declared fields are type-checked against `bucket_schema` on write; undeclared fields pass through. Unique-key policy dedupes here. |
| **Output** | An object record in Mongo, blobs referencing storage locations |
| **Resources** | API pod CPU (small), one Mongo write, object-storage metadata read |
| **Where** | Mixpeek software in your namespace; Mongo in your data plane |
**A video is one object with blobs, not many objects.** Decomposition into chunks, frames
and scenes happens in stage ⑤, which is what makes tiers necessary.
### ③ Batch
| | |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | `object_ids[]` (or a bucket-wide selector), `dedup_strategy` |
| **Work** | Plan gates check `max_batch_size` and `max_active_batches`; the cost estimator prices it; the ledger excludes objects already extracted at the same content and config hash. |
| **Output** | A batch record with one task per processing tier |
| **Resources** | API pod CPU, Mongo write, credit ledger read |
| **Where** | Control-plane logic in your namespace; billing meters back to Mixpeek |
`dedup_strategy` is the cost-multiplying knob: a re-run with the wrong value re-pays full
extraction. **Tier 0 is bucket to collection; tier N above 0 is collection to collection.**
A pipeline that turns a video into chunks, chunks into frames, and frames into embeddings is
three tiers, each independently tracked and retryable.
### ④ Schedule
This is where the extractor stops being a name and becomes a machine.
| | |
| ------------- | -------------------------------------------------------------------------------------------------------- |
| **Input** | The collection's `feature_extractor`, the batch's object count, the namespace's `compute_tier` |
| **Work** | Route to a Celery queue, resolve a GKE compute profile, compute worker counts |
| **Output** | A RayJob spec: head plus CPU workers plus optional GPU workers, with requests, limits and replica bounds |
| **Resources** | Celery worker slot (7 queue-sharded deployments), Redis as broker |
| **Where** | Your namespace |
Profile resolution is a six-rule ladder:
| # | Rule | Result |
| - | --------------------------------- | ----------------------------------------------------------- |
| 1 | explicit `profile_name` | that profile |
| 2 | extractor declares `gke_profile` | that profile |
| 3 | plugin job and not `requires_gpu` | `plugin`, hard-isolated on a tainted node |
| 4 | not `requires_gpu` | `cpu-only`, skips the GPU pool |
| 5 | job-type default | batch → `default` · cluster → `small` · benchmark → `small` |
| 6 | fallback | `default`, which carries a GPU worker group |
**Only 6 of 12 built-in extractors declare a profile.** For the rest, `requires_gpu` decides
the hardware, not anything named in the definition. Choosing an extractor chooses a node
pool and therefore a bill.
Sizing is `object_count ÷ objects_per_worker → replicas`, clamped by the tenancy slice and a
350Gi ceiling. Shared tenancy gets gpu 2 / cpu 32 / api 4; `DEDICATED_GPU` rises to gpu 8 /
api 12.
### ⑤ Execute
| | |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | RayJob spec, object references, extractor config |
| **Work** | KubeRay materializes a per-batch cluster; workers read blobs from object storage, decompose, call models, emit documents |
| **Output** | Document payloads with vectors and `_internal` lineage, in memory |
| **Resources** | Head pod plus N CPU workers plus optional GPU workers; the node autoscaler provisions from your pools |
| **Where** | Your Kubernetes, your nodes, under your taints and quotas. A model-backed extractor is the exception, and it sends the media itself: see [What leaves your account](#what-leaves-your-account) |
```mermaid theme={null}
flowchart LR
subgraph EPH["Ephemeral RayJob · dies with the batch"]
HEAD["head
1 cpu / 4–12Gi"]
CW["cpu workers
N × 1 cpu / 2–6Gi"]
GW["gpu workers
0–4 × 2 cpu / 12–32Gi + 1×GPU
(only if the profile has the group)"]
end
subgraph LIVE["RayService · long-lived, autoscaled, shared"]
SERVE["Serve deployments
embedders · rerankers · VLM adapters"]
end
OS[("object storage")] -->|"read blobs"| CW
CW -->|"inference RPC"| SERVE
GW -->|"inference RPC"| SERVE
CW --> OUT["documents"]
GW --> OUT
```
**Two Ray shapes, and confusing them is the usual mistake.** The RayService is long-lived
and holds the models. The RayJob is ephemeral, one per batch, and holds no models at all for
Serve-backed extractors. A `cpu-only` worker's real peak is about 0.27 core and 0.77 GiB,
which is why its requests are 1 cpu / 2Gi and why one node holds seven or eight concurrent
batches instead of three.
Two mechanics worth carrying:
* **Kubernetes `cpu_req` is not Ray `num-cpus`.** `cpu_workers.ray_num_cpus` decouples them,
so a worker can advertise 2 schedulable CPU to Ray for placement groups while requesting 1
from Kubernetes. Without it, multi-step streaming placement cannot schedule and the job
orphans.
* **Requests drive the autoscaler; limits are the OOM gate.** Limits stay generous so a
decode or frame burst is absorbed rather than OOMKilled.
### ⑥ Persist
| | |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | Document payloads from the workers |
| **Work** | Writer → MVS coordinator → primary shard gRPC → WAL append → staging buffer → index flush → PQ train → seal WAL → ship to object storage |
| **Output** | Queryable points; metadata in Mongo; usage events to ClickHouse |
| **Resources** | MVS pod 2–4 CPU / 4–64 GiB; object-storage writes for WAL and snapshots |
| **Where** | Your Kubernetes, your disks, your object storage |
```mermaid theme={null}
flowchart LR
W["Ray worker
SyncVectorWriter"] --> C["MVS coordinator
(Python, per namespace)"]
C -->|"UpsertDocuments gRPC :50051"| P["primary shard (Rust)"]
P --> WAL["WAL append
local disk"]
P --> STG["staging buffer"]
STG -->|"≥1000 or interval"| LIRE["LIRE index flush"]
LIRE -->|"partition ≥ 50k vectors"| PQ["train PQ 32×8"]
WAL -->|"seal + ship"| OBJ[("object storage
sealed WAL + snapshot")]
OBJ -->|"cold-start restore ~seconds"| P
C -->|"SearchDense reads"| R["replica shard(s)"]
OBJ -.->|"restore"| R
```
The knobs that decide cost and recall:
| Knob | Default | Trades |
| ------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `MVS_NPROBE` | 8 | partitions searched per query: the recall against latency dial |
| `pq_subvectors / pq_bits` | 32 / 8 | \~32× memory reduction; PQ serves the hot path, raw vectors retained for reconstruction |
| `pq_training_threshold` | 50,000 | below it, compression buys nothing |
| `max_vectors` per shard | 50,000,000 | split point: more shards, more fan-out |
| `MVS_IDLE_EVICT_MINUTES` | 30 | idle namespaces evict and restore on demand. This is what makes the cost math work: one pod amortized across hundreds of idle namespaces |
| `MVS_FLUSH_INTERVAL_SECS` | env | bounds RPO. Worst-case loss is writes since the last sealed WAL shipped |
**Durability, stated plainly:** the recovery source is object storage. A rescheduled primary
replays the sealed WAL plus snapshot, so RPO is bounded by the flush interval rather than by
zero. Replicas serve reads and do not shorten that gap.
Replica counts vary by namespace, and a namespace under maintenance can run a shape that
does not match the steady state above. Treat this section as the design, and read the
actual pod counts in your own namespace before depending on them.
### ⑦ Enrich and classify
| | |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | Persisted documents |
| **Work** | Taxonomy enrichment (flat or hierarchical) writes labels back onto documents; clustering runs group them |
| **Output** | Documents carrying classification fields |
| **Resources** | A Serve `taxonomy_join` deployment, or LLM providers for label generation; clustering runs as its own RayJob on the `small` profile |
| **Where** | Your namespace. LLM-based hierarchy inference egresses; flat enrichment does not. See [What leaves your account](#what-leaves-your-account) |
Flat taxonomy enrichment calls no language model. It matches against a reference collection
and writes labels back. Hierarchy inference has three methods, and only the LLM-based one
calls a provider.
Taxonomy is a small part of the egress picture, and the section below is the whole of it.
### ⑧ Decide
**This stage composes primitives; there is no moderation resource in the API.**
| | |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Input** | Enriched documents |
| **Work** | A retriever with a `score_threshold` reduce stage, or a taxonomy assignment read, produces a verdict. A trigger (`cron`, `interval`, `event`, `conditional`) acts on it. |
| **Output** | A pass, flag or hold decision, and whatever the trigger's action writes |
| **Resources** | MVS query CPU; reranker Serve replicas if the retriever sorts; LLM tokens if it enriches |
| **Where** | Your namespace, plus any external provider a stage calls |
The platform computes the verdict and fires the trigger, but writing that decision back out
as a destination object with lineage is the weak link. If moderation must end with "a file
lands in a `reviewed/` bucket", treat that as integration work today rather than a
primitive.
## What leaves your account
Egress is not a property of one stage. It is a property of which extractors and which
retriever stages a pipeline is configured with, and it exists at **both ingest and query
time**. Enumerated from the call sites rather than characterised, because every attempt to
describe this as a category has been wrong:
**At ingest**, a model-backed extractor sends the content itself. `gemini_multifile_extractor`
builds Gemini parts with `Part.from_bytes(data=content, mime_type=...)`, so for an image or a
PDF the **media** goes to the provider, not a label derived from it. The same holds for
`multimodal_extractor` v2 Gemini embedding, `universal_extractor`, `scrolling_text_extractor`,
`document_graph_extractor`'s VLM corrector, and `web_scraper`'s semantic router. Cluster
labelling calls a model too.
**At query time**, seven retriever stages call providers: `enrich/llm`, `enrich/agentic`,
`filter/llm`, `filter/agent_search`, `filter/attribute`, `filter/query_expand` and
`reduce/summarize`. The `enrich` family is the volume path, once per result, so an enrich
stage over 100 results is 100 calls.
**Those same seven also run at ingest**, whenever a taxonomy or a collection's
`retriever_enrichments` points at a retriever. That path fires once per *document* rather
than once per result, which makes it the larger volume of the two on a bulk load. Reading
the collection will not tell you, because its config carries only a `retriever_id`; you
have to follow the reference to the retriever and read its stages.
**Which of these are in your pipeline is configuration.** A pipeline built on self-hosted
models in your own Serve deployments makes none of these calls. One built on
`gemini_multifile_extractor` sends every object's bytes to Google at ingest. Both are
supported, and the difference is a choice you make per collection and per retriever rather
than a property of the platform.
Three things change the answer, and all three are available today:
* **Your own provider credentials.** Organization-default LLM credentials resolve per request,
so these calls can run on your OpenAI, Anthropic or Vertex account. The egress becomes your
contract with the provider rather than ours.
* **Self-hosted models.** Anything served from your own Ray Serve deployments stays inside
your VPC end to end.
* **Replace the path.** A custom plugin can take over the enrich batch entirely, so the
built-in provider call is never made.
If a reviewer needs the exhaustive list for a given namespace, derive it from that
namespace's own configuration rather than from this page. Ask us to run the namespace
egress check against it: the check reads your collections, taxonomies, clusters and
retrievers, names every configured component that reaches a provider, and refuses to
report a clean result when it hits a reference it could not follow. The set above grows
whenever an extractor or a stage is added, so a page cannot stay exhaustive.
### Model weights are a different kind of egress
No customer content leaves on a self-hosted-model pipeline, which is the commitment
above. The **weights** do arrive from outside, carrying nothing of yours, from
HuggingFace and similar hosts.
Two of the six model sets are baked into the engine image and need no network at
all. The other four are fetched **the first time each replica loads them**, not at
image build:
| Model | Size | When |
| -------------------------------- | ------- | -------------------- |
| ArcFace (`buffalo_l`) | 601 MB | baked into the image |
| `multilingual-e5-large-instruct` | 1.1 GB | baked into the image |
| `bge-reranker-v2-m3` | 2.2 GB | first replica load |
| `siglip-base-patch16-224` | 779 MB | first replica load |
| `all-MiniLM-L6-v2` | 88 MB | first replica load |
| CLAP `630k-audioset-best.pt` | 1.86 GB | first audio object |
It matters operationally rather than for a privacy review, and the failure is easy
to misread. A namespace with no outbound to the public internet brings the Ray
cluster up healthy and then leaves those four applications unable to load, which
reads as a slow install for as long as it takes someone to look at the Serve status.
Decide it before the first deploy. Either allow egress to `huggingface.co` on first
start, or ask us for an image with all six baked.
## Query path
Retrieval reuses the index the ingest path built.
| Stage family | Executes against | Cost driver |
| ------------ | ------------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| **filter** | MVS gRPC (vector and payload pushdown); `llm` and `query_expand` hit the LLM service | MVS CPU × `nprobe`; LLM tokens |
| **sort** | `rerank` goes to a Ray Serve reranker; the rest run in-process | reranker replica time |
| **reduce** | database-level aggregation; `cluster` goes to Serve; `summarize` to an LLM | mostly free; summarize is tokens |
| **enrich** | LLM providers, `taxonomy_join` Serve, sandbox for code | **per result**: an enrich stage on 100 results is 100 LLM calls |
| **apply** | external HTTP, your SQL, web search | third-party limits and egress |
Read mechanics: coordinator → `SearchDense` RPC (fan-out across the hash ring for
multi-shard namespaces) → shard probes `nprobe` partitions via PQ codes → re-ranks top-N
against raw vectors → returns point id, score and payload.
**Filter predicates are evaluated coordinator-side, after the shard returns.** That is known
waste, and it is what compute pushdown is for.
## Resource map
| Stage | Compute | Storage touched | State | Scales with |
| ------------ | ------------------------------------------ | ---------------------------------- | -------------------- | ----------------------------------- |
| ① Arrival | none (presigned) or sync worker | object storage write | none | upload concurrency |
| ② Object | API pod, small | Mongo write | Mongo | objects/sec |
| ③ Batch | API pod, small | Mongo write | Mongo, credit ledger | batch creation rate |
| ④ Schedule | Celery slot | Redis (broker, locks) | Redis | active batches |
| ⑤ Execute | RayJob head plus N workers, optionally GPU | object storage read | ephemeral | `object_count ÷ objects_per_worker` |
| ⑤b Inference | RayService Serve replicas, autoscaled | model cache | long-lived | concurrent inference RPS |
| ⑥ Persist | MVS pod 2–4 CPU / 4–64 GiB | WAL local disk plus object storage | **the durable one** | vectors, partitions |
| ⑦ Enrich | Serve replicas and/or LLM providers | Mongo, MVS | documents | documents × labels |
| ⑧ Decide | MVS query plus optional Serve or LLM | reads only | none | verdicts/sec |
| Query | MVS plus Serve plus providers | reads only | none | QPS × `nprobe` × stages |
| Store | Role | Loss impact |
| ------------------ | ------------------------------------------------------------------- | -------------------------------------------- |
| **Object storage** | bucket objects, MVS snapshots and sealed WAL, exports | catastrophic: it is the durability floor |
| **MVS** | vectors plus the `_internal` payload envelope | rebuildable from source objects, expensively |
| **Mongo** | resource metadata, batch and tier state, non-vector document fields | control-plane truth |
| **Redis** | Celery broker, distributed locks, caches, heartbeats | in-flight work, not durable state |
| **ClickHouse** | usage and analytics events | billing and analytics history |
## Where it breaks
Back-pressure and failure points, in the order they bite:
| Point | Symptom | Absorbed by |
| ---------------- | ----------------------------------- | -------------------------------------------------------------- |
| Plan gate | batch refused at creation | `max_batch_size`, `max_active_batches` |
| Tenancy clamp | fewer workers than sizing asked for | shared slice (gpu 2 / cpu 32 / api 4) |
| Node autoscaler | RayJob pods pend | your pool max and quota, your policy rather than ours |
| Spot preemption | worker dies mid-batch | Ray retry; drivers pinned to on-demand in dedicated tenancy |
| Serve saturation | inference RPC latency climbs | Serve autoscaling within replica bounds |
| MVS staging | write latency spikes at flush | staging threshold 1000, flush interval |
| MVS eviction | first query after idle is slow | 30-minute idle evict, restore on demand |
| Tier failure | one tier fails, the batch continues | per-tier tracking, retry without reprocessing successful tiers |
One correctness note, because it is invisible otherwise: a batch that finishes with a failed
tier reports `COMPLETED_WITH_ERRORS`, not `COMPLETED`. **If you build a gate that polls batch
status, poll for the honest values.**
## What this page does not establish
* **Where the control plane sits.** The section above states the disagreement rather than
picking a side. The install kit deploys no API, so it is the permission grant rather than
the workload and does not settle it either.
* **A live BYOC deployment.** Stage-by-stage resource numbers are read from the shared
production cluster, which is the same software on Mixpeek's substrate. Your pools, quotas
and storage classes are yours and will differ.
* **Networking and egress detail.** VPC layout, private-link options and per-provider network
paths are untraced. What is enumerated is the set of call sites that reach a model provider,
listed above. That list was wrong three times before it was right, each time by describing
the category instead of counting: treat it as accurate on its date and derive the live set
from a namespace's own configured extractors and stages.
* **Moderation as a product surface.** It does not exist as a resource. Stage ⑧ is a
composition, and the destination-write gap is real.
* **Continuous sync latency.** Sync polls on `polling_interval_seconds`. End-to-end arrival
latency is unmeasured for every provider.
The shapes Mixpeek runs in, and which planes sit where.
Dedicated tenancy, and what changes about the numbers above.