End-to-End Data Flow

One object entering the system, through every layer, to a decision and a queryable index. Written extractor-agnostic: the extractor is a variable, and this page shows where that variable is resolved and what it changes. The primary walk is customer-hosted (BYOC), because that is the shape where “what runs where” matters. Mixpeek Cloud is the same pipeline with every plane in Mixpeek’s account.

Same diagram as Mermaid

Same system as a UML deployment diagram

Nodes are execution environments and artifacts are what runs on them, so this view answers “what is deployed where” rather than “what happens next”.

PlantUML source, so the next editor regenerates rather than redraws:

@startuml
node "Mixpeek account" as MXP {
  artifact "API gateway · auth · billing · plan gates" as GW
}
node "Your Kubernetes cluster" as K8S {
  node "Mixpeek services" as SVC {
    artifact "API pods" as API
    artifact "Celery workers · 7 queue shards" as CEL
    artifact "Redis · broker, locks" as REDIS
  }
  node "Ray on KubeRay" as RAY {
    artifact "RayJob · one per batch\nephemeral · holds no models" as JOB
    artifact "RayService · Serve\nlong-lived · holds the models" as SERVE
  }
  node "Stateful services" as STATE {
    artifact "MVS shards · Rust" as MVS
    artifact "MongoDB" as MONGO
    artifact "ClickHouse" as CH
  }
}
node "Your object storage" as OS {
  artifact "source objects" as OBJ
  artifact "sealed WAL" as WAL
  artifact "MVS snapshots" as SNAP
}
node "Hosted LLM provider" as LLM {
  artifact "OpenAI · Anthropic · Vertex" as PROV
}
GW ..> API : "https (managed control plane only)"
JOB ..> OBJ : "reads blobs"
JOB ..> SERVE : "inference RPC"
MVS ..> WAL : "ships sealed WAL"
MVS ..> SNAP : "ships snapshots"
SERVE ..> PROV : "egress, only if a stage uses a hosted model"
@enduml

The boundary, first

BYOC splits on one line: you provide the substrate, Mixpeek provides and operates the software.

Where the control plane sits is an install-time choice. The diagrams above draw the managed shape.Six collections are control-plane by definition: API keys, organizations, users, tenant routing, the Canvas app registry, and an internal repair queue. On a managed install those resolve to a Mixpeek-operated database and your namespace egresses to it, which is the egress the install kit’s network policy opens. On a fully self-hosted install the control-plane database is left unset, the same six resolve to the database in your own namespace, and no pipeline stage needs outbound to Mixpeek.The eight stages below are identical in both shapes. Two things outside the pipeline differ, and both are worth knowing before you design around self-hosted:

Canvas apps. The build worker, the bundle store and the app runtime all operate on one shared registry. A registry sitting in your own namespace is not reachable by them, so Canvas apps are unavailable on that shape.
Usage reporting. Your organization record and your usage records both sit in your namespace, so nothing reports consumption back to Mixpeek. Metering becomes a commercial arrangement rather than something the platform does for you.

Which shape you get is set at install time, so tell us which one your environment requires and it is configured that way from the first deploy.

The eight stages below each answer the same four questions: input, work, output, resources.

Stage by stage

① Arrival


Input	Bytes, from one of two doors
Work	Door A: `POST /v1/buckets/{id}/uploads` returns `upload_id` and `presigned_url`; the client PUTs straight to object storage, so bytes never transit the API. Door B: a sync pulls from a source you already have.
Output	An object in your bucket, at the storage class the bucket’s `storage_tier` names
Resources	Object storage write. No compute. No Mixpeek pod touches the payload.
Where	Your cloud. The API only mints and validates the URL.

Sync providers today: google_drive, s3, snowflake, sharepoint, tigris, iconik. Modes are initial_only or continuous with a polling_interval_seconds. Continuous sync is polling, not an event subscription, which sets your latency expectation for “a video arrives”.

② Object


Input	The stored bytes plus caller metadata
Work	Create the bucket row: `object_id`, `key_prefix`, `blobs[]`, `source_details[]`. Declared fields are type-checked against `bucket_schema` on write; undeclared fields pass through. Unique-key policy dedupes here.
Output	An object record in Mongo, blobs referencing storage locations
Resources	API pod CPU (small), one Mongo write, object-storage metadata read
Where	Mixpeek software in your namespace; Mongo in your data plane

A video is one object with blobs, not many objects. Decomposition into chunks, frames and scenes happens in stage ⑤, which is what makes tiers necessary.

③ Batch


Input	`object_ids[]` (or a bucket-wide selector), `dedup_strategy`
Work	Plan gates check `max_batch_size` and `max_active_batches`; the cost estimator prices it; the ledger excludes objects already extracted at the same content and config hash.
Output	A batch record with one task per processing tier
Resources	API pod CPU, Mongo write, credit ledger read
Where	Control-plane logic in your namespace; billing meters back to Mixpeek

dedup_strategy is the cost-multiplying knob: a re-run with the wrong value re-pays full extraction. Tier 0 is bucket to collection; tier N above 0 is collection to collection. A pipeline that turns a video into chunks, chunks into frames, and frames into embeddings is three tiers, each independently tracked and retryable.

④ Schedule

This is where the extractor stops being a name and becomes a machine.


Input	The collection’s `feature_extractor`, the batch’s object count, the namespace’s `compute_tier`
Work	Route to a Celery queue, resolve a GKE compute profile, compute worker counts
Output	A RayJob spec: head plus CPU workers plus optional GPU workers, with requests, limits and replica bounds
Resources	Celery worker slot (7 queue-sharded deployments), Redis as broker
Where	Your namespace

Profile resolution is a six-rule ladder:

#	Rule	Result
1	explicit `profile_name`	that profile
2	extractor declares `gke_profile`	that profile
3	plugin job and not `requires_gpu`	`plugin`, hard-isolated on a tainted node
4	not `requires_gpu`	`cpu-only`, skips the GPU pool
5	job-type default	batch → `default` · cluster → `small` · benchmark → `small`
6	fallback	`default`, which carries a GPU worker group

Only 6 of 12 built-in extractors declare a profile. For the rest, requires_gpu decides the hardware, not anything named in the definition. Choosing an extractor chooses a node pool and therefore a bill. Sizing is object_count ÷ objects_per_worker → replicas, clamped by the tenancy slice and a 350Gi ceiling. Shared tenancy gets gpu 2 / cpu 32 / api 4; DEDICATED_GPU rises to gpu 8 / api 12.

⑤ Execute


Input	RayJob spec, object references, extractor config
Work	KubeRay materializes a per-batch cluster; workers read blobs from object storage, decompose, call models, emit documents
Output	Document payloads with vectors and `_internal` lineage, in memory
Resources	Head pod plus N CPU workers plus optional GPU workers; the node autoscaler provisions from your pools
Where	Your Kubernetes, your nodes, under your taints and quotas. A model-backed extractor is the exception, and it sends the media itself: see What leaves your account

Two Ray shapes, and confusing them is the usual mistake. The RayService is long-lived and holds the models. The RayJob is ephemeral, one per batch, and holds no models at all for Serve-backed extractors. A cpu-only worker’s real peak is about 0.27 core and 0.77 GiB, which is why its requests are 1 cpu / 2Gi and why one node holds seven or eight concurrent batches instead of three. Two mechanics worth carrying:

Kubernetes cpu_req is not Ray num-cpus. cpu_workers.ray_num_cpus decouples them, so a worker can advertise 2 schedulable CPU to Ray for placement groups while requesting 1 from Kubernetes. Without it, multi-step streaming placement cannot schedule and the job orphans.
Requests drive the autoscaler; limits are the OOM gate. Limits stay generous so a decode or frame burst is absorbed rather than OOMKilled.

⑥ Persist


Input	Document payloads from the workers
Work	Writer → MVS coordinator → primary shard gRPC → WAL append → staging buffer → index flush → PQ train → seal WAL → ship to object storage
Output	Queryable points; metadata in Mongo; usage events to ClickHouse
Resources	MVS pod 2–4 CPU / 4–64 GiB; object-storage writes for WAL and snapshots
Where	Your Kubernetes, your disks, your object storage

The knobs that decide cost and recall:

Knob	Default	Trades
`MVS_NPROBE`	8	partitions searched per query: the recall against latency dial
`pq_subvectors / pq_bits`	32 / 8	~32× memory reduction; PQ serves the hot path, raw vectors retained for reconstruction
`pq_training_threshold`	50,000	below it, compression buys nothing
`max_vectors` per shard	50,000,000	split point: more shards, more fan-out
`MVS_IDLE_EVICT_MINUTES`	30	idle namespaces evict and restore on demand. This is what makes the cost math work: one pod amortized across hundreds of idle namespaces
`MVS_FLUSH_INTERVAL_SECS`	env	bounds RPO. Worst-case loss is writes since the last sealed WAL shipped

Durability, stated plainly: the recovery source is object storage. A rescheduled primary replays the sealed WAL plus snapshot, so RPO is bounded by the flush interval rather than by zero. Replicas serve reads and do not shorten that gap.

Replica counts vary by namespace, and a namespace under maintenance can run a shape that does not match the steady state above. Treat this section as the design, and read the actual pod counts in your own namespace before depending on them.

⑦ Enrich and classify


Input	Persisted documents
Work	Taxonomy enrichment (flat or hierarchical) writes labels back onto documents; clustering runs group them
Output	Documents carrying classification fields
Resources	A Serve `taxonomy_join` deployment, or LLM providers for label generation; clustering runs as its own RayJob on the `small` profile
Where	Your namespace. LLM-based hierarchy inference egresses; flat enrichment does not. See What leaves your account

Flat taxonomy enrichment calls no language model. It matches against a reference collection and writes labels back. Hierarchy inference has three methods, and only the LLM-based one calls a provider. Taxonomy is a small part of the egress picture, and the section below is the whole of it.

⑧ Decide

This stage composes primitives; there is no moderation resource in the API.


Input	Enriched documents
Work	A retriever with a `score_threshold` reduce stage, or a taxonomy assignment read, produces a verdict. A trigger (`cron`, `interval`, `event`, `conditional`) acts on it.
Output	A pass, flag or hold decision, and whatever the trigger’s action writes
Resources	MVS query CPU; reranker Serve replicas if the retriever sorts; LLM tokens if it enriches
Where	Your namespace, plus any external provider a stage calls

The platform computes the verdict and fires the trigger, but writing that decision back out as a destination object with lineage is the weak link. If moderation must end with “a file lands in a reviewed/ bucket”, treat that as integration work today rather than a primitive.

What leaves your account

Egress is not a property of one stage. It is a property of which extractors and which retriever stages a pipeline is configured with, and it exists at both ingest and query time. Enumerated from the call sites rather than characterised, because every attempt to describe this as a category has been wrong: At ingest, a model-backed extractor sends the content itself. gemini_multifile_extractor builds Gemini parts with Part.from_bytes(data=content, mime_type=...), so for an image or a PDF the media goes to the provider, not a label derived from it. The same holds for multimodal_extractor v2 Gemini embedding, universal_extractor, scrolling_text_extractor, document_graph_extractor’s VLM corrector, and web_scraper’s semantic router. Cluster labelling calls a model too. At query time, seven retriever stages call providers: enrich/llm, enrich/agentic, filter/llm, filter/agent_search, filter/attribute, filter/query_expand and reduce/summarize. The enrich family is the volume path, once per result, so an enrich stage over 100 results is 100 calls. Those same seven also run at ingest, whenever a taxonomy or a collection’s retriever_enrichments points at a retriever. That path fires once per document rather than once per result, which makes it the larger volume of the two on a bulk load. Reading the collection will not tell you, because its config carries only a retriever_id; you have to follow the reference to the retriever and read its stages. Which of these are in your pipeline is configuration. A pipeline built on self-hosted models in your own Serve deployments makes none of these calls. One built on gemini_multifile_extractor sends every object’s bytes to Google at ingest. Both are supported, and the difference is a choice you make per collection and per retriever rather than a property of the platform. Three things change the answer, and all three are available today:

Your own provider credentials. Organization-default LLM credentials resolve per request, so these calls can run on your OpenAI, Anthropic or Vertex account. The egress becomes your contract with the provider rather than ours.
Self-hosted models. Anything served from your own Ray Serve deployments stays inside your VPC end to end.
Replace the path. A custom plugin can take over the enrich batch entirely, so the built-in provider call is never made.

If a reviewer needs the exhaustive list for a given namespace, derive it from that namespace’s own configuration rather than from this page. Ask us to run the namespace egress check against it: the check reads your collections, taxonomies, clusters and retrievers, names every configured component that reaches a provider, and refuses to report a clean result when it hits a reference it could not follow. The set above grows whenever an extractor or a stage is added, so a page cannot stay exhaustive.

Model weights are a different kind of egress

No customer content leaves on a self-hosted-model pipeline, which is the commitment above. The weights do arrive from outside, carrying nothing of yours, from HuggingFace and similar hosts. Two of the six model sets are baked into the engine image and need no network at all. The other four are fetched the first time each replica loads them, not at image build:

Model	Size	When
ArcFace (`buffalo_l`)	601 MB	baked into the image
`multilingual-e5-large-instruct`	1.1 GB	baked into the image
`bge-reranker-v2-m3`	2.2 GB	first replica load
`siglip-base-patch16-224`	779 MB	first replica load
`all-MiniLM-L6-v2`	88 MB	first replica load
CLAP `630k-audioset-best.pt`	1.86 GB	first audio object

It matters operationally rather than for a privacy review, and the failure is easy to misread. A namespace with no outbound to the public internet brings the Ray cluster up healthy and then leaves those four applications unable to load, which reads as a slow install for as long as it takes someone to look at the Serve status. Decide it before the first deploy. Either allow egress to huggingface.co on first start, or ask us for an image with all six baked.

Query path

Retrieval reuses the index the ingest path built.

Stage family	Executes against	Cost driver
filter	MVS gRPC (vector and payload pushdown); `llm` and `query_expand` hit the LLM service	MVS CPU × `nprobe`; LLM tokens
sort	`rerank` goes to a Ray Serve reranker; the rest run in-process	reranker replica time
reduce	database-level aggregation; `cluster` goes to Serve; `summarize` to an LLM	mostly free; summarize is tokens
enrich	LLM providers, `taxonomy_join` Serve, sandbox for code	per result: an enrich stage on 100 results is 100 LLM calls
apply	external HTTP, your SQL, web search	third-party limits and egress

Read mechanics: coordinator → SearchDense RPC (fan-out across the hash ring for multi-shard namespaces) → shard probes nprobe partitions via PQ codes → re-ranks top-N against raw vectors → returns point id, score and payload. Filter predicates are evaluated coordinator-side, after the shard returns. That is known waste, and it is what compute pushdown is for.

Resource map

Stage	Compute	Storage touched	State	Scales with
① Arrival	none (presigned) or sync worker	object storage write	none	upload concurrency
② Object	API pod, small	Mongo write	Mongo	objects/sec
③ Batch	API pod, small	Mongo write	Mongo, credit ledger	batch creation rate
④ Schedule	Celery slot	Redis (broker, locks)	Redis	active batches
⑤ Execute	RayJob head plus N workers, optionally GPU	object storage read	ephemeral	`object_count ÷ objects_per_worker`
⑤b Inference	RayService Serve replicas, autoscaled	model cache	long-lived	concurrent inference RPS
⑥ Persist	MVS pod 2–4 CPU / 4–64 GiB	WAL local disk plus object storage	the durable one	vectors, partitions
⑦ Enrich	Serve replicas and/or LLM providers	Mongo, MVS	documents	documents × labels
⑧ Decide	MVS query plus optional Serve or LLM	reads only	none	verdicts/sec
Query	MVS plus Serve plus providers	reads only	none	QPS × `nprobe` × stages

Store	Role	Loss impact
Object storage	bucket objects, MVS snapshots and sealed WAL, exports	catastrophic: it is the durability floor
MVS	vectors plus the `_internal` payload envelope	rebuildable from source objects, expensively
Mongo	resource metadata, batch and tier state, non-vector document fields	control-plane truth
Redis	Celery broker, distributed locks, caches, heartbeats	in-flight work, not durable state
ClickHouse	usage and analytics events	billing and analytics history

Where it breaks

Back-pressure and failure points, in the order they bite:

Point	Symptom	Absorbed by
Plan gate	batch refused at creation	`max_batch_size`, `max_active_batches`
Tenancy clamp	fewer workers than sizing asked for	shared slice (gpu 2 / cpu 32 / api 4)
Node autoscaler	RayJob pods pend	your pool max and quota, your policy rather than ours
Spot preemption	worker dies mid-batch	Ray retry; drivers pinned to on-demand in dedicated tenancy
Serve saturation	inference RPC latency climbs	Serve autoscaling within replica bounds
MVS staging	write latency spikes at flush	staging threshold 1000, flush interval
MVS eviction	first query after idle is slow	30-minute idle evict, restore on demand
Tier failure	one tier fails, the batch continues	per-tier tracking, retry without reprocessing successful tiers

One correctness note, because it is invisible otherwise: a batch that finishes with a failed tier reports COMPLETED_WITH_ERRORS, not COMPLETED. If you build a gate that polls batch status, poll for the honest values.

What this page does not establish

Where the control plane sits. The section above states the disagreement rather than picking a side. The install kit deploys no API, so it is the permission grant rather than the workload and does not settle it either.
A live BYOC deployment. Stage-by-stage resource numbers are read from the shared production cluster, which is the same software on Mixpeek’s substrate. Your pools, quotas and storage classes are yours and will differ.
Networking and egress detail. VPC layout, private-link options and per-provider network paths are untraced. What is enumerated is the set of call sites that reach a model provider, listed above. That list was wrong three times before it was right, each time by describing the category instead of counting: treat it as accurate on its date and derive the live set from a namespace’s own configured extractors and stages.
Moderation as a product surface. It does not exist as a resource. Stage ⑧ is a composition, and the destination-write gap is real.
Continuous sync latency. Sync polls on polling_interval_seconds. End-to-end arrival latency is unmeasured for every provider.

Deployment models

The shapes Mixpeek runs in, and which planes sit where.

Single tenant

Dedicated tenancy, and what changes about the numbers above.

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

End-to-End Data Flow

The boundary, first

Stage by stage

① Arrival

② Object

③ Batch

④ Schedule

⑤ Execute

⑥ Persist

⑦ Enrich and classify

⑧ Decide

What leaves your account

Model weights are a different kind of egress

Query path

Resource map

Where it breaks

What this page does not establish

Deployment models

Single tenant

​The boundary, first

​Stage by stage

​① Arrival

​② Object

​③ Batch

​④ Schedule

​⑤ Execute

​⑥ Persist

​⑦ Enrich and classify

​⑧ Decide

​What leaves your account

​Model weights are a different kind of egress

​Query path

​Resource map

​Where it breaks

​What this page does not establish

Deployment models

Single tenant

The boundary, first

Stage by stage

① Arrival

② Object

③ Batch

④ Schedule

⑤ Execute

⑥ Persist

⑦ Enrich and classify

⑧ Decide

What leaves your account

Model weights are a different kind of egress

Query path

Resource map

Where it breaks

What this page does not establish