What is Collection

How It Works

You declare a collection against a source, name the feature extractors that should run, and the platform processes each object into one or more documents. A single video object might become many documents, one per scene, each with its own transcript span, on-screen text, thumbnail and embeddings. Because the collection is derived rather than authored, it can be rebuilt: change the extractor configuration and re-run, and the documents are regenerated from the same untouched source objects.

Technical Details

A collection's source can be a bucket or another collection, which is what lets pipelines chain: extract once, then build a narrower or enriched collection on top without re-reading the original media. Where several sources have different field shapes, a field map declares which fields to keep and what to call them in the merged view. Documents keep lineage back to the object and bucket they came from, which is what makes deletion, entitlement filtering and debugging possible after the merge. Extraction is asynchronous and batched, so objects land immediately and documents appear when the collection next runs.

Common Pitfalls

• Expecting a document to be queryable the instant it is written, when extraction is asynchronous and read-after-write is not immediate
• Writing documents directly instead of through the pipeline, which increments the count without ever embedding anything, so the document never becomes searchable
• Assuming one object equals one document, when a video or a long PDF becomes many
• Merging sources with different schemas without declaring a field map, or declaring one where two sources claim the same target field
• Losing lineage in a merged collection and then being unable to honour a takedown or an entitlement rule that applies to only one source

Relevance to Multimodal Systems

The collection is where multimodal work happens. Text alone rarely needs the distinction, because a chunk of text is close to its own searchable representation. A two-hour video is not: it has to be segmented, transcribed, read for on-screen text, embedded across several models, and only then is there something to rank. The collection is the object that holds all of that and presents it as one searchable set. In Mixpeek it is one of the core primitives alongside buckets, namespaces, extractors, retrievers, taxonomies and clusters. See collections for the capability, the ingestion documentation for the API, and merging sources with different schemas for the field-map case.