Semantic Join
Bridge extracted content features with business reference data. Join video clips to product catalogs, detected faces to employee directories, or documents to compliance frameworks-all via embedding similarity.
"Find marketing videos featuring products from our electronics catalog with matched SKUs"
Why This Matters
Better search isn't about better embeddings-it's about connecting extracted content to existing business systems. Query by product taxonomy, not embedding distance.
import requestsfrom mixpeek import Mixpeekclient = Mixpeek(api_key="YOUR_API_KEY", namespace="catalog-join")API = "https://api.mixpeek.com/v1"HEADERS = {"Authorization": "Bearer YOUR_API_KEY", "X-Namespace": "catalog-join"}# 1. The product catalog: one photo per product, with its SKU and category as metadatacatalog_bucket = client.buckets.create(bucket_name="product-catalog",bucket_schema={"properties": {"photo": {"type": "image"}}},)catalog = client.collections.create(collection_name="product-catalog",source={"type": "bucket", "bucket_ids": [catalog_bucket["bucket_id"]]},feature_extractor={"feature_extractor_name": "multimodal_extractor","version": "v1",},)client.buckets.upload(catalog_bucket["bucket_id"],blobs=[{"property": "photo", "type": "image", "data": "s3://your-bucket/catalog/sku-001.jpg"}],metadata={"sku": "SKU-001", "category": "Electronics"},)client.collections.trigger(catalog["collection_id"])# 2. The retriever that finds the closest catalog productmatcher = client.retrievers.create(retriever_name="product-matcher",collection_identifiers=["product-catalog"],input_schema={"query": {"type": "text", "required": True}},stages=[{"stage_name": "search","stage_id": "feature_search","parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding","query": {"input_mode": "text", "value": "{{INPUT.query}}"},"top_k": 3,},],"final_top_k": 3,},},],)# 3. A flat taxonomy is the join: each match copies the product's SKU and category onto# the matching document. The SDK has no taxonomies resource, so this part is REST.taxonomy = requests.post(API + "/taxonomies", headers=HEADERS, json={"taxonomy_name": "product-matcher","config": {"taxonomy_type": "flat","retriever_id": matcher["retriever_id"],"input_mappings": [{"input_key": "query", "source_type": "payload", "path": "description"}],"source_collection": {"collection_id": catalog["collection_id"],"enrichment_fields": [{"field_path": "metadata.sku", "target_field": "matched_sku", "merge_mode": "replace"},{"field_path": "metadata.category", "target_field": "matched_category", "merge_mode": "replace"},],},},}).json()# 4. Marketing video scenes are described, then joined to the catalog as they are writtenvideo_bucket = client.buckets.create(bucket_name="marketing-videos",bucket_schema={"properties": {"video": {"type": "video"}}},)videos = client.collections.create(collection_name="marketing-videos",source={"type": "bucket", "bucket_ids": [video_bucket["bucket_id"]]},feature_extractor={"feature_extractor_name": "multimodal_extractor","version": "v1","parameters": {"split_method": "scene","run_video_description": True,},},taxonomy_applications=[{"taxonomy_id": taxonomy["taxonomy_id"], "execution_mode": "materialize"}],)client.buckets.upload(video_bucket["bucket_id"],blobs=[{"property": "video", "type": "video", "data": "s3://your-bucket/marketing/spring-launch.mp4"}],)client.collections.trigger(videos["collection_id"])# 5. Search the videos; results carry the joined product fieldssearch = client.retrievers.create(retriever_name="video-search",collection_identifiers=["marketing-videos"],input_schema={"query": {"type": "text", "required": True}},stages=[{"stage_name": "search","stage_id": "feature_search","parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding","query": {"input_mode": "text", "value": "{{INPUT.query}}"},"top_k": 20,},],"final_top_k": 20,},},],)results = client.retrievers.execute(search["retriever_id"], inputs={"query": "product demos"})for doc in results["documents"]:print(doc["document_id"], doc.get("matched_sku"), doc.get("matched_category"))
Feature Extractors
Multimodal Extractor
Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.
Retriever Stages
feature search
Search and filter documents by vector similarity using feature embeddings
Related Blog Posts
Documentation
Related Recipes & Resources
Explore these related resources to deepen your understanding and discover more powerful features
Multimodal Extractor
Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.
Multimodal RAG
Retrieval-augmented generation across video, images, and text. Retrieve relevant multimodal context, then pass to your LLM with citations back to source timestamps and frames.
Semantic Multimodal Search
Unified semantic search across all content types. Query by natural language and retrieve relevant video clips, images, audio segments, and documents based on meaning-not keywords or manual tags.
Hierarchical Classification
Assign content to multi-level category hierarchies using embedding-based classification. Define your taxonomy once, then classify new content automatically with confidence scores.
Video Content Analytics Pipeline
Analyze video content at scale to extract insights: scene composition, speaker time, topic distribution, and sentiment across your video library.
Feature Extraction
Multi-tier feature extraction that decomposes content into searchable components: embeddings, transcripts, detected objects, OCR text, scene boundaries, and more. The foundation for all downstream retrieval and analysis.