Contextual video match
An article goes in, a ranked playlist of video comes out, and every match carries the entities, keywords and categories that produced it. Both sides are embedded twice, once by a text model and once by a multimodal one, so nothing is embedded at query time and a clip in another language still matches.
Match video to an article by what the article is ABOUT, not by the words it shares. The hierarchy the buyer cares about is entity first, then keyword, then IAB category, and the retriever returns the overlap on each of the three beside every result, so an editor can read why a clip was chosen.
Ad platforms and publishers placing video against editorial pages
Studio will also guide you to Your own video library and a handful of live article URLs for zero-setup evaluation, then to a continuous data connection for production.
- videosOne object per video. The template cuts each into segments and indexes transcription, on-screen text and a multimodal embedding per segment.
- articlesOne object per page: the article text, plus article_url, published_at and tenant_id as fields. The tenant field is what a query-time scope filter reads.
- video-scenesSegment documents with a 1408-dimension multimodal embedding, timing, transcription, OCR text, and the entities, keywords and IAB categories read off the segment.
- scene-transcriptsThe spoken side of each segment as a 1024-dimension text embedding, so a text-first query reaches it directly.
- article-multimodalThe article in the same 1408-dimension space as the video, which is what makes the cross-modal half of the match a document reference rather than a query embedding.
- article-textThe article as a 1024-dimension text embedding, with its URL, publish time and tenant carried as fields.
How the namespace is wired
2 buckets, 4 collections, 0 clean views, 1 retrievers. The diagram generates the manifest below; they cannot drift apart.
Reward signals
How reviewer decisions move the thresholds
Thresholds at ingest drift as the corpus changes. The reviewers working the queue are the ones who see where a threshold is wrong first, so this template routes their decisions back into the model that set it.
None. No editor thumbs, no accepted-playlist marks are recorded by the template as shipped.
None from the application. Mixpeek records retriever executions server-side, which is telemetry about queries rather than feedback about which match was right.
mxp_retriever_executionsSystem collections in your namespace, on the same vector store as the rest of the template. They are yours to query.
The loop that would close it is the customer's own answer key: a set of articles with the playlist they consider correct. That is an evaluation input rather than a signal the template can collect on its own.
One file spins up the namespace. Generated from the diagram above. Also served at /templates/contextual-video-match.namespace.yaml.
# contextual-video-match: one manifest spins up the namespace.
# Platform manifest schema (GET /v1/discovery/schema). Validate with POST /v1/manifest/validate,
# apply with POST /v1/manifest/apply or the Deploy button. Wiring comes from the flow diagram:
# edges are bucket -> collection sources, collection -> retriever scope, retriever -> view.
version: '1.0'
metadata:
name: contextual-video-match
description: "Namespace template contextual-video-match. Generated from the flow diagram on mixpeek.com/templates/contextual-video-match."
namespaces:
- name: contextual-video-match
description: "Everything below lives in this namespace."
feature_extractors:
- name: multimodal_extractor
version: v1
- name: text_extractor
version: v1
# Data sources. A storage connection carries credentials, so it is created in Studio
# (or POST /v1/organizations/storage-connections) and synced into the bucket named here.
# video-corpus: s3, continuous, the video library you want matched against, in your account -> bucket videos
# article-feed: manual, continuous, one object per page: the article text, its URL, its publish time and the tenant it belongs to -> bucket articles
buckets:
- name: videos
namespace: contextual-video-match
description: "Fed by video-corpus (s3, continuous)."
schema:
properties:
content:
type: video
- name: articles
namespace: contextual-video-match
description: "Fed by article-feed (manual, continuous)."
schema:
properties:
content:
type: text
# Connect your own storage. Create the two secrets, uncomment, and apply again;
# apply live-tests the connection, so it must have real credentials to succeed.
# Until then the buckets above accept direct uploads.
# storage_connections:
# - name: video-corpus-connection
# provider: s3
# description: "Read-only access to the s3 location holding the video library you want matched against, in your account."
# config:
# region: us-east-1
# credentials:
# type: access_key
# access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
# secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
# bucket_syncs:
# - name: video-corpus-sync
# bucket: videos
# connection: video-corpus-connection
# source_path: "videos/"
# sync_mode: continuous
# polling_interval_seconds: 300
# skip_duplicates: true
# file_filters:
# include_patterns: ["*.mp4", "*.mov", "*.webm"]
# schema_mapping:
# mappings:
# content:
# target_type: blob
# source: {type: file}
# blob_type: video
collections:
- name: video-scenes
namespace: contextual-video-match
description: "multimodal_extractor@v1 over bucket videos. Feeds scene-transcripts, contextual-match."
source:
type: bucket
bucket: videos
feature_extractor:
name: multimodal_extractor
version: v1
input_mappings:
video: content
field_passthrough:
- source_path: file_location
- source_path: segment_id
- source_path: start_time
- source_path: end_time
- source_path: tenant_id
enabled: true
- name: article-multimodal
namespace: contextual-video-match
description: "multimodal_extractor@v1 over bucket articles. Feeds contextual-match."
source:
type: bucket
bucket: articles
feature_extractor:
name: multimodal_extractor
version: v1
input_mappings:
text: content
field_passthrough:
- source_path: article_url
- source_path: published_at
- source_path: tenant_id
enabled: true
- name: scene-transcripts
namespace: contextual-video-match
description: "text_extractor@v1 over collection video-scenes. Feeds contextual-match."
source:
type: collection
collection: video-scenes
feature_extractor:
name: text_extractor
version: v1
input_mappings:
text: transcription
field_passthrough:
- source_path: segment_id
- source_path: video_segment_url
- source_path: start_time
- source_path: end_time
enabled: true
- name: article-text
namespace: contextual-video-match
description: "text_extractor@v1 over bucket articles. Feeds contextual-match."
source:
type: bucket
bucket: articles
feature_extractor:
name: text_extractor
version: v1
input_mappings:
text: content
field_passthrough:
- source_path: article_url
- source_path: published_at
- source_path: tenant_id
enabled: true
retrievers:
- name: contextual-match
namespace: contextual-video-match
description: "Searches scene-transcripts, video-scenes, article-text, article-multimodal across 2 feature indexes with rrf fusion."
collections:
- scene-transcripts
- video-scenes
- article-text
- article-multimodal
input_schema:
query:
type: text
required: true
description: "What to look for; searched across every index below"
stages:
- stage_name: search
stage_id: feature_search
parameters:
searches:
- feature_uri: "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
query:
input_mode: text
value: "{{INPUT.query}}"
top_k: 30
- feature_uri: "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding"
query:
input_mode: text
value: "{{INPUT.query}}"
top_k: 30
fusion: rrf
final_top_k: 30
tags:
- template:contextual-video-match