Contextual video matching

A publisher’s video player sits inside an article page. For each article, it should play the videos that fit the story and are safe for the advertisers on the page. This guide builds that on Mixpeek:

Videos sync from your S3 bucket and are embedded and described.
Articles arrive through the API with their text.
A labeling retriever writes IAB categories, entities, sentiment and a brand-safety rating onto every article and video at ingest.
A playlist retriever takes an article and returns ranked, brand-safe videos.
Your player calls it behind a feature flag, keeps its own playlist as the fallback, and sends viewer events back.

Every resource below can live in one manifest, so dev, staging and production run the same configuration (environment branching).

Prerequisites

An S3 bucket of video files and an IAM role Mixpeek can assume, set up as in AWS S3.
A Mixpeek API key and a namespace. Requests below use $MP_API_KEY and $MP_NAMESPACE.

1. Bring in the video library

Create a videos bucket whose schema holds the file and the metadata your player filters on:

curl -sS -X POST "https://api.mixpeek.com/v1/buckets" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket_name": "videos",
    "bucket_schema": {
      "properties": {
        "video":        { "type": "video" },
        "video_id":     { "type": "string", "required": true },
        "title":        { "type": "string" },
        "publisher_id": { "type": "string" },
        "channel_id":   { "type": "string" },
        "is_shared":    { "type": "boolean" },
        "breaking":     { "type": "boolean" },
        "published_at": { "type": "datetime" }
      }
    }
  }'

Then sync your S3 prefix into it. Upload each video with its metadata as x-amz-meta-* headers, and map those keys to the bucket fields with metadata sources, as shown in Object metadata. A continuous sync with a 60-second poll picks up a new upload within a minute. For the existing catalog, run the backfill at a lower priority than new uploads. See Backfill and one-at-a-time uploads.

2. Embed and describe the videos

A collection with multimodal_extractor v2 embeds each video with Gemini Embedding 2. With run_video_description, Gemini also watches and listens to the video and writes a text description, which the labeling step reads:

curl -sS -X POST "https://api.mixpeek.com/v1/collections" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{
    "collection_name": "video-signals",
    "source": { "type": "bucket", "bucket_ids": ["bkt_videos"] },
    "feature_extractor": {
      "feature_extractor_name": "multimodal_extractor",
      "version": "v2",
      "input_mappings": { "video": "video" },
      "field_passthrough": [
        { "source_path": "video_id" }, { "source_path": "title" },
        { "source_path": "publisher_id" }, { "source_path": "channel_id" },
        { "source_path": "is_shared" }, { "source_path": "published_at" }
      ],
      "parameters": {
        "run_multimodal_embedding": true,
        "run_video_description": true,
        "description_prompt": "Describe what this news video shows and says. Name every person, organisation, place, event and product seen or heard, and copy any on-screen text.",
        "split_method": "time",
        "time_split_interval": 120
      }
    }
  }'

field_passthrough copies the bucket fields onto each document, so the playlist can filter on them. See Multimodal extractor for every parameter.

3. Send articles

Create an articles bucket with article_id, title, body and publisher_id, and an article-signals collection on it with the same extractor and "input_mappings": { "text": "body" }. Articles and videos are then embedded in the same vector space, so an article can search the videos directly. Send each article as an object, one blob per field, with its text in body. Strip the page markup before you send it: the text extractor reads body as plain text.

curl -sS -X POST "https://api.mixpeek.com/v1/buckets/bkt_articles/objects" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{
    "blobs": [
      { "property": "article_id",   "type": "text", "data": "a_5521" },
      { "property": "title",        "type": "text", "data": "City council approves new stadium" },
      { "property": "publisher_id", "type": "text", "data": "pub_123" },
      { "property": "body",         "type": "text", "data": "The council voted 7 to 2 on Tuesday to ..." }
    ]
  }'

Submit the new object in a batch, and pass a webhook_url or subscribe to collection.documents.batch_completed to learn when its document is ready (Webhooks).

4. Label articles and videos at ingest

The signals come from a retriever that runs once on every new document, as a retriever enrichment. Its first stage selects the document being labeled. Its second is an llm_enrich stage that returns the labels as structured output:

{
  "retriever_name": "label-content",
  "collection_identifiers": ["article-signals", "video-signals"],
  "input_schema": { "document_id": { "type": "string", "required": true } },
  "stages": [
    {
      "stage_name": "this_document",
      "stage_type": "filter",
      "config": {
        "stage_id": "attribute_filter",
        "parameters": { "field": "document_id", "operator": "eq", "value": "{{INPUT.document_id}}" }
      }
    },
    {
      "stage_name": "label",
      "stage_type": "enrich",
      "config": {
        "stage_id": "llm_enrich",
        "parameters": {
          "provider": "google",
          "model_name": "gemini-2.5-flash-lite",
          "prompt": "Label this news content for contextual video advertising. Title: {{DOC.title}}. Text: {{DOC.body}} {{DOC.description}}",
          "output_field": "labels",
          "output_schema": {
            "type": "object",
            "properties": {
              "iab_categories": { "type": "array", "items": { "type": "string" },
                                  "description": "1 to 3 IAB Content Taxonomy 3.1 paths, best fit first" },
              "entities":       { "type": "array", "items": { "type": "string" },
                                  "description": "Named people, organisations, places, events and products" },
              "sentiment":      { "type": "string", "enum": ["positive", "neutral", "negative"] },
              "brand_safety":   { "type": "string", "enum": ["safe", "sensitive", "unsafe"],
                                  "description": "GARM brand safety floor and suitability" }
            }
          }
        }
      }
    }
  ]
}

Attach it to both collections. Each run writes the label fields back onto the document it labeled:

{
  "retriever_enrichments": [
    {
      "retriever_id": "ret_label_content",
      "input_mappings": [
        { "input_key": "document_id", "source": { "source_type": "document_field", "path": "document_id" } }
      ],
      "write_back_fields": [
        { "source_field": "labels.iab_categories", "target_field": "iab_categories", "mode": "first" },
        { "source_field": "labels.entities",       "target_field": "entities",       "mode": "first" },
        { "source_field": "labels.sentiment",      "target_field": "sentiment",      "mode": "first" },
        { "source_field": "labels.brand_safety",   "target_field": "brand_safety",   "mode": "first" }
      ]
    }
  ]
}

Put the full IAB 3.1 list in the prompt, so the model copies a category rather than inventing one. Articles and videos get the same field names, so the playlist can compare them.

5. Build the playlist retriever

The playlist retriever takes an article’s document_id and searches the videos with that article’s own embedding, using input_mode: "document". No new inference runs at query time. The search filter keeps brand-unsafe videos out and applies the publisher’s rights: its own videos or the shared library, minus channels it has excluded (filter example). Pass excluded_channels as [] when a publisher excludes none.

{
  "retriever_name": "playlist",
  "collection_identifiers": ["video-signals"],
  "input_schema": {
    "article_document_id": { "type": "string", "required": true },
    "publisher_id":        { "type": "string", "required": true },
    "excluded_channels":   { "type": "array", "required": true }
  },
  "stages": [
    {
      "stage_name": "candidates",
      "stage_type": "filter",
      "config": {
        "stage_id": "feature_search",
        "parameters": {
          "searches": [
            {
              "feature_uri": "mixpeek://multimodal_extractor@v2/gemini-embedding-2",
              "query": {
                "input_mode": "document",
                "document_ref": { "collection_id": "article-signals", "document_id": "{{INPUT.article_document_id}}" }
              },
              "top_k": 100,
              "filters": {
                "AND": [
                  { "field": "brand_safety", "operator": "ne", "value": "unsafe" },
                  { "OR": [
                    { "field": "publisher_id", "operator": "eq", "value": "{{INPUT.publisher_id}}" },
                    { "field": "is_shared", "operator": "eq", "value": true }
                  ] },
                  { "NOT": { "field": "channel_id", "operator": "in", "value": "{{INPUT.excluded_channels}}" } }
                ]
              }
            }
          ],
          "final_top_k": 100
        }
      }
    },
    {
      "stage_name": "dedupe_titles",
      "stage_type": "reduce",
      "config": { "stage_id": "deduplicate", "parameters": { "strategy": "field", "fields": ["title"] } }
    },
    {
      "stage_name": "top5",
      "stage_type": "reduce",
      "config": { "stage_id": "limit", "parameters": { "limit": 5 } }
    }
  ]
}

To rank on more than vector similarity, add stages between search and limit. A document_enrich stage can attach the article’s entities and IAB categories to each candidate. A code_execution stage can then score shared entities and IAB depth, and sort_attribute can order by that score.

6. Call it from the player, behind a flag

Execute the retriever with the article’s document id. Keep your current playlist as the fallback: use it when the flag is off, the call fails or times out, the run is degraded, or fewer videos come back than the player needs.

async function playlistFor(article, fallback) {
  if (!flags.mixpeekPlaylist) return fallback;
  try {
    const res = await fetch(`https://api.mixpeek.com/v1/retrievers/${RETRIEVER_ID}/execute`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${MP_API_KEY}`,
        "X-Namespace": MP_NAMESPACE,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        inputs: {
          article_document_id: article.documentId,
          publisher_id: article.publisherId,
          excluded_channels: article.excludedChannels,
        },
      }),
      signal: AbortSignal.timeout(800),
    });
    if (!res.ok) return fallback;
    const run = await res.json();
    if (run.status !== "completed" || run.documents.length < 3) return fallback;
    return { videos: run.documents, executionId: run.execution_id };
  } catch {
    return fallback;
  }
}

Read documents, check status, and log warnings. A run that skips a stage answers 200 with status: "degraded", so a short list is not always a lack of matches. Keep execution_id: the next step needs it.

7. Send viewer and editor signals back

Record what happened to each video as an interaction, with the execution_id and retriever_id of the run that chose it: impression when it is shown, play, long_view when watched to the end, and skip. During a pilot, record editors’ verdicts on a match as positive_feedback or negative_feedback. These rows land in your namespace’s system collections, joined to the query that produced them. Use them to build an evaluation set, compare ranking configurations, and roll a new one out to a share of traffic with auto-tune rollout.

Next steps

Manifests

Declare the buckets, collections and retrievers above in one file.

Batch ingestion at scale

Size the backfill and keep new uploads ahead of it.

Filters

Every operator for rights, recency and brand-safety rules.

Reward signals

How plays and completions feed ranking.

​Prerequisites

​1. Bring in the video library

​2. Embed and describe the videos

​3. Send articles

​4. Label articles and videos at ingest

​5. Build the playlist retriever

​6. Call it from the player, behind a flag

​7. Send viewer and editor signals back

​Next steps