Agentic Audiences contextual embeddings from video: building the ext.aa segment

What this guide builds

Agentic Audiences lets a seller send an embedding inside an OpenRTB bid request instead of a list of segment names. The OpenRTB community extension puts it at BidRequest.user.data[].segment[].ext.aa, with five fields: ver, vector (Base64 of little-endian float32 values), dimension, model, and type, an array where 2 means contextual (OpenRTB Agentic Audiences extension). Prebid.js merged a module for it in April 2026 (Prebid Agentic Audiences module).

Most examples assume text: embed the page and send the vector. This guide does it for video. It takes the per-scene embeddings Mixpeek already stores for a video, turns them into one contextual vector for the whole video or one for the scene playing at an ad break, encodes the envelope, and validates it the way a buyer should. Every step below ran on 2026-10-19 against a real Mixpeek collection; the numbers in What we measured come from that run.

Where does a contextual embedding go in the bid request?

In user.data, even though it describes the content. The extension defines one placement for all three signal types (1 identity, 2 contextual, 3 reinforcement), and type says which one a segment carries:

{
  "user": {
    "data": [{
      "name": "mixpeek",
      "segment": [{
        "id": "video",
        "name": "contextual-video",
        "ext": {
          "aa": {
            "ver": "1.0.0",
            "vector": "yXy4vAMaZT2vaFM72eEvPB4G...",
            "dimension": 1408,
            "model": "google-vertex-multimodal-1408@2026-10",
            "type": [2]
          }
        }
      }]
    }]
  }
}

type is an array because one entry may encode several signal types. A contextual-only segment sends [2]. The vector above is cut short; the full value is 7,512 characters.

Step 1: index the video so each scene has an embedding

Upload the videos to a bucket and process them with a collection that uses the multimodal extractor with split_method: "scene", so each scene is embedded separately (Mixpeek docs: multimodal extractor). Each scene becomes a document with start_time, end_time and the embedding under the vector's name. The collection used here stores vertex_multimodal_embedding, 1,408 dimensions, one vector per scene, already unit length.

Step 2: read the scene vectors back

List the video's documents with return_vectors=true (list documents). Embedding values come back as root fields named after the vector, and _vectors names which fields those are:

import json, os, urllib.parse, urllib.request

def scene_documents(collection_id, namespace, source_page):
    params = {
        "return_vectors": "true",
        "page_size": "100",
        "filters": json.dumps({"AND": [{"field": "source_page", "operator": "eq", "value": source_page}]}),
    }
    req = urllib.request.Request(
        f"https://api.mixpeek.com/v1/collections/{collection_id}/documents?{urllib.parse.urlencode(params)}",
        headers={"Authorization": f"Bearer {os.environ['MIXPEEK_API_KEY']}", "X-Namespace": namespace},
    )
    return json.load(urllib.request.urlopen(req, timeout=60))["results"]

Request vectors without select. On 2026-10-19, adding select to the same call returned the documents with _vectors: [] and no embedding values, even with the vector's name in the list.

Step 3: choose one vector per video, or one per ad break

A 20-second video gave four scenes. You can send two kinds of contextual vector:

The whole video. Average the unit-length scene vectors and normalize the result. Use this when the ad runs before the content (pre-roll) or when the player cannot tell you where the break falls.

The scene at the break. For a mid-roll at time *t*, send the scene playing at *t*, or the last one that ended before it. This describes what the viewer was watching when the ad starts, which is the point of a contextual signal.

import math

def l2_normalize(v):
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]

def mean_pool(vectors):
    units = [l2_normalize(v) for v in vectors]
    return l2_normalize([sum(col) / len(units) for col in zip(*units)])

def scene_at(scenes, t):
    playing = [s for s in scenes if s["start_time"] <= t < s["end_time"]]
    if playing:
        return playing[0]
    return max((s for s in scenes if s["end_time"] <= t), key=lambda s: s["end_time"])

Step 4: encode the vector exactly as the extension says

Little-endian float32, concatenated, standard Base64. In Python that is one struct.pack call:

import base64, struct

def encode_vector(values):
    return base64.b64encode(struct.pack(f"<{len(values)}f", *values)).decode("ascii")

def decode_vector(b64):
    raw = base64.b64decode(b64, validate=True)
    if len(raw) % 4:
        raise ValueError("byte length must be a multiple of 4")
    return list(struct.unpack(f"<{len(raw) // 4}f", raw))

Check it against the extension's own worked example before you trust it: encoding the ten floats [1.2345678, -2.5, 0.0, 3.1415927, 12345.678, -0.00012345, 42.42, -999.999, 0.000001, 987654.25] must give UQaePwAAIMAAAAAA2w9JQLbmQEZbcgG5FK4pQvD/ecS9N4Y1ZCBxSQ==. The function above does, byte for byte. A big-endian or float64 encoder produces a valid-looking Base64 string that decodes to garbage, so this test is worth keeping in CI.

Step 5: build the envelope

def build_segment(vector, model, seg_id, name):
    return {
        "id": seg_id,
        "name": name,
        "ext": {"aa": {
            "ver": "1.0.0",
            "vector": encode_vector(vector),
            "dimension": len(vector),
            "model": model,
            "type": [2],
        }},
    }

Pick the model string carefully, because it is the only thing that tells a buyer whether your vector can be compared with theirs. Include the model name, the dimension and a version you control, such as google-vertex-multimodal-1408@2026-10. If you change the model, the dimension or anything that moves vectors (a different input resolution, a different pooling rule), change the string and re-embed. A vector from one model compared with a vector from another produces a number with no meaning, and nothing in the payload will warn the buyer.

The Agentic Audiences repository's fuller embedding format describes the model with id, version, type, architecture, dimension, metric and embedding_space_id (embedding format schema). The OpenRTB envelope carries only model and dimension, so publish the rest wherever your buyers look up your model IDs.

Step 6: validate before you trust a segment

The extension asks consumers to check that dimension equals the decoded byte length divided by 4 and to reject anything else. A buyer should also refuse models it does not know and dimensions that do not match the model:

def validate_segment(seg, expected):  # expected: {model_id: dimension}
    aa = seg["ext"]["aa"]
    if aa["model"] not in expected:
        raise ValueError(f"unknown model {aa['model']}")
    if 2 not in aa["type"]:
        raise ValueError("not a contextual segment")
    vec = decode_vector(aa["vector"])
    if aa["dimension"] != len(vec):
        raise ValueError(f"dimension {aa['dimension']} but payload holds {len(vec)} values")
    if aa["dimension"] != expected[aa["model"]]:
        raise ValueError("dimension does not match the model")
    return vec

These are the same gaps reported as open issues on the Agentic Audiences repository in September 2026: a declared dimension that is never checked against the vector, and model compatibility matched by name only (agentic-audiences issues).

What we measured

On 2026-10-19 we ran the steps above on one video from a Mixpeek sample collection, an aerial shot of a forest waterfall split into four scenes (0 to 7 s, 7 to 13 s, 13 to 19 s, 19 to 20.5 s), with 1,408-dimension vertex_multimodal_embedding vectors:



The pooled vector stays close to every scene of its own video and well away from unrelated footage. This is one video and one comparison, so read it as a sanity check on the pipeline, not a benchmark. In a longer video whose scenes change subject, the scene vectors should spread further apart than these four did, and the break-time scene is then the one that describes the moment. Measure that on your own catalog before choosing between the two.

How big does the bid request get?

Base64 turns every 3 bytes into 4 characters, and each float32 is 4 bytes:



The extension says real embeddings are typically 256 to 1,024 values. A 1,408-dimension vector sits above that range. Do not shrink it by cutting off the tail: truncating a vector only keeps its meaning if the model was trained for it (Matryoshka-style training), and you would need a new model string either way. If the model offers a smaller output size, request that size from the model and publish it as a separate model ID.

Where Mixpeek fits

Mixpeek does the indexing: splitting video into scenes, embedding each scene with a multimodal model, and storing the vectors with their timestamps so you can read them back by video and by time. The encoding and validation above are plain Python with no Mixpeek dependency. Mixpeek does not generate Agentic Audiences segments for you, and does not sit in the bid path. Mixpeek has also proposed a reference contextual encoder for the Agentic Audiences repository as an open pull request (agentic-audiences PR #11); it is not merged. If you also need IAB Content Taxonomy categories for the same content, see IAB Content Taxonomy classification, and for the other agentic standards, AdCP, AAMP, ARTF and Agentic Audiences: what each one does.

Changelog

2026-10-19: first version.

Frequently Asked Questions

What does type 2 mean in Agentic Audiences?

Contextual: what the user is doing right now, such as the page or video they are watching. 1 is identity and 3 is reinforcement (how the user responds to ads). type is an array, so a contextual-only segment sends [2] (OpenRTB Agentic Audiences extension).

How is the vector encoded in ext.aa?

Each value is written as a 4-byte little-endian IEEE 754 float32, the bytes are concatenated, and the result is standard Base64 (RFC 4648). dimension must equal the decoded byte length divided by 4.

Which embedding model should I use for contextual Agentic Audiences segments?

The extension does not name one; it carries whatever model ID you set, and similarity only means something between vectors from the same model. Agree the model and dimension with the buyers who will score your vectors, and version the ID so a model change cannot be mistaken for the old one.

Can I send one embedding for a whole video?

Yes. Average the unit-length scene embeddings and normalize the result. For a mid-roll, sending the scene at the ad break describes the moment better than the whole-video average.

Why is a contextual embedding in user.data and not site.content?

That is where the OpenRTB extension places every Agentic Audiences segment, whatever its type. The type array, not the location, says the segment is contextual.

Sources

OpenRTB Agentic Audiences community extension

Agentic Audiences repository

Agentic Audiences embedding format schema

Agentic Audiences issues

Prebid Agentic Audiences module

Prebid.js PR 14626, merged 2026-04-14

Mixpeek docs: list documents