NEWVectors or files. Pick a path.Start →
    Provider APIEmbeddingVerified 2026-11-05

    Vertex AI multimodalembedding@001

    multimodalembedding@001

    Vertex AI multimodalembedding@001 is a embedding model. Called on Mixpeek through the provider API.

    Mixpeek calls the provider's API by name. Provider terms apply.

    Use in a new namespaceAdd to an existing namespace Record as JSON

    New opens the Designer in Studio. For an existing namespace, open it in Studio and use its Designer tab.

    No demo for Vertex AI multimodalembedding@001 yet

    A video, image, slideshow or playground goes here once we record one on our own data.

    What it is good and bad at

    Best at
    Category-level text-to-video and text-to-image search in one 1408-d space (committed mAP 0.855 on a 10-video text eval).
    Worst at
    Telling instances apart: the TubeScience re-ID test linked across categories (recall 0.630 at 0.059 false), where DINOv2/v3 separated them.
    Task
    Embedding
    Input
    Image, Video, Text
    Output
    Vector
    Use cases
    Visual search, Text search, Video understanding

    Templates that use it

    • AI data curation

      Find the data your model is missing. Turn model failures into training and evaluation datasets with multimodal search and sampling.

      Deploy this template
    • Contextual video match

      An article goes in, a ranked playlist of video comes out, and every match carries the entities, keywords and categories that produced it. Both sides are embedded twice, once by a text model and once by a multimodal one, so nothing is embedded at query time and a clip in another language still matches.

      Deploy this template
    • Creative DNA for Ad Archives

      Search every scene of every ad you have run, and find the ones that keep earning spend

      Deploy this template
    • Footage Intelligence at Archive Scale

      Millions of scenes you can move through instead of search. Footage, ads and editor assets land in three clean collections, and scenes are clustered one partition at a time so every run stays under the sample-size caps.

      Deploy this template
    • UX session analysis

      Moderated sessions become a corpus you can question. The transcript is joined to what was on screen at that second, so one query reaches every session at once and returns the moment rather than the recording.

      Deploy this template

    Input and output

    Input

    image
    image bytes or GCS URI
    video
    video segment (one vector per segment)
    text
    text query

    Output

    vector
    dims
    1408
    distance
    cosine

    How Mixpeek wires it

    Extractors
    • multimodal_extractor@v1 fixed
    Feature URIs
    • mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding
    Vector indexes
    multimodal_extractor_v1_multimodal_embedding
    Retriever stages
    feature_search
    Combines with
    • multimodal_extractor v1: video segment vector beside the E5 transcription vector; v2 replaced it with Gemini Embedding 2

    How it runs

    Status
    Provider API Vertex AI API. multimodal_extractor v1 calls it from VertexMultimodalEmbeddingBatch; query embeds go through universal__api (model multimodalembedding).
    Serve apps
    universal__api
    Runtime
    Vertex AI MultiModalEmbeddingModel (vertexai SDK)
    Weights
    Provider API, no remote code
    Parameters
    Not recorded
    Size on disk
    Not recorded
    Preprocessing
    Modality detected per row; video cut into segments by ffmpeg before the call.
    Postprocessing
    Vector as returned (1408 dims).
    Serving
    both
    Hardware
    Provider API

    Scale to zero

    Minimum replicas
    0
    Maximum replicas
    6
    Idle timeout
    600 s
    Memory
    Not recorded
    CPUs
    0.1
    GPUs
    0
    Cold start, node warm
    Not recorded
    Cold start, new node
    Not recorded

    Measured cost

    Unmeasured

    No committed full measurement. customers/primis/BENCHMARK_RESULTS.md notes ~7 s first call and 1.3 to 2.1 s warm queries end to end, not the model alone.

    Our evaluations

    Dataset and metricScoreBaselineDate
    TubeScience instance-retrieval text eval, 10 videos
    mAP, text-to-instance search
    customers/tubescience/archived/instance-retrieval/SOTA_EVAL_RESULTS.md
    		0.855SigLIP2 base on SAM2 crops: 0.292026-09-09
    TubeScience cross-video instance re-ID (same dog in 3 clips + cat + person)
    best recall with at most 0.10 cross-category false links (false rate 0.059)
    customers/tubescience/archived/instance-retrieval/SOTA_EVAL_RESULTS.md
    		0.63None2026-09-09

    Questions

    What is Vertex AI multimodalembedding@001 best at?
    Category-level text-to-video and text-to-image search in one 1408-d space (committed mAP 0.855 on a 10-video text eval).
    What are the limits of Vertex AI multimodalembedding@001?
    Telling instances apart: the TubeScience re-ID test linked across categories (recall 0.630 at 0.059 false), where DINOv2/v3 separated them.
    Can Vertex AI multimodalembedding@001 be used commercially?
    Yes. Paid Google Cloud API under Google's terms.
    How does Vertex AI multimodalembedding@001 run on Mixpeek?
    Vertex AI API. multimodal_extractor v1 calls it from VertexMultimodalEmbeddingBatch; query embeds go through universal__api (model multimodalembedding).
    What does Vertex AI multimodalembedding@001 output?
    A 1408-dimensional dense vector, compared by cosine similarity.