What is Mixpeek?

[Back to Blog](/blog)

[](https://twitter.com/intent/tweet?text=%F0%9F%A7%A0%20Multimodal%20Monday%20%231%20-%20State%20of%20the%20Stack&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-state-of-the-stack)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-state-of-the-stack&title=%F0%9F%A7%A0%20Multimodal%20Monday%20%231%20-%20State%20of%20the%20Stack)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-state-of-the-stack&title=%F0%9F%A7%A0%20Multimodal%20Monday%20%231%20-%20State%20of%20the%20Stack)

This was originally in our newsletter.

Get Updates

Apr 2, 2025•5 min read

# 🧠 Multimodal Monday #1 - State of the Stack

Researchers introducing new methods to replace embeddings with discrete IDs for faster cross-modal search

![🧠 Multimodal Monday #1 - State of the Stack](https://blog.mixpeek.com/content/images/2025/04/mm-1.png)

Multimodal Monday

TABLE OF CONTENTS

This week in multimodal AI:

*   Researchers introducing new methods to replace embeddings with discrete IDs for **faster cross-modal search**
*   New **video retrieval systems** showing improvement by integrating audio with visual cues
*   Tutorials to build **Multimodal RAG** applications
*   Multimodal use cases expanding across **healthcare and e-commerce**
*   Cool demos showcasing **visual reasoning** and **multimodal agents**

## 🧠 Research Highlights

*   **GENIUS** – Proposes a universal _generative_ retrieval model that outputs discrete identifiers for targets instead of embeddings ([GENIUS: A Generative Framework for Universal Multimodal Search](https://arxiv.org/html/2503.19868?ref=blog.mixpeek.com#:~:text=in%20performance,speed%20across%20database%20size%2C%20with)). _Significance:_ Matches the accuracy of embedding-based methods on cross-modal benchmarks with much faster search at scale, pointing to a new efficient paradigm for multimodal retrieval ([GENIUS: A Generative Framework for Universal Multimodal Search](https://arxiv.org/html/2503.19868?ref=blog.mixpeek.com#:~:text=varied%20query%20forms,based%20methods%20while%20preserving%20efficiency)).

![Refer to caption](https://arxiv.org/html/2503.19868v1/x1.png)

*   **MegaPairs & BGE-VL** – Introduces a synthetic dataset of 26M image-text pairs to train retrieval mod ([GitHub - VectorSpaceLab/MegaPairs: MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval](https://github.com/VectorSpaceLab/MegaPairs?ref=blog.mixpeek.com#:~:text=In%20this%20work%2C%20we%20introduce,MLLM)). _Significance:_ The resulting BGE-VL models set a new state-of-the-art on composed image retrieval tasks, outperforming prior methods by ~8% (mAP) and even beating models 50× lar ([GitHub - VectorSpaceLab/MegaPairs: MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval](https://github.com/VectorSpaceLab/MegaPairs?ref=blog.mixpeek.com#:~:text=tasks,art%20model)), showing data generation can trump brute-scale.

```python
# Multimodal Retrieval with BGE-VL
# Demonstrates how to use BGE-VL for image + text queries

import torch
from transformers import AutoModel
from PIL import Image

# Load the model - choose from base or large versions
model = AutoModel.from_pretrained("BAAI/BGE-VL-base", trust_remote_code=True)
model.set_processor("BAAI/BGE-VL-base")  # Initialize the processor
model.eval()  # Set to evaluation mode

# Example: Combined image + text query
# The power of multimodal retrieval is combining both modalities
with torch.no_grad():
    # Encode a query using both image and text instruction
    query_embedding = model.encode(
        images="./product_image.jpg", 
        text="Find this in blue color with leather material"
    )
    
    # Encode candidate images from your database
    candidate_embeddings = model.encode(
        images=["./candidate1.jpg", "./candidate2.jpg", "./candidate3.jpg"]
    )
    
    # Calculate similarity scores
    similarity_scores = query_embedding @ candidate_embeddings.T
    
    # Get the most similar item
    best_match_idx = torch.argmax(similarity_scores).item()
    print(f"Best match: candidate{best_match_idx+1}.jpg with score {similarity_scores[0][best_match_idx]:.4f}")
```

*   **MMMORRF** – A multilingual video retrieval system that combines visual **and** audio features using a modality-aware rank fusion ([MMMORRF: Multimodal Multilingual MOdularized Reciprocal Rank Fusion](https://arxiv.org/html/2503.20698v1?ref=blog.mixpeek.com#:~:text=signals,modality%20retrieval)). _Significance:_ By integrating multiple modalities, it achieved dramatic gains on video search benchmarks ([MMMORRF: Multimodal Multilingual MOdularized Reciprocal Rank Fusion](https://arxiv.org/html/2503.20698v1?ref=blog.mixpeek.com#:~:text=extracts%20text%20and%20features%20from,value%20of%20integrating%20diverse%20modalities)), underscoring the importance of indexing more than just visuals for video understanding.

![Refer to caption](https://arxiv.org/html/2503.20698v1/x1.png)

## 🛠️ Tools & Techniques

*   **RAP (Retrieval-Augmented Personalization)** – A new open-source library (with a CVPR’25 paper) that lets you inject personalized knowledge into a multimodal LLM via retri ([GitHub - Hoar012/RAP-MLLM: \[CVPR 2025\] RAP: Retrieval-Augmented Personalization](https://github.com/Hoar012/RAP-MLLM?ref=blog.mixpeek.com#:~:text=Personalize%20Your%20Multimodal%20Large%20Language,Model%20via%20Retrieval%20Augmented%20Generation)). _How you might use it:_ You can have a vision-language model “remember” custom concepts (e.g. who your family members are in photos) by feeding it a private image-text database, enabling personalized Q&A or content generation without retraining.

![RAP-MLLM](https://github.com/Hoar012/RAP-MLLM/raw/main/images/teaser.png)

*   **Multimodal RAG with ImageBind + Elasticsearch** – A step-by-step tutorial from Elastic shows how to build a retrieval-augmented generation pipeline unifying text, images, audio, and even depth ([Building a Multimodal RAG system with Elasticsearch: The story of Gotham City - Elasticsearch Labs](https://www.elastic.co/search-labs/blog/building-multimodal-rag-system?ref=blog.mixpeek.com#:~:text=In%20this%20blog%2C%20you%27ll%20learn,generate%20a%20comprehensive%20final%20report)). It uses Meta’s ImageBind to embed different data types into one vector space and Elasticsearch for fast k-NN se ([Building a Multimodal RAG system with Elasticsearch: The story of Gotham City - Elasticsearch Labs](https://www.elastic.co/search-labs/blog/building-multimodal-rag-system?ref=blog.mixpeek.com#:~:text=ImageBind%20to%20generate%20embeddings%20for,generate%20a%20comprehensive%20final%20report)). _How you might use it:_ Follow their open notebook to spin up a “crime scene AI” that indexes clues of various modalities and lets a GPT-4 agent reason over them to solve myste ([Building a Multimodal RAG system with Elasticsearch: The story of Gotham City - Elasticsearch Labs](https://www.elastic.co/search-labs/blog/building-multimodal-rag-system?ref=blog.mixpeek.com#:~:text=1,response%20and%20identifies%20the%20suspect)) – a template for any app needing multimodal search with an LLM backend.

![Multimodal rag flow](https://www.elastic.co/search-labs/_next/image?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fme0ej585%2Fsearch-labs-import-testing%2Fa4ffdb44582738991813c045be37312dacb0d4f3-1488x1436.png&w=3840&q=75)

## 🏗️ Real-World Applications

*   **Healthcare:** Google Cloud’s Vertex AI Search for healthcare just went multimodal – doctors can now query patient data with both text and medical images thanks to a new _Visual Q&A_ fe ([Google Cloud Enhances Vertex AI Search for Healthcare with Multimodal AI | bakercityherald](https://bakercityherald.com/2025/03/03/google-cloud-enhances-vertex-ai-search-for-healthcare-with-multimodal-ai/?ref=blog.mixpeek.com#:~:text=SUNNYVALE%2C%20Calif,comprehensive%20view%20of%20patient%20health)). This gives clinicians a more comprehensive view (e.g. analyzing X-rays alongside notes), as multimodal analysis of diverse patient data helps improve diagnosis and decision-m ([Google Cloud Enhances Vertex AI Search for Healthcare with Multimodal AI | bakercityherald](https://bakercityherald.com/2025/03/03/google-cloud-enhances-vertex-ai-search-for-healthcare-with-multimodal-ai/?ref=blog.mixpeek.com#:~:text=%E2%80%9CWe%E2%80%99re%20helping%20clinicians%20to%20work,making.%E2%80%9D)).
*   **E-commerce:** Online retailers are leveraging multimodal AI to enrich product search. For example, Amazon researchers showed that a multimodal LLM can extract product attributes implicitly shown in images or text descrip ([EIVEN: Efficient implicit attribute value extraction using multimodal LLM - Amazon Science](https://www.amazon.science/publications/eiven-efficient-implicit-attribute-value-extraction-using-multimodal-llm?ref=blog.mixpeek.com#:~:text=often%20struggle%20with%20implicit%20attribute,source%20datasets%20for%20multimodal%20implicit)). This approach (dubbed **EIVEN**) outperforms traditional methods while using far less labeled data, making product catalogs more searchable and acc ([EIVEN: Efficient implicit attribute value extraction using multimodal LLM - Amazon Science](https://www.amazon.science/publications/eiven-efficient-implicit-attribute-value-extraction-using-multimodal-llm?ref=blog.mixpeek.com#:~:text=often%20struggle%20with%20implicit%20attribute,source%20datasets%20for%20multimodal%20implicit)) ([EIVEN: Efficient implicit attribute value extraction using multimodal LLM - Amazon Science](https://www.amazon.science/publications/eiven-efficient-implicit-attribute-value-extraction-using-multimodal-llm?ref=blog.mixpeek.com#:~:text=Additionally%2C%20we%20construct%20initial%20open,while%20requiring%20less%20labeled%20data)).
*   **Assistive tech:** Multimodal systems are being applied to improve accessibility. Alibaba’s Qwen2.5-Omni, for instance, can generate real-time audio descriptions of visual scenes, which could help visually impaired users navigate their enviro ([Alibaba Cloud Releases Qwen2.5-Omni-7B An End-to-end Multimodal AI Model - Alibaba Cloud Community](https://www.alibabacloud.com/blog/alibaba-cloud-releases-qwen2-5-omni-7b-an-end-to-end-multimodal-ai-model_602095?ref=blog.mixpeek.com#:~:text=Despite%20its%20compact%207B,video%20ingredients%2C%20or%20powering%20intelligent)). Similarly, the model can analyze live video (e.g. a cooking demo) and guide a user through tasks step by step, hinting at more intuitive AI assistance in daily ([Alibaba Cloud Releases Qwen2.5-Omni-7B An End-to-end Multimodal AI Model - Alibaba Cloud Community](https://www.alibabacloud.com/blog/alibaba-cloud-releases-qwen2-5-omni-7b-an-end-to-end-multimodal-ai-model_602095?ref=blog.mixpeek.com#:~:text=Despite%20its%20compact%207B,video%20ingredients%2C%20or%20powering%20intelligent)).

## 📈 Trends & Predictions

*   **“All-in-One” models:** There’s a clear movement toward unified models that handle many modalities simultaneously. The debut of Qwen2.5-Omni (vision + audio + text in one) and other efforts (e.g. Google’s Gemini vision-language upgrades) suggest that future AI stacks will favor integrated multimodal understanding over siloed models – all while keeping model size manageable (7B parameters in Qwen’s case, optimized for edge depl ([Alibaba Cloud Releases Qwen2.5-Omni-7B An End-to-end Multimodal AI Model - Alibaba Cloud Community](https://www.alibabacloud.com/blog/alibaba-cloud-releases-qwen2-5-omni-7b-an-end-to-end-multimodal-ai-model_602095?ref=blog.mixpeek.com#:~:text=videos%2C%20while%20generating%20real,like%20mobile%20phones%20and%20laptops)).
*   **Retrieval-augmented multimodality:** Many updates this week highlight retrieval as a crucial component of multimodal systems. From using external knowledge to ground image answers, to personalization via retrieved user data (RAP), to _generative retrieval_ replacing indexes – combining search with multimodal models is becoming a standard strategy to boost accuracy and controllability.
*   **Efficiency and scalability:** A growing theme is making multimodal models and searches more efficient. Approaches like GENIUS avoid costly nearest-neighbor search by generating IDs on t ([GENIUS: A Generative Framework for Universal Multimodal Search](https://arxiv.org/html/2503.19868?ref=blog.mixpeek.com#:~:text=varied%20query%20forms,based%20methods%20while%20preserving%20efficiency))3-L68】, and new architectures (e.g. Qwen’s blockwise encoders and Thinker-Talker design) enable streaming inputs/outputs without giant compute ov ([\[2503.20215\] Qwen2.5-Omni Technical Report](https://arxiv.org/abs/2503.20215?ref=blog.mixpeek.com#:~:text=,interference%20between%20the%20two%20modalities)). We foresee a push toward resource-friendly multimodal AI that can run in real-time and at scale.

## 🧩 Community + Shoutouts

*   **QVQ-Max visual reasoning demo:** The Alibaba AI team unveiled _QVQ-Max_, a visual reasoning module for their Qwen chat—allowing users to upload an image or video and then see the model’s step-by-step “thinking” process when answering a question ([Multimodal: AI News Week Ending 03/28/2025 - Ethan B. Holland](https://ethanbholland.com/2025/03/28/multimodal-ai-news-week-ending-03-28-2025/?ref=blog.mixpeek.com#:~:text=%E2%80%9C%E2%9C%A8%20Excited%20to%20share%20QVQ,com%2FAlibaba_Qwen%2Fstatus%2F1905342260100956210)). This peek under the hood of a multimodal LLM got the community excited about new ways to interpret and trust AI’s visual answers.

*   **Together Chat (multimodal agent):** Startup **Together** released a free web demo that combines several open-source models to handle diverse tasks. Dubbed _Together Chat_, it can perform web search, code writing, image generation, and even image analysis in one in ([Multimodal: AI News Week Ending 03/28/2025 - Ethan B. Holland](https://ethanbholland.com/2025/03/28/multimodal-ai-news-week-ending-03-28-2025/?ref=blog.mixpeek.com#:~:text=%E2%80%9CIntroducing%20Together%20Chat%21%20Use%20DeepSeek,com%2Ftogethercompute%2Fstatus%2F1904204860217500123)). It showcases the power of connecting multimodal tools (like a language model + vision model) – all accessible to users for free, demonstrating the community’s push toward open, all-in-one AI assistants.

[ES

Ethan Steininger

Former lead of MongoDB's Search Team, Ethan noticed the most common problem customers faced was building indexing and search infrastructure on their S3 buckets. Mixpeek was born.



](/blog/authors/ethan)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_end&utm_campaign=mvs_onramp)[Talk to us](/demo)

## Continue Learning

[

![Multimodal Monday #45: Birds, Whales, and the End of Latency](https://blog.mixpeek.com/content/images/2026/02/mm_45.png)

### Multimodal Monday #45: Birds, Whales, and the End of Latency

Your Weekly Multimodal AI Roundup (Feb 9 - Feb 16)

Read More





](/blog/multimodal-monday-45)[

![Multimodal Monday #44: Agents Ship Code, Robots Skateboard](https://blog.mixpeek.com/content/images/2026/02/mm44---with-milo.png)

### Multimodal Monday #44: Agents Ship Code, Robots Skateboard

Feb 2 - 9: GPT-5.3-Codex and Claude Opus 4.6 handle full software lifecycles from debug to deploy, MiniCPM-o 4.5 beats GPT-4o on vision tasks at 9B parameters running on-device, HUSKY skateboards using physics-based control, and TinyLoRA fine-tunes models with a single parameter.

Read More





](/blog/multimodal-monday-44)

### Author

ES

Ethan Steininger

![Mixpeek Icon](/brand/logo-dark-icon.webp)

#### Multimodal Search, On Demand

Index, retrieve, and reason over videos, images, audio, and docs, all from a single API.

[Start Building](/start)

### Share

[](https://twitter.com/intent/tweet?text=%F0%9F%A7%A0%20Multimodal%20Monday%20%231%20-%20State%20of%20the%20Stack&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-state-of-the-stack)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-state-of-the-stack&title=%F0%9F%A7%A0%20Multimodal%20Monday%20%231%20-%20State%20of%20the%20Stack)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-state-of-the-stack&title=%F0%9F%A7%A0%20Multimodal%20Monday%20%231%20-%20State%20of%20the%20Stack)