What is Mixpeek?

[Back to Blog](/blog)

[](https://twitter.com/intent/tweet?text=Multimodal%20Monday%20%2339%3A%20Generative%20Giants%2C%20Retrieval%20Weaklings&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fgenerative-giants-retrieval-weaklings)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fgenerative-giants-retrieval-weaklings&title=Multimodal%20Monday%20%2339%3A%20Generative%20Giants%2C%20Retrieval%20Weaklings)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fgenerative-giants-retrieval-weaklings&title=Multimodal%20Monday%20%2339%3A%20Generative%20Giants%2C%20Retrieval%20Weaklings)

This was originally in our newsletter.

Get Updates

Jan 5, 2026•8 min read

# Multimodal Monday #39: Generative Giants, Retrieval Weaklings

Weeks of December 29 - January 4, 2026: Research reveals why MLLMs like Qwen2-VL fall down, MegaRAG builds multimodal knowledge graphs without manual construction, HiStream generates 1080p video 107x faster, and Google DeepMind discovers geometric memory in sequence models.

![Multimodal Monday #39: Generative Giants, Retrieval Weaklings](https://blog.mixpeek.com/content/images/2026/01/mm_39.png)

Multimodal Monday

TABLE OF CONTENTS

Happy New Year! This roundup covers the past two weeks and 2026 has started with a bang, it is going to be a big year for multimodal AI.

## Quick Hits (TL;DR)

*   **MLLMs can't retrieve what they can generate.** New research shows models like Qwen2-VL and Paligemma2 fail at zero-shot multimodal retrieval despite excelling at generation. CLIP still wins for search tasks.
*   **Sequence models discovered a new memory type.** Google DeepMind identified geometric memory in deep sequence models, where embeddings encode global entity relationships rather than just co-occurrence patterns.
*   **Video generation hit a speed breakthrough.** HiStream eliminates spatial, temporal, and timestep redundancy to generate 1080p video 107x faster than previous methods.

## Tools, Models and Techniques

### **Tencent HY-Motion 1.0**

Tencent released a billion-parameter text-to-motion model using Diffusion Transformer architecture and flow matching. The model generates 3D character animations from text prompts.

 Your browser does not support the video tag.

**Why it matters:** You can now generate fluid, diverse character animations without motion capture data.  
Links: [Project Page](https://hunyuan.tencent.com/motion?ref=blog.mixpeek.com) | [Github](https://github.com/Tencent-Hunyuan/HY-Motion-1.0?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/tencent/HY-Motion-1.0?ref=blog.mixpeek.com) | [Technical report](https://arxiv.org/pdf/2512.23464?ref=blog.mixpeek.com)

### **Diffusion Knows Transparency (DKT)**

DKT repurposes video diffusion models for transparent object depth and normal estimation. It achieves zero-shot SOTA on ClearPose/DREDS benchmarks, runs at 0.17s per frame, and maintains temporal consistency across videos.

 Your browser does not support the video tag.

**Why it matters:** Transparent objects finally get reliable depth estimation without specialized training data.  
Links: [Hugging Face](https://huggingface.co/Daniellesry/DKT-Normal-14B?ref=blog.mixpeek.com) | [Paper](https://huggingface.co/papers/2512.23705?ref=blog.mixpeek.com) | [Website](https://daniellli.github.io/projects/DKT/?ref=blog.mixpeek.com) | [Models](https://huggingface.co/collections/Daniellesry/dkt-models?ref=blog.mixpeek.com)

### **LongVideoAgent**

A multi-agent framework where a master LLM coordinates a grounding agent for segment localization and a vision agent for observation extraction. The system uses reinforcement learning to optimize multi-agent cooperation with step limits.

**Why it matters:** You can now query hour-long videos with targeted questions instead of processing entire timelines.  
Links: [Paper](https://huggingface.co/papers/2512.20618?ref=blog.mixpeek.com) | [Website](https://longvideoagent.github.io/?ref=blog.mixpeek.com) | [GitHub](https://github.com/longvideoagent/LongVideoAgent?ref=blog.mixpeek.com)

### **Qwen-Image-2512**

Qwen's new text-to-image model delivers more realistic humans, finer natural textures, and stronger text rendering. It sets a new SOTA benchmark for image generation quality.  
Links: [Hugging Face](https://huggingface.co/Qwen/Qwen-Image-2512?ref=blog.mixpeek.com) | [GitHub](https://github.com/QwenLM/Qwen-Image?ref=blog.mixpeek.com) | [Blog](https://qwen.ai/blog?id=qwen-image-2512&ref=blog.mixpeek.com) | [Demo](https://huggingface.co/spaces/Qwen/Qwen-Image-2512?ref=blog.mixpeek.com) | [GGUF](https://huggingface.co/unsloth/Qwen-Image-2512-GGUF?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **Yume-1.5**

A text-controlled interactive world generation model that creates explorable 3D environments. Users can navigate and interact with generated spaces in real time.  
Links: [Website](https://stdstu12.github.io/YUME-Project/?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/stdstu123/Yume-5B-720P?ref=blog.mixpeek.com) | [Paper](https://huggingface.co/papers/2512.22096?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **TwinFlow**

Enables one-step generation on large models using self-adversarial flows. The approach eliminates iterative sampling while maintaining output quality.  
Links: [Hugging Face](https://huggingface.co/inclusionAI/TwinFlow-Z-Image-Turbo?ref=blog.mixpeek.com)

### **Stable Video Infinite 2.0 Pro**

The new version launched with immediate ComfyUI wrapper support from Kijai. Models are already available for download and integration.  
Links: [Hugging Face](https://huggingface.co/Kijai/WanVideo_comfy/tree/main/LoRAs/Stable-Video-Infinity/v2.0?ref=blog.mixpeek.com) | [GitHub](https://github.com/vita-epfl/Stable-Video-Infinity?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **Soprano**

An ultra-lightweight TTS model that generates 10 hours of 32kHz audio in under 20 seconds. It streams with sub-15ms latency using only 80M parameters and less than 1GB VRAM.  
Links: [GitHub](https://github.com/ekwek1/soprano?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **Wan-NVFP4**

A fast video model claiming 28x faster render speeds than previous versions. Released by lightx2v on Hugging Face.  
Links: [Hugging Face](https://huggingface.co/lightx2v/Wan-NVFP4?ref=blog.mixpeek.com)

### **JavisGPT**

A unified multi-modal LLM for sounding-video comprehension and generation. The model handles video analysis and audio-visual synthesis in one framework.  
Links: [Paper](https://arxiv.org/abs/2512.22905?ref=blog.mixpeek.com) | [GitHub](https://github.com/JavisVerse/JavisGPT?ref=blog.mixpeek.com) | [Models](https://huggingface.co/collections/JavisVerse/javisgpt?ref=blog.mixpeek.com)

### **Dream-VL & Dream-VLA**

Open vision-language and vision-language-action models using a diffusion language model backbone. Both models integrate visual understanding with either language or robotic action outputs.  
Links: [Paper](https://arxiv.org/html/2512.22615v1?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/Dream-org/Dream-VL-7B?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/Dream-org/Dream-VLA-7B?ref=blog.mixpeek.com) | [GitHub](https://github.com/DreamLM/Dream-VLX?ref=blog.mixpeek.com)

### **HyperCLOVA X SEED Omni 8B**

A unified multimodal model handling text, vision, audio, and video inputs with text, image, and audio outputs.  
Links: [Hugging Face](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Omni-8B/tree/main?ref=blog.mixpeek.com)

### **AMD ROCm**

AMD published a guide for accelerating multimodal inference in vLLM using batch-level dynamic programming switches.  
Links: [Blog](https://rocm.blogs.amd.com/software-tools-optimization/vllm-dp-vision/README.html?ref=blog.mixpeek.com)

## Research Highlights

### **Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?**

MLLMs like Qwen2-VL and Paligemma2 fail at zero-shot multimodal retrieval despite excelling at generation. Using sparse autoencoders, researchers identified three limitations: training objectives optimize for generation not retrieval, evaluation focuses on generative tasks, and autoregressive architectures compute poor similarity scores.

![](https://substackcdn.com/image/fetch/$s_!q8eK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c1dd7e4-d885-4d63-9038-6951767152df_1274x332.jpeg)

Distribution of modality scores for learned concepts by (a) CLIP, (b) SigLIP2, (c) Qwen3-VL-8BInstruct, and (d) Paligemma2-3b-Mix-224. The Modality Score quantifies the bias of each concept towards the image modality (blue region) or the text modality (red region).

**Why it matters:** You need specialized models like CLIP for retrieval tasks until MLLMs address these architectural limitations.  
Links: [Paper](https://arxiv.org/abs/2512.19115?ref=blog.mixpeek.com)

### **ReaSeq: Unleashing World Knowledge via Reasoning for Sequential Modeling**

ReaSeq solves two problems in recommender systems: knowledge poverty in ID-based representations and systemic blindness to off-platform user interests. The framework uses reasoning to incorporate world knowledge into sequential modeling.

![](https://substackcdn.com/image/fetch/$s_!jCbn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd48fcdfe-584e-44a6-9386-920786df2667_1894x1024.jpeg)

**Why it matters:** Recommendation systems can now reason about user interests beyond platform interaction logs.  
Links: [Paper](https://arxiv.org/abs/2512.21257?ref=blog.mixpeek.com)

### **MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation**

MegaRAG automatically constructs multimodal knowledge graphs integrating text, visual, and spatial information from documents. It uses a two-round, page-based approach where LLMs extract entities in parallel then refine the graph by retrieving relevant subgraphs.

![](https://substackcdn.com/image/fetch/$s_!q6i9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faec3ca2e-a78c-4ce7-8074-b46444328989_1838x1184.jpeg)

**Why it matters:** RAG systems can now handle visually rich documents without manual graph construction.  
Links: [Paper](https://arxiv.org/abs/2512.20626?ref=blog.mixpeek.com)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_mid&utm_campaign=mvs_onramp)[Talk to us](/demo)

### **Retrieval-augmented Prompt Learning for Pre-trained Foundation Models**

This framework enhances prompt learning by decoupling knowledge from memorization. The approach addresses instability in low-resource settings where parametric models overfit to shallow patterns.

![](https://substackcdn.com/image/fetch/$s_!soGy!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8189038-4480-49eb-9be6-7c406166c0fb_4178x1874.png)

**Why it matters:** Prompt learning becomes reliable even with limited training data.  
Links: [Paper](https://arxiv.org/abs/2512.20145?ref=blog.mixpeek.com) | [GitHub](https://github.com/zjunlp/PromptKG/tree/main/research/RetroPrompt?ref=blog.mixpeek.com)

### **Latent Implicit Visual Reasoning**

LIVR discovers visual reasoning tokens without explicit supervision. The method outperforms approaches requiring costly annotations.

![](https://substackcdn.com/image/fetch/$s_!N55D!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57883c9a-f539-4d4c-9617-a45ce23c6904_1500x1189.jpeg)

**Why it matters:** Visual reasoning models no longer need expensive manual labeling.  
Links: [Paper](https://huggingface.co/papers/2512.21218?ref=blog.mixpeek.com)

### **Geometric Memory in Sequence Models**

Google DeepMind identified geometric memory in deep sequence models, where embeddings encode global relationships between all entities including those never co-occurring in training. This contrasts with associative memory's brute-force lookup approach.

![](https://substackcdn.com/image/fetch/$s_!vFG8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1dce63-c500-42fc-8223-b100a981bdad_1526x1492.jpeg)

Associative vs. geometric memory of models trained on various graphs. There are two dramatically different ways to memorize a dataset of atomic facts.

**Why it matters:** Sequence models can now perform multi-step reasoning through single-step navigation in embedding space.

Links: [Paper](https://arxiv.org/abs/2510.26745?ref=blog.mixpeek.com)

### **Step-DeepResearch**

A 32B parameter research agent matching OpenAI and Gemini DeepResearch through atomic capability training. It decomposes research into planning, information gathering, verification, and writing, achieving 61.42 on ResearchRubrics.  
Links: [Paper](https://arxiv.org/abs/2512.20491?ref=blog.mixpeek.com) | [GitHub](https://github.com/stepfun-ai/StepDeepResearch?ref=blog.mixpeek.com)

### **SpatialTree**

A 4-level cognitive hierarchy mapping spatial abilities in MLLMs from perception to agentic competence. Benchmarks 27 sub-abilities across 16 models and reveals transfer patterns.  
Links: [Website](https://spatialtree.github.io/?ref=blog.mixpeek.com) | [Paper](https://arxiv.org/abs/2512.20617v1?ref=blog.mixpeek.com) | [Benchmark](https://spatialtree.github.io/?ref=blog.mixpeek.com#benchmark)

 Your browser does not support the video tag.

### **FlowBlending**

Stage-aware multi-model sampling for fast and high-fidelity video generation. The approach optimizes different models for different stages of the generation process.  
Links: [Paper](https://huggingface.co/papers/2512.24724?ref=blog.mixpeek.com)

### **SpaceTimePilot**

Adobe's video diffusion model disentangles space and time for controllable rendering. From a single input video, it enables independent control of camera viewpoint and motion for bullet-time, slow motion, and mixed trajectories.  
Links: [Website](https://zheninghuang.github.io/Space-Time-Pilot/?ref=blog.mixpeek.com) | [Paper](https://huggingface.co/papers/2512.25075?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **HiStream**

Meta's autoregressive framework for 1080p video generation eliminates spatial, temporal, and timestep redundancy. HiStream achieves SOTA quality with up to 107.5x speedup.  
Links: [Website](http://haonanqiu.com/projects/HiStream.html?ref=blog.mixpeek.com) | [Paper](https://huggingface.co/papers/2512.21338?ref=blog.mixpeek.com) | Code

![](https://substackcdn.com/image/fetch/$s_!-gmz!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe199653-9e57-4fa3-a9ac-04b196ebf294_3840x2536.jpeg)

### **InsertAnywhere**

Bridges 4D scene geometry and diffusion models for realistic video object insertion. The method maintains spatial and temporal consistency across frames.  
Links: [Paper](https://huggingface.co/papers/2512.17504?ref=blog.mixpeek.com) | [Website](https://myyzzzoooo.github.io/InsertAnywhere/?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **Robust-R1**

A framework making multimodal models robust to visual degradations through explicit degradation-aware reasoning chains. Achieves SOTA robustness on R-Bench while maintaining interpretability.  
Links: [Paper](https://huggingface.co/papers/2512.17532?ref=blog.mixpeek.com) | [Demo](https://huggingface.co/spaces/Jiaqi-hkust/Robust-R1?ref=blog.mixpeek.com) | [Dataset](https://huggingface.co/datasets/Jiaqi-hkust/Robust-R1?ref=blog.mixpeek.com)

### **StoryMem**

ByteDance's multi-shot long video storytelling framework with memory. The system maintains narrative consistency across extended video sequences.  
Links: [Website](https://kevin-thu.github.io/StoryMem/?ref=blog.mixpeek.com) | [Code](https://github.com/Kevin-thu/StoryMem?ref=blog.mixpeek.com)

### **Spatia**

Microsoft's video generation system maintains a 3D scene point cloud as persistent spatial memory. Enables long-horizon, spatially consistent video generation with explicit camera control and 3D-aware editing.  
Links: [Website](https://zhaojingjing713.github.io/Spatia/?ref=blog.mixpeek.com) | [Paper](https://huggingface.co/papers/2512.15716?ref=blog.mixpeek.com) | [Video](https://cdn-uploads.huggingface.co/production/uploads/64f8962bce75bb0fb50bdbdb/eC4hVgIfk0MxzPgn6mvGC.mp4?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **DiffThinker**

Enables generative multimodal reasoning with diffusion models. The approach integrates reasoning capabilities directly into the diffusion generation process.  
Links: [Paper](https://huggingface.co/papers/2512.24165?ref=blog.mixpeek.com) | [Website](https://diffthinker-project.github.io/?ref=blog.mixpeek.com)

## Trends & Predictions

**MLLMs Are Not Taking Over... Yet**

New research reveals a fundamental problem with multimodal large language models. Despite their impressive text generation, models like Qwen2-VL and Paligemma2 fail at zero-shot multimodal retrieval. They can describe an image in detail but cannot tell you which image best matches a query. They can answer complex questions about a video but cannot find relevant videos in a database.

The problem is architectural. MLLMs train to generate text, not retrieve images. They optimize for autoregressive generation, not similarity computation. Their evaluation metrics focus on generation quality, not retrieval accuracy.

This matters for your systems. If you need multimodal retrieval, use CLIP. It was designed for retrieval. MLLMs were not. CLIP runs faster, retrieves more accurately, and trains easier. The specialized model wins.

This pattern extends beyond retrieval. Step-DeepResearch shows a 32B parameter model matches much larger research agents by focusing on atomic capabilities. HiStream achieves 107x speedup by eliminating redundancy rather than adding parameters. Soprano runs real-time speech synthesis on consumer hardware with 80M parameters.

The trend is clear. Specialized models beat general-purpose models at specific tasks. You need to match your model to your task. The ecosystem is diversifying, not consolidating. Different problems need different architectures.

MLLMs will improve at retrieval. Someone will solve the architectural issues. But that day is not today. Build with the tools that work now, not the tools you hope will work later.

## Community + Shoutouts

### **ComfyUI Segmentation Agent**

Adam Barbato released an LLM-based character segmentation agent for ComfyUI using SAM 3.  
Links: [GitHub](https://github.com/adambarbato/ComfyUI-Segmentation-Agent?ref=blog.mixpeek.com)

![](https://raw.githubusercontent.com/adambarbato/ComfyUI-Segmentation-Agent/refs/heads/main/assets/example.png)

### **CosyVoice 3 ComfyUI**

Machine Delusion released a voice cloning node pack featuring CosyVoice 3 for ComfyUI.  
Links: [Announcement](https://x.com/machinedelusion/status/2004003141247959482?s=42&ref=blog.mixpeek.com) | [GitHub](https://github.com/filliptm/ComfyUI_FL-CosyVoice3?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

### **SAM3 Video Tracking in X-AnyLabeling**

Important\_Priority76 integrated SAM3 video object tracking into X-AnyLabeling for easy annotation workflows.  
Links: [Reddit](https://www.reddit.com/r/computervision/comments/1q2zplt/just_integrated_sam3_video_object_tracking_into/?ref=blog.mixpeek.com) | [GitHub](https://github.com/CVHub520/X-AnyLabeling/blob/main/examples/interactive_video_object_segmentation/sam3/README.md?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

* * *

**That's a wrap for Multimodal Monday #39!** This week exposed fundamental limitations in current multimodal models while demonstrating paths forward. MLLMs excel at generation but fail at retrieval due to architectural constraints. Meanwhile, HiStream proves speed comes from eliminating redundancy, MegaRAG shows automatic multimodal knowledge graphs work, and geometric memory reveals how models encode relationships they never saw in training.

_Ready to build multimodal solutions that actually work?_ [_Let's talk_](https://mixpeek.com/demo?ref=blog.mixpeek.com)

[PB

Philip Bankier

Philip’s expertise in RAG, data engineering, and AI agents makes him a natural fit to write about scaling multimodal data warehousing systems.



](/blog/authors/philip)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_end&utm_campaign=mvs_onramp)[Talk to us](/demo)

## Continue Learning

[

![Multimodal Monday #45: Birds, Whales, and the End of Latency](https://blog.mixpeek.com/content/images/2026/02/mm_45.png)

### Multimodal Monday #45: Birds, Whales, and the End of Latency

Your Weekly Multimodal AI Roundup (Feb 9 - Feb 16)

Read More





](/blog/multimodal-monday-45)[

![Multimodal Monday #44: Agents Ship Code, Robots Skateboard](https://blog.mixpeek.com/content/images/2026/02/mm44---with-milo.png)

### Multimodal Monday #44: Agents Ship Code, Robots Skateboard

Feb 2 - 9: GPT-5.3-Codex and Claude Opus 4.6 handle full software lifecycles from debug to deploy, MiniCPM-o 4.5 beats GPT-4o on vision tasks at 9B parameters running on-device, HUSKY skateboards using physics-based control, and TinyLoRA fine-tunes models with a single parameter.

Read More





](/blog/multimodal-monday-44)

### Author

PB

Philip Bankier

![Mixpeek Icon](/brand/logo-dark-icon.webp)

#### Multimodal Search, On Demand

Index, retrieve, and reason over videos, images, audio, and docs, all from a single API.

[Start Building](/start)

### Share

[](https://twitter.com/intent/tweet?text=Multimodal%20Monday%20%2339%3A%20Generative%20Giants%2C%20Retrieval%20Weaklings&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fgenerative-giants-retrieval-weaklings)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fgenerative-giants-retrieval-weaklings&title=Multimodal%20Monday%20%2339%3A%20Generative%20Giants%2C%20Retrieval%20Weaklings)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fgenerative-giants-retrieval-weaklings&title=Multimodal%20Monday%20%2339%3A%20Generative%20Giants%2C%20Retrieval%20Weaklings)