What is Mixpeek?

[Back to Blog](/blog)

[](https://twitter.com/intent/tweet?text=Multimodal%20Monday%20%2330%3A%20Smarter%20Agents%2C%20Real-Time%203D&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-30-smarter-agents)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-30-smarter-agents&title=Multimodal%20Monday%20%2330%3A%20Smarter%20Agents%2C%20Real-Time%203D)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-30-smarter-agents&title=Multimodal%20Monday%20%2330%3A%20Smarter%20Agents%2C%20Real-Time%203D)

This was originally in our newsletter.

Get Updates

Oct 27, 2025•7 min read

# Multimodal Monday #30: Smarter Agents, Real-Time 3D

Multimodal Monday #30: WALT and UltraCUA make websites API-smart, Seed3D 1.0 builds 3D assets from one image, DeepSeek-OCR compresses docs 10x with 97% accuracy via optical mapping and AGILE lifts VLM accuracy from 9.5% to 82.8% with interactive puzzles.

![Multimodal Monday #30: Smarter Agents, Real-Time 3D](https://blog.mixpeek.com/content/images/2025/10/mm_30-fix.png)

Multimodal Monday

TABLE OF CONTENTS

### Week of October 20-26, 2025

## 📢 Quick Hits (TL;DR)

**Web Agents Stop Clicking, Start Understanding** - Salesforce and Apple both released frameworks this week that teach agents to reverse-engineer websites into reusable tools. No more brittle click-and-type scripts. These agents now extract the underlying functionality and call it like an API.

**3D Generation Picks up Speed** - Tencent, ByteDance, Ant Group, and Krea AI all shipped major updates focused on real production use: video-to-3D conversion, multi-shot narratives, and real-time generation on single GPUs.

**Learning by Doing Beats Passive Training** - New research shows VLMs improve dramatically when they learn interactively. One model jumped from 9.5% to 82.8% accuracy on a task just by solving jigsaw puzzles through trial and error.

## 🧠 Research Highlights

#### **WALT: Web Agents that Learn Tools**

Salesforce built WALT to make browser agents stop clicking around like lost tourists. Instead, agents now reverse-engineer website features into structured APIs through a demonstrate-generate-validate loop, turning messy UI interactions into clean function calls like `search(query)`.

![WALT Overview](https://substackcdn.com/image/fetch/$s_!UUKI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71d6969-4654-4255-86d3-f82501bf4995_6583x1666.png "WALT Overview")

**Why It Matters:** You get agents that understand what a website does, not just where to click, making automation 10x more reliable.  
**Links:** [Paper](https://arxiv.org/abs/2510.01524?ref=blog.mixpeek.com) | [GitHub](https://github.com/SalesforceAIResearch/WALT?ref=blog.mixpeek.com) | [Announcement](https://x.com/caimingxiong/status/1981133607529042245?s=42&ref=blog.mixpeek.com)

#### **AGILE: Agentic Jigsaw Interaction Learning**

Researchers trained a VLM by making it solve jigsaw puzzles through trial and error. The model observes the puzzle, generates code to swap pieces, sees the result, and tries again. This simple interactive loop took accuracy from 9.5% to 82.8% and improved performance on nine other vision tasks by an average of 3.1%.

![](https://yuzeng0-0.github.io/AGILE/figure/main1.png)

Overview of ****AGILE****.

**Why It Matters:** We can generate unlimited training data by building interactive environments where models learn by doing, not just watching.  
**Links:** [Project Page](https://yuzeng0-0.github.io/AGILE/?ref=blog.mixpeek.com) | [Paper](https://arxiv.org/abs/2510.01304?ref=blog.mixpeek.com) | [GitHub](https://github.com/yuzeng0-0/AGILE?ref=blog.mixpeek.com)

#### **Sa2VA: Dense Grounded Understanding of Images and Videos**

ByteDance combined SAM-2’s segmentation with LLaVA’s vision-language understanding into one unified model. Sa2VA handles both images and videos, producing pixel-precise masks for any object you ask about through conversational prompts.

![Composite image with multiple panels demonstrating Sa2VA model outputs. Top section shows video frames of a woman in a red dress carrying a black bag walking on a brightly lit urban street at night, with bounding boxes and masks segmenting the person and bag. Middle section includes colorful polar plots labeled with metrics like mIoU, PQ, and STQ for different tasks. Bottom section features abstract visualizations of image and video segmentation results with overlaid masks and color-coded regions for objects like animals, weather elements, and graffiti.](https://substackcdn.com/image/fetch/$s_!NMiu!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a0bc258-4427-49e5-9d8d-bb486a42e15d_1200x853.jpeg "Composite image with multiple panels demonstrating Sa2VA model outputs. Top section shows video frames of a woman in a red dress carrying a black bag walking on a brightly lit urban street at night, with bounding boxes and masks segmenting the person and bag. Middle section includes colorful polar plots labeled with metrics like mIoU, PQ, and STQ for different tasks. Bottom section features abstract visualizations of image and video segmentation results with overlaid masks and color-coded regions for objects like animals, weather elements, and graffiti.")

**Why It Matters:** You get both high-level understanding and pixel-level precision in one model for video editing, robotics, and visual search applications.  
**Links:** [Paper](https://huggingface.co/papers/2501.04001?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/ByteDance/Sa2VA-Qwen3-VL-4B?ref=blog.mixpeek.com)

#### **UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action**

Apple’s UltraCUA mixes low-level GUI actions with high-level API calls in one model. Train it with supervised learning, then online RL on hybrid action trajectories. The result beats baselines by 22% while running 11% faster.

![](https://substackcdn.com/image/fetch/$s_!wB7Q!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b0da10e-02f5-4e7b-90ff-7f429063645d_2246x936.jpeg)

An overview of UltraCUA’s design. The agent adaptively switches between visual grounding and programmatic tool call, establishing the hybrid action mechanism.

**Why It Matters:** Agents now pick the most efficient method for each task, clicking buttons when needed and making API calls when possible.  
**Links:** [Paper](https://arxiv.org/abs/2510.17790?ref=blog.mixpeek.com) | [Announcement](https://x.com/zhegan4/status/1980471759251075578?s=42&ref=blog.mixpeek.com)

#### **Grasp Any Region (GAR): Precise Pixel-Level Understanding for MLLMs**

GAR lets you ask detailed questions about any specific region of an image. It uses global context plus a region-of-interest replay mechanism to beat a 78B-parameter baseline with a much smaller model, and works zero-shot on video tasks.

**Why It Matters:** You can now have detailed conversations about specific parts of images for product inspection, medical analysis, or robotics applications.  
**Links:** [Paper](https://arxiv.org/abs/2510.18876?ref=blog.mixpeek.com)

**Text or Pixels? It Takes Half:** A new study investigates the token efficiency of visual text inputs in Multimodal LLMs. **Links:** [Paper](https://arxiv.org/abs/2510.18279?ref=blog.mixpeek.com) | [GitHub](https://github.com/yanhong-lbh/text_or_pixels?ref=blog.mixpeek.com)

**Glyph: Scaling Context Windows via Visual-Text Compression** introduces a method for scaling context windows. **Links:** [Paper](https://arxiv.org/pdf/2510.17800?ref=blog.mixpeek.com)

**VISTA: A Test-Time Self-Improving Video Generation Agent** from Google presents a new agent for video generation. **Links:** [Paper](https://huggingface.co/papers/2510.15831?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

**Nyx: a new mixed-modal retriever for Universal RAG** is a unified retriever tailored for queries and documents containing a mix of text and images. **Links:** [Paper](https://huggingface.co/papers/2510.17354?ref=blog.mixpeek.com) | [Collection](https://huggingface.co/collections/SnowNation/nyx?ref=blog.mixpeek.com)

**Inpaint4Drag** repurposes inpainting models for drag-based editing by decomposing the task into pixel-space bidirectional warping and inpainting. **Links:** [Project Page](https://visual-ai.github.io/inpaint4drag/?ref=blog.mixpeek.com) | [Paper](https://arxiv.org/pdf/2509.04582?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

**MMAO-Bench: MultiModal All-in-One Benchmark for “Omnimodels”** proposes a unified evaluation suite for multimodal foundation models that handle text, vision, and audio together. **Links:** [Paper](https://arxiv.org/abs/2510.18915?ref=blog.mixpeek.com)

## 🛠️ Tools & Techniques

#### **DeepSeek OCR**

DeepSeek’s OCR reads text in 100 languages and parses complex structures like charts and tables into HTML. It combines CLIP and SAM features for better grounding and a more efficient performance-to-vision-token ratio.

![fig1.png](https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/assets/fig1.png?raw=true)

**Why It Matters:** The model understands document structure and semantics, not just text, which transforms how you digitize financial reports and scientific papers.  
**Links:** [GitHub](https://github.com/deepseek-ai/DeepSeek-OCR?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-OCR?ref=blog.mixpeek.com) | [Paper](https://arxiv.org/abs/2510.18234?ref=blog.mixpeek.com)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_mid&utm_campaign=mvs_onramp)[Talk to us](/demo)

#### **Tencent Hunyuan World 1.1 (WorldMirror)**

Tencent open-sourced WorldMirror, a feed-forward 3D reconstruction model that now handles video-to-3D and multi-view-to-3D. It runs on a single GPU and delivers complete 3D attributes in one forward pass within seconds.

 Your browser does not support the video tag.

**Why It Matters:** You can now quickly create digital twins of real objects from video for e-commerce, gaming, and VR applications.  
**Links:** [Project Page](https://3d-models.hunyuan.tencent.com/world/?ref=blog.mixpeek.com) | [GitHub](https://github.com/Tencent-Hunyuan/HunyuanWorld-Mirror?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/tencent/HunyuanWorld-Mirror?ref=blog.mixpeek.com)

#### **ByteDance Seed3D 1.0**

ByteDance released Seed3D 1.0, which generates high-fidelity, simulation-ready 3D assets from a single image. The output works directly in physics simulations without additional processing.

 Your browser does not support the video tag.

**Why It Matters:** You can rapidly create diverse training environments for autonomous vehicles and robots from simple images.  
**Links:** [Paper](https://arxiv.org/pdf/2510.19944?ref=blog.mixpeek.com) | [Announcement](https://x.com/jianfengzhang95/status/1982076343593353400?s=42&ref=blog.mixpeek.com)

#### **HoloCine by Ant Group**

Generates complete cinematic narratives from text prompts. The model maintains global consistency across multiple shots, creating coherent stories instead of disconnected clips.

 Your browser does not support the video tag.

**Why It Matters:** You now get actual narratives with a beginning, middle, and end instead of random video clips.  
**Links:** [Paper](https://huggingface.co/papers/2510.20822?ref=blog.mixpeek.com) | [Hugging Face](https://huggingface.co/hlwang06/HoloCine?ref=blog.mixpeek.com)

#### **Krea Realtime by Krea AI**

Krea AI released a 14B autoregressive model that generates video at 11 fps on a single B200 GPU. It’s 10x larger than any open-source alternative and handles long-form video generation in real time.

 Your browser does not support the video tag.

**Why It Matters:** Real-time video generation opens up interactive applications where you generate content on the fly in conversations or games.  
**Links:** [Hugging Face](https://huggingface.co/krea/krea-realtime-video?ref=blog.mixpeek.com) | [Announcement](https://x.com/krea_ai/status/1980358158376988747?s=42&ref=blog.mixpeek.com)

**OpenAI Atlas AI Browser** is a new AI browser from OpenAI. **Links:** [Website](https://chatgpt.com/atlas?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

**LightOnOCR-1B** is a new efficient multimodal OCR model. **Links:** [Hugging Face](https://huggingface.co/lightonai/LightOnOCR-1B-1025?ref=blog.mixpeek.com)

**Video-As-Prompt** from ByteDance provides unified semantic control for video generation, animating images with the style and motion of a reference video. **Links:** [Project Page](https://bytedance.github.io/Video-As-Prompt/?ref=blog.mixpeek.com) | [Paper](https://arxiv.org/pdf/2510.20888?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

**Ctrl-World** is a controllable world model that generalizes zero-shot to new environments, cameras, and objects. **Links:** [GitHub](https://github.com/Robert-gyj/Ctrl-World?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

**World-in-World** is the first open benchmark that ranks world models by closed-loop task success, not pixels. **Links:** [Project Page](https://world-in-world.github.io/?ref=blog.mixpeek.com) | [GitHub](https://github.com/World-In-World/world-in-world?ref=blog.mixpeek.com)

**BADAS 1.0** from Nexar is a new incident prediction model built on real-world data and FAIR’s V-JEPA 2. **Links:** [Website](https://nexar-ai.com/?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

**Embody 3D Dataset** from Meta’s Codec Avatars Lab is a large-scale dataset of 3D tracked human motion with audio and text annotations. **Links:** [Project Page](https://www.meta.com/emerging-tech/codec-avatars/embody-3d/?ref=blog.mixpeek.com) | [GitHub](https://github.com/facebookresearch/embody-3d?ref=blog.mixpeek.com)

 Your browser does not support the video tag.

**BLIP3o-NEXT** is a new model with a project page and paper. **Links:** [Project Page](https://jiuhaichen.github.io/BLIP3o-NEXT.github.io/?ref=blog.mixpeek.com) | [Paper](https://huggingface.co/papers/2510.15857?ref=blog.mixpeek.com)

## 📈 Trends & Predictions

#### **Web Agents Learn to Think in Functions, Not Pixels**

Salesforce and Apple both shipped the same insight this week: stop teaching agents to click buttons and start teaching them to extract functionality. WALT and UltraCUA both move from pixel-level automation to API-level understanding.

Here’s why pixel-clicking fails. You train an agent to navigate a website by clicking specific coordinates or finding specific UI elements. Then the site updates its design. Or it loads slower than expected. Or a popup appears. Your agent breaks. Every edge case becomes a new failure mode. You’re essentially teaching the agent to memorize a choreographed dance routine on a stage that keeps changing. WALT and UltraCUA flip this. Instead of “click the search button at these coordinates,” the agent learns “this website has a search function that takes a query parameter.” The agent demonstrates an action through the UI once, then generates a reusable tool from it. Search becomes `search(query)`. Filtering becomes `filter(category, value)`. The agent now reasons about capabilities, not pixel locations. When the UI changes, the agent adapts because it understands the function, not just the button.

This makes automation fundamentally more robust. You’re no longer fighting with UI variations. The agent doesn’t care if the search button is blue or green, top or bottom, 200 pixels or 250 pixels from the edge. It knows the site can search and it knows how to invoke that capability. The browser becomes an API surface the agent can reason about, not a maze of buttons to navigate.

## 🧩 Community + Shoutouts

**Hugging Face OCR Open Model Breakdown** Hugging Face has updated their comprehensive breakdown of open-source OCR models, a great resource for anyone working with document AI. **Links:** [Blog Post](https://huggingface.co/blog/ocr-open-models?ref=blog.mixpeek.com)

**Video Generation Prompting Guide** Shout out to Mitch Leeuwe for sharing an excellent resource for upgrading your video generation prompting skills. **Links:** [Post](https://x.com/MitchLeeuwe/status/1981018613466517971?ref=blog.mixpeek.com)

![Diagram divided into sections showing camera perspectives with purple stick figures: top left illustrates eye levels from low to high with horizon lines and simple scenes like a car; top right displays shot types including pull shot, medium shot, close shot, extreme close shot with figures waving or smiling; bottom left features over the shoulder, dutch angle, bird eye views with figures and clapperboards; bottom right shows push in pull out, dutch angle, movements to left right, and warnings about eye levels and mistakes with a cat and car illustrations.](https://substackcdn.com/image/fetch/$s_!UDpQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70520a50-7002-47f1-95cb-afb5d87b434d_1290x1600.jpeg "Diagram divided into sections showing camera perspectives with purple stick figures: top left illustrates eye levels from low to high with horizon lines and simple scenes like a car; top right displays shot types including pull shot, medium shot, close shot, extreme close shot with figures waving or smiling; bottom left features over the shoulder, dutch angle, bird eye views with figures and clapperboards; bottom right shows push in pull out, dutch angle, movements to left right, and warnings about eye levels and mistakes with a cat and car illustrations.")

* * *

That’s a wrap for Multimodal Monday #30! From WALT and UltraCUA turning websites into smart APIs, to Seed3D 1.0 and WorldMirror delivering production-ready 3D assets in seconds, to AGILE boosting VLM accuracy through interactive learning, this week highlights multimodal AI’s shift toward practical, real-time solutions for automation, creation, and understanding.

_Ready to build multimodal solutions that actually work?_ [_Let's talk._](https://mixpeek.com/contact?ref=blog.mixpeek.com)

[PB

Philip Bankier

Philip’s expertise in RAG, data engineering, and AI agents makes him a natural fit to write about scaling multimodal data warehousing systems.



](/blog/authors/philip)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_end&utm_campaign=mvs_onramp)[Talk to us](/demo)

## Continue Learning

[

![Multimodal Monday #45: Birds, Whales, and the End of Latency](https://blog.mixpeek.com/content/images/2026/02/mm_45.png)

### Multimodal Monday #45: Birds, Whales, and the End of Latency

Your Weekly Multimodal AI Roundup (Feb 9 - Feb 16)

Read More





](/blog/multimodal-monday-45)[

![Multimodal Monday #44: Agents Ship Code, Robots Skateboard](https://blog.mixpeek.com/content/images/2026/02/mm44---with-milo.png)

### Multimodal Monday #44: Agents Ship Code, Robots Skateboard

Feb 2 - 9: GPT-5.3-Codex and Claude Opus 4.6 handle full software lifecycles from debug to deploy, MiniCPM-o 4.5 beats GPT-4o on vision tasks at 9B parameters running on-device, HUSKY skateboards using physics-based control, and TinyLoRA fine-tunes models with a single parameter.

Read More





](/blog/multimodal-monday-44)

### Author

PB

Philip Bankier

![Mixpeek Icon](/brand/logo-dark-icon.webp)

#### Multimodal Search, On Demand

Index, retrieve, and reason over videos, images, audio, and docs, all from a single API.

[Start Building](/start)

### Share

[](https://twitter.com/intent/tweet?text=Multimodal%20Monday%20%2330%3A%20Smarter%20Agents%2C%20Real-Time%203D&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-30-smarter-agents)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-30-smarter-agents&title=Multimodal%20Monday%20%2330%3A%20Smarter%20Agents%2C%20Real-Time%203D)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fmultimodal-monday-30-smarter-agents&title=Multimodal%20Monday%20%2330%3A%20Smarter%20Agents%2C%20Real-Time%203D)