Agent Perception

18 min read

Updated 2026-06-06

# Computer-Use Agent Memory: How to Search Screens, Tools, and UI State

A practical architecture guide for indexing computer-use agent traces. Learn how to turn screenshots, UI elements, actions, tool outputs, and state transitions into searchable memory with evidence, provenance, budgets, and evals.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

AI Agents

Computer Use

Agent Memory

Screenshots

Multimodal Search

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## Why Computer-Use Agents Need Searchable Memory

  
  
Computer-use agents operate through a loop:  
  
1\. Observe the screen. 2. Choose an action. 3. Click, type, scroll, or call a tool. 4. Observe the new screen. 5. Decide what changed.  
  
That loop creates a rich trace, but most systems store it as logs and screenshots. Logs capture actions but miss visual state. Screenshots capture pixels but are hard to search. Browser events capture URLs but not intent. Tool outputs capture results but not whether the agent used them correctly.  
  
If the agent fails, the debugging question is rarely "what did the model say?" It is:  
  
*   Which screen state caused the bad action?
*   Did the expected button exist before the click?
*   Was the agent looking at stale context?
*   Did a tool result contradict the visible UI?
*   Has this failure happened before in a similar workflow?
*   Which prior run contains the closest successful state?
  
  
Those are retrieval questions over unstructured agent traces. A computer-use memory system turns every step into searchable evidence.  
  
The hard gate for this topic is direct: searchable computer-use memory helps AI agents see screens, inspect UI state, and search unstructured task traces.  
  

## The Core Architecture

  
  
Treat each agent step as an observation packet.

computer-use run
  -> state transition
  -> observation packet
  -> multimodal indexes
  -> bounded retrieval tool
  -> evidence for the next agent step

An observation packet is not just a screenshot. It combines what the agent saw, what it did, what changed, and how to cite the evidence.

{
  "observation\_id": "obs\_run\_42\_step\_08",
  "run\_id": "run\_42",
  "step": 8,
  "timestamp": "2026-06-06T15:20:11Z",
  "task": "reset account password",
  "environment": "browser",
  "source\_uri": "s3://agent-runs/run\_42/step\_08.png",
  "url": "https://app.example.com/settings",
  "action\_before": {
    "type": "click",
    "target": "Save",
    "x": 812,
    "y": 718
  },
  "visual\_state": {
    "caption": "Settings page with password form and disabled save button",
    "ocr": \["Current password", "New password", "Save"\],
    "ui\_elements": \[
      {"role": "button", "text": "Save", "bbox": \[760, 690, 860, 735\], "enabled": false}
    \]
  },
  "tool\_outputs": \[
    {"tool": "validate\_password\_policy", "result": "missing\_special\_character"}
  \],
  "provenance": {
    "screenshot\_model": "Hcompany/Holo-3.1-4B",
    "ocr\_model": "PaddleOCR-VL",
    "extractor\_version": "2026-06-06"
  }
}

This packet gives future agents something concrete to retrieve. It can be searched by visible text, UI layout, action type, tool result, task, URL, or visual similarity.  
  

## Segment Runs by State Transitions

  
  
Document RAG chunks text. Video RAG chunks time. Computer-use memory chunks state transitions.  
  
A state transition is the interval between one observation and the next:

screen\_before + action + tool\_output + screen\_after

This is the useful retrieval unit because it preserves causality:  
  
*   what was visible before the action
*   what action the agent chose
*   what tool or browser event executed
*   what changed on the screen
*   whether the result matched the intent
  
  
Store both single-state packets and transition packets.  
  
Single-state packets answer:  
  
*   "Find screens where the Save button was disabled."
*   "Find prior runs on the billing page."
*   "Find screenshots with error code AUTH-429."
  
  
Transition packets answer:  
  
*   "Find cases where clicking Save did nothing."
*   "Find successful transitions from checkout to payment confirmation."
*   "Find runs where the agent clicked a hidden or disabled control."
  
  
Most failures are transition failures, not state failures. The screen looked plausible, but the next action was wrong.  
  

## Extract Multiple Channels

  
  
Do not force screenshots, actions, OCR, DOM, and tool calls into one vector. Use separate channels and fuse results later.  
  

### Screenshot Caption Channel

  
  
A vision-language model can summarize the screen:  
  
*   page or app type
*   visible controls
*   layout
*   error messages
*   affordances
*   apparent state
  
  
This channel supports natural language queries like "modal open with a disabled submit button" or "pricing table where annual billing is selected."  
  

### OCR and UI Text Channel

  
  
OCR and accessibility text preserve exact strings. This matters for:  
  
*   error codes
*   button labels
*   form labels
*   product names
*   amounts
*   table values
*   legal or compliance text
  
  
Dense visual embeddings are weak at exact strings. Keep OCR searchable with lexical retrieval and filters.  
  

### UI Element Channel

  
  
When possible, extract structured UI elements:  
  
*   role: button, input, link, menu, checkbox
*   text or accessible name
*   bounding box
*   enabled or disabled state
*   selected state
*   hierarchy
*   DOM selector or accessibility path
  
  
For web and desktop agents, this channel is often more reliable than pixels alone. For custom canvases, mobile screenshots, and older apps, you may need OCR and vision grounding as the fallback.  
  

### Action and Tool Channel

  
  
Actions and tool outputs form the agent's operational memory:  
  
*   click coordinates
*   typed text, redacted when sensitive
*   scroll direction
*   selected tool
*   tool arguments
*   tool output
*   safety checks
*   retry count
*   cancellation reason
  
  
This channel explains why a state exists. It is the difference between "the page showed an error" and "the page showed an error after the agent submitted a password that failed policy validation."  
  

### Visual Embedding Channel

  
  
Visual embeddings support fuzzy search over layouts and states:  
  
*   similar modal layouts
*   similar checkout failures
*   similar visual diff patterns
*   same UI in different themes or languages
*   repeated app states with different text
  
  
Use visual embeddings for recall. Use OCR, UI elements, and reranking for precision.  
  

## Use Multi-Index Retrieval

  
  
A computer-use memory search should query several indexes independently:

agent task or debug query
  -> screenshot caption search
  -> OCR/BM25 search
  -> UI element filter
  -> action/tool search
  -> visual embedding search
  -> fusion
  -> reranking
  -> evidence packet

The score scales are not comparable. BM25 over OCR, cosine similarity over screenshots, and a UI element filter do not produce the same kind of confidence. Start with rank-based fusion such as Reciprocal Rank Fusion, then rerank the top candidates with the full query and packet context.  
  
The result should tell the agent why it matched:

{
  "source\_uri": "s3://agent-runs/run\_42/step\_08.png",
  "run\_id": "run\_42",
  "step": 8,
  "summary": "Password settings page with disabled Save button",
  "matched\_channels": \["ocr", "ui\_element", "visual\_embedding"\],
  "matched\_evidence": \[
    {"channel": "ocr", "text": "Save"},
    {"channel": "ui\_element", "role": "button", "enabled": false},
    {"channel": "tool", "result": "missing\_special\_character"}
  \],
  "next\_actions": \["inspect\_neighboring\_steps", "compare\_successful\_transition"\]
}

The agent should not receive a vague memory blob. It should receive evidence with a screen, a step, a reason, and allowed follow-up actions.  
  

## Design the Agent Tool

  
  
Expose computer-use memory as a bounded tool.

{
  "tool": "search\_computer\_use\_memory",
  "input\_schema": {
    "query": "string",
    "run\_filters": {
      "app": "string",
      "environment": "browser | desktop | mobile",
      "task": "string",
      "url\_contains": "string"
    },
    "channels": \["screenshot", "ocr", "ui\_element", "action", "tool\_output"\],
    "top\_k": 20,
    "include\_neighbor\_steps": true,
    "budget\_ms": 2000
  }
}

Make the tool return observations, not conclusions. The agent can use the observations to decide whether to retry, inspect a neighbor step, compare against a successful run, or ask a human for review.  
  
Good tool outputs include:  
  
*   screenshot URI or signed URL
*   step number and run ID
*   matched channels
*   UI element coordinates or selectors
*   action before and after
*   tool outputs that influenced the state
*   confidence and rank
*   provenance for every extracted feature
  
  
Bad tool outputs only say "similar screen found." That forces the model to infer too much.  
  

## Evaluate Memory Quality

  
  
Computer-use memory needs retrieval evals and trace evals.  
  
Retrieval eval examples:  
  
*   Find the prior run where the billing modal failed after submit.
*   Find the screenshot with error code AUTH-429.
*   Find successful states that look like this failed state.
*   Find the step before the agent clicked a disabled button.
*   Find cases where the tool result said success but the UI still showed pending.
  
  
Trace eval examples:  
  
*   Did the agent call memory before retrying the same failed action?
*   Did it search the right channel, OCR for an error code and visual similarity for a layout?
*   Did it cite the exact step it used?
*   Did it inspect neighbor steps before concluding cause?
*   Did it stop when memory returned no high-confidence match?
  
  
Useful metrics:  
  
*   Recall@k for known prior states.
*   nDCG by task class.
*   UI element match accuracy.
*   Step localization error.
*   Screenshot citation rate.
*   Unsupported visual claim rate.
*   Stale-context reuse rate.
*   Cost per successful retrieved observation.
*   P95 tool latency.
*   Cancellation rate for superseded searches.
  
  
The key is separating "retrieved the right packet" from "used the packet correctly." A memory system can retrieve the right evidence while the agent ignores it.  
  

## Security and Privacy Controls

  
  
Computer-use traces can contain sensitive data. Treat them as production data, not debug leftovers.  
  
Design controls before indexing:  
  
*   redact passwords, tokens, addresses, and payment fields
*   store screenshot crops instead of full screens when possible
*   attach tenant, user, run, and app ACLs to every packet
*   separate raw screenshots from derived features
*   expire temporary task memory
*   preserve audit lineage for who searched what
*   block retrieval across tenants by default
  
  
Memory is useful because it is searchable. That also makes access control more important.  
  

## Mixpeek Implementation Pattern

  
  
Use object storage as the source of truth for screenshots, screen recordings, and trace artifacts. Then index each observation packet with multiple feature channels.

from mixpeek import Mixpeek

mx = Mixpeek(api\_key="YOUR\_API\_KEY", namespace="my-namespace")

# Two collections over the run's screenshots, one feature channel each, since a
# collection takes one extractor. run\_id, task and step travel as object
# metadata on the screenshots themselves.

# Fuzzy screen similarity: mixpeek://image\_extractor@v1/google\_siglip\_base\_v1
mx.collections.create(
    collection\_name="screens\_visual",
    source={"type": "bucket", "bucket\_ids": \["bkt\_agent\_runs"\]},
    feature\_extractor={"feature\_extractor\_name": "image\_extractor", "version": "v1"},
)

# On-screen text and a generated description of the screen state, with a
# Gemini embedding: mixpeek://universal\_extractor@v1/gemini-embedding-2
mx.collections.create(
    collection\_name="screens\_text",
    source={"type": "bucket", "bucket\_ids": \["bkt\_agent\_runs"\]},
    feature\_extractor={
        "feature\_extractor\_name": "universal\_extractor",
        "version": "v1",
        "parameters": {"extract\_text": True, "generate\_description": True},
    },
)

Then expose a retrieval tool to the agent:

results = mx.retrievers.execute(
    "your-retriever-id",
    inputs={"query": "disabled save button after password policy validation error"},
)

If you bring your own screenshot embeddings, MVS can store and search them on object storage. If you want Mixpeek to run the perception layer, Managed indexing can populate captions, OCR, visual embeddings, and trace features from the objects you already keep in storage.  
  

## Design Checklist

  
  
*   Store one observation packet per screen state.
*   Store transition packets for before-action-after sequences.
*   Keep screenshot, OCR, UI element, action, tool, and visual embedding channels separate.
*   Preserve run ID, step number, timestamp, source URI, model ID, and extractor version.
*   Use lexical retrieval for exact visible text and error codes.
*   Use visual embeddings for fuzzy screen similarity.
*   Fuse multiple channels before reranking.
*   Return evidence packets, not final answers.
*   Let agents request neighbor steps and successful comparable transitions.
*   Evaluate both retrieval quality and agent tool behavior.
*   Enforce ACLs, redaction, and retention before indexing.
*   Track latency, budget, cancellation, and stale-context reuse in production.
  
  

## Key Takeaways

  
  
1\. Computer-use traces are multimodal evidence, not just logs.  
  
2\. The best retrieval unit is the observation packet: screen state, action, tool output, and provenance.  
  
3\. Screenshots need multiple indexes. Visual embeddings, OCR, UI elements, and action traces answer different questions.  
  
4\. Agent memory tools should return bounded evidence with citations and follow-up actions.  
  
5\. Evaluation must test retrieval, step localization, and whether the agent used the memory correctly.  
  
6\. Searchable memory makes agents more capable, but only if access control and redaction are designed into the index.  
  

## Further Reading

  
  
*   [Object Decomposition and Layered Indexing](/guides/object-decomposition-layered-indexing)
*   [Agent Perception Evals](/guides/agent-perception-evals)
*   [Retrieval Control Planes for AI Agents](/guides/retrieval-control-planes-for-ai-agents)
*   [MCP Tool Design for Multimodal Search](/guides/mcp-tool-design-for-multimodal-search)
*   [OpenAI Computer Use](https://platform.openai.com/docs/guides/tools-computer-use)
*   [OpenAI Agent Evals](https://platform.openai.com/docs/guides/agent-evals)
*   [Model Context Protocol tools](https://modelcontextprotocol.io/specification/draft/server/tools)
*   [Holo-3.1-4B on Hugging Face](https://huggingface.co/Hcompany/Holo-3.1-4B)

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Agent Perception

### Agent Perception Evals: Testing Whether AI Agents Can See, Hear, and Search

A practical guide to evaluating agent perception systems. Learn how to test multimodal ingestion coverage, retrieval quality, temporal grounding, tool-call behavior, evidence citation, and production drift for agents that search video, audio, images, and documents.

Read guide →](/guides/agent-perception-evals)[

Agent Perception

### MCP Tool Design for Multimodal Search

A technical guide to building MCP (Model Context Protocol) tools that expose multimodal search to AI agents. Covers tool schema design, embedding handoff patterns, streaming results, and how to structure image/audio/video search as composable MCP resources.

Read guide →](/guides/mcp-tool-design-for-multimodal-search)[

Agent Perception

### Why Is My AI Agent Using Thousands of Tokens Before I Ask It Anything?

Every MCP server you connect sends its full tool list with every request, and the schemas cost more than the descriptions. We counted the tokens for Mixpeek's own MCP server (68,605 for all 67 tools) and Google's official Google Ads MCP server (3,668 for 3 tools), and show how to cut the bill.

Read guide →](/guides/why-is-my-ai-agent-using-thousands-of-tokens-before-i-ask-it-anything)