Retrieval

10 min read

Updated 2026-10-28

# Cyberdefense as a search problem: narrow billions of audit logs to the few an AI investigator should read

How to build a two-stage threat detection loop on Mixpeek: keep every audit event in object storage, count activity per identity, flag rate changes and first-time actions, then let a read-only model investigate the few dozen actors that changed. Working retriever configs for each step.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Security

Threat Detection

Audit Logs

Agentic Search

Object Storage

MVS

[Index every log on object storage. Try it as you read.Mixpeek Vector Store →](/mvs)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## Why treat security monitoring as search?

  
  
Because the expensive part of finding an attacker is reading. A cloud fleet produces hundreds of millions of audit events a day, and almost all of them are routine: certificates rotating, CI jobs deploying, engineers on call. An attacker with a stolen credential produces events that look like those. Static rules catch the clumsy attempts. A careful one uses a real identity to do things that identity is allowed to do.  
  
A frontier model can tell the two apart if it reads the right events with enough context. It cannot read all of them. At a few hundred tokens an event, one day of logs from a mid-sized fleet is tens of billions of tokens. So the job splits in two, the same way a search engine splits it:  
  
1\. **Retrieval.** Cheap, deterministic queries over every event narrow millions of events an hour to a few dozen actors whose behavior changed. 2. **Reranking.** A model investigates only those actors, pulling whatever raw events it needs, and decides which ones a human should see.  
  
turbopuffer described running exactly this loop on its own infrastructure in [Cyberdefense is a search problem](https://x.com/conor__landry/status/2115529397382091247) (October 2026), with Claude doing the investigation. This guide shows how to build the same loop on Mixpeek, where the index lives in your object storage bucket.  
  

## What you need from the storage layer

  
  

Requirement

Why

Mixpeek feature

Keep every event, cheaply

You cannot know in advance which source will matter

Namespaces stored as \[`.mvs` datasets\](https://mixpeek.com/docs/vector-store/storage-format) in object storage, with [automatic tiering](https://mixpeek.com/docs/vector-store/tiering) of idle data to cold and frozen classes

Count by actor, action and resource

"This actor did this, this many times" is the unit of analysis

The [aggregate stage](https://mixpeek.com/docs/retrieval/stages/aggregate) in a retriever, and the collection [aggregate endpoint](https://mixpeek.com/docs/api-reference/collection-documents/aggregate-documents) for exact counts over a whole collection

Compare a window with the one before

A sudden change in an actor's rate is the main signal

The [temporal stage](https://mixpeek.com/docs/retrieval/stages/temporal) with drift detection

Exact and fuzzy lookup

Investigators search for a key ID, then for "anything like this"

Payload indexes for filters, [BM25 keyword search](https://mixpeek.com/docs/retrieval/stages/feature-search) and vector search in one retriever

A model that can only read

The investigator must never change infrastructure

The [agent search stage](https://mixpeek.com/docs/retrieval/stages/agent-search), whose tools are retriever stages

  
  

## Step 1: normalize events and write them to a namespace

  
  
Keep the raw logs where they land, in a bucket that blocks deletes and overwrites. That copy is your evidence. Then normalize each event into a small common schema, such as a subset of [OCSF](https://schema.ocsf.io/): who acted, what they did, on what, from where, whether it worked and when. Add plain lookups such as the credential type and the kind of network. No model is needed for this step.  
  
Every Mixpeek document carries at least one vector, so embed a one-line summary of each event with a small text model. That costs little and lets an investigator later search events by meaning ("service account reading a bucket it does not own").

import requests

API = "https://api.mixpeek.com/v1"
H = {"Authorization": "Bearer MIXPEEK\_API\_KEY", "Content-Type": "application/json"}
NS = "audit-gcp-2026-10-28"   # one namespace per source per day

# One-time: create the namespace and index the fields you filter and group on.
requests.post(f"{API}/namespaces/standalone", headers=H, json={
    "namespace\_id": NS,
    "vector\_configs": \[{"name": "audit\_summary\_v1", "dimension": 384, "metric": "cosine"}\],
})
requests.patch(f"{API}/namespaces/{NS}", headers=H, json={"payload\_indexes": \[
    {"field\_name": "actor", "type": "keyword"},
    {"field\_name": "action", "type": "keyword"},
    {"field\_name": "resource", "type": "keyword"},
    {"field\_name": "network", "type": "keyword"},
    {"field\_name": "event\_time", "type": "datetime"},
    {"field\_name": "summary", "type": "text"},
\]})

# Then, continuously, in batches of up to 1,000 events:
requests.post(f"{API}/namespaces/{NS}/documents/upsert", headers=H, json={"documents": \[{
    "document\_id": event\["id"\],
    "vectors": {"audit\_summary\_v1": embed(event\["summary"\])},
    "payload": {
        "actor": event\["actor"\], "action": event\["action"\], "resource": event\["resource"\],
        "network": event\["network"\], "credential\_type": event\["credential\_type"\],
        "outcome": event\["outcome"\], "event\_time": event\["time"\], "summary": event\["summary"\],
    },
} for event in batch\]})

A namespace per source per day keeps queries small and lets old days go cold on their own. Set the idle windows with the `tiering` field, or exempt a namespace under legal hold with `{"tiering": {"exempt": "legal hold"}}`.  
  

## Step 2: count activity per actor

  
  
Every action traces back to an identity: an engineer, a service account, a CI job. A fleet has a few thousand of them, and each one has a purpose (written in your Terraform) and habits (visible in its history). The first query turns an hour of events into one row per actor, action and resource.

requests.post(f"{API}/retrievers", headers={\*\*H, "X-Namespace": NS}, json={
    "retriever\_name": "iam-changes-per-actor",
    "input\_schema": {"start": {"type": "string", "required": True},
                     "end": {"type": "string", "required": True}},
    "stages": \[
        {"stage\_name": "iam\_actions", "stage\_type": "filter", "config": {
            "stage\_id": "attribute\_filter",
            "parameters": {"field": "action", "operator": "in",
                           "value": \["SetIamPolicy", "CreateServiceAccountKey", "CreateRole"\],
                           "max\_scan\_documents": 100000}}},
        {"stage\_name": "after\_start", "stage\_type": "filter", "config": {
            "stage\_id": "attribute\_filter",
            "parameters": {"field": "event\_time", "operator": "gte", "value": "{{INPUT.start}}"}}},
        {"stage\_name": "before\_end", "stage\_type": "filter", "config": {
            "stage\_id": "attribute\_filter",
            "parameters": {"field": "event\_time", "operator": "lt", "value": "{{INPUT.end}}"}}},
        {"stage\_name": "per\_actor", "stage\_type": "reduce", "config": {
            "stage\_id": "aggregate",
            "parameters": {
                "group\_by": \[{"field": "actor"}, {"field": "action"}, {"field": "resource"}\],
                "aggregations": \[{"function": "count", "alias": "events"},
                                 {"function": "count\_distinct", "field": "network", "alias": "networks"}\],
                "sort\_by": "events", "limit": 1000}}},
    \],
})

The groups come back in `stage_statistics.stages.per_actor.metadata.aggregations`. A retriever's first stage scans up to 100,000 documents per collection, which covers an hour of one sensitive action family on most fleets. For exact counts over a whole day or more, call `POST /v1/collections/{collection}/documents/aggregate` instead. It takes the same `group_by` and `aggregations`, adds `filters` and `having`, and has no row cap.  
  

## Step 3: compare each actor with its own history

  
  
Two kinds of change matter.  
  
**Rate changes.** An actor that usually makes no IAM changes and made a dozen this hour is worth a look; a doubling of routine reads usually is not. Run the temporal stage over one actor's recent history with drift detection, and it flags windows whose count moved more than your threshold:

{"stage\_name": "actor\_rate", "stage\_type": "reduce", "config": {
  "stage\_id": "temporal",
  "parameters": {
    "time\_field": "event\_time", "window": "hour",
    "aggregations": \[{"function": "count", "alias": "events"}\],
    "drift": {"enabled": true, "metric": "events", "threshold": 300.0}}}}

**First-time events.** An actor touching a bucket, a project or a network for the first time is one of the strongest signals of a stolen credential. Keep a small "first seen" namespace keyed by a hash of actor, action and resource. Before writing, `GET /v1/namespaces/{ns}/documents/{document_id}`: if it exists, the pair is not new; if it returns 404, write it with its `first_seen` time and pass it on. Upserts overwrite, so this read is what keeps the first timestamp from being replaced.  
  
Add static tripwires that always go to investigation regardless of score, such as a service-account key created outside your approved pipeline.  
  

## Step 4: score and keep the top few

  
  
Score each signal by how far it strayed and how sensitive it is. A new export path or a new CI branch opens a direct route out for data or code, so it scores high. A new network scores low for an engineer who travels and high for a service account. Then give each actor the score of its strongest signal. Taking the maximum stops a pile of weak routine signals from adding up to a false alarm, and keeps one serious signal from being averaged away.  
  
This step is plain Python over the rows from steps 2 and 3. The output is a few dozen actors with their scores and the signals behind them.  
  

## Step 5: let a model investigate, read-only

  
  
Each candidate actor goes to an investigating model with the ranker's context. The model needs to pull raw events and search past activity, and it must not be able to change anything. An `agent_search` stage gives it exactly that: its tools are other retriever stages.

{"stage\_name": "investigate", "stage\_type": "filter", "config": {
  "stage\_id": "agent\_search",
  "parameters": {
    "strategy": "custom",
    "stages": \["attribute\_filter", "feature\_search"\],
    "system\_prompt": "You are investigating actor {{INPUT.actor}}, flagged for: {{INPUT.signals}}. Pull the raw events behind each signal with attribute\_filter, compare with the actor's earlier activity, and search past findings. Report whether this needs a human, with the event IDs that support your answer.",
    "provider": "anthropic",
    "max\_iterations": 8,
    "timeout\_seconds": 120.0}}}

You can also run the investigator outside Mixpeek and give it the retriever as a tool through the [Mixpeek MCP server](https://mixpeek.com/docs/integrations/developer-tools/mcp-server). Either way, the model reads; it holds no credentials to your cloud.  
  

## Step 6: notify a human and remember the finding

  
  
When the investigator decides a finding needs a person, post it to your incident channel. Then write every finding back, false positives included, with its severity, evidence, a full-text summary and a fingerprint of the activity. Index the summary as `text` so the next investigation can run a BM25 search (`"lexical": true` in `feature_search`) over past findings and pick up where an earlier one stopped, rather than reopening the same activity every hour.  
  

## Logs are not the only evidence

  
  
Some security evidence is not a log line: the screenshot attached to a phishing report, the recording of a voice-phishing call, a badge-reader camera feed, the PDF a user was tricked into opening. Mixpeek extracts these into the same namespaces: [image](https://mixpeek.com/docs/processing/extractors/image) embeddings for screenshots, [document](https://mixpeek.com/docs/processing/extractors/document) text and layout for PDFs, [audio and video](https://mixpeek.com/docs/processing/extractors/multimodal) for recordings, and [live camera streams](https://mixpeek.com/docs/integrations/streaming/rtsp). An investigator can then search them alongside the events, for example every phishing report whose screenshot looks like this fake login page.  
  

## What this costs, and where it does not fit

  
  
The index sits in object storage, and idle days demote to cold and frozen classes on their own. Compute goes to the hourly queries and to the few dozen investigations, which is the point of the design.  
  
Three limits to plan around:  
  
*   **Cold data has to come back first.** A collection in the `cold` [lifecycle state](https://mixpeek.com/docs/vector-store/collection-lifecycle) is rehydrated before vector search runs on it. Keep the days you query every hour active.
*   **Retriever scans are bounded.** Use the collection aggregate endpoint for exact counts beyond 100,000 documents per collection.
*   **This finds changes in behavior.** An attacker who behaves exactly like the identity they stole, from the same network at the same hours, will not stand out. Pair this with your existing rules and endpoint tooling.
  
  

## Frequently asked questions

  
  
**Do I need a SIEM to do this?** No. The raw logs stay in your bucket, the normalized events live in Mixpeek namespaces on object storage, and the queries above replace the correlation searches. You can keep a SIEM for compliance reporting and run this loop for detection.  
  
**Why not send every log to the model?** Cost and attention. Even at a few hundred tokens per event, one day of logs from a fleet is tens of billions of tokens, and most of it is routine. Narrowing first means the model reads a few dozen actors with full context.  
  
**Which model should investigate?** Any model that can call tools. The `agent_search` stage takes a `provider` and `model_name`, and an external agent can call the same retrievers over MCP.  
  
**Can the investigating model change my infrastructure?** Not through this setup. Its only tools are retriever stages, which read from the index. Give it no cloud credentials.  
  

## Sources

  
  
*   turbopuffer, [Cyberdefense is a search problem: finding threats in our infra with turbopuffer and Mythos](https://x.com/conor__landry/status/2115529397382091247), October 2026
*   [OCSF schema](https://schema.ocsf.io/)
*   Mixpeek docs: [aggregate stage](https://mixpeek.com/docs/retrieval/stages/aggregate), [temporal stage](https://mixpeek.com/docs/retrieval/stages/temporal), [agent search](https://mixpeek.com/docs/retrieval/stages/agent-search), [storage tiering](https://mixpeek.com/docs/vector-store/tiering), [storage format](https://mixpeek.com/docs/vector-store/storage-format)

## Index every log on object storage

Mixpeek keeps each namespace as a documented dataset in object storage, with keyword, vector and BM25 search, aggregations and automatic tiering of idle data.

[Mixpeek Vector Store](/mvs)[Agent search docs](https://mixpeek.com/docs/retrieval/stages/agent-search)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Retrieval

### How Do You Run Agentic Search Over a Hierarchy of Clusters?

Cluster the corpus into a labeled tree, then answer a query by walking the tree from the top, choosing the child cluster most likely to hold the answer at every level. A decision model such as TypeSafe's Jev returns a probability for each branch, so the walk can keep several paths open and rank them. This covers how to build the tree, how the walk works, and when it beats a single vector search.

Read guide →](/guides/agentic-search-over-cluster-hierarchies)[

Retrieval

### Adaptive Indexing for Agentic Search: Query Logs, Payload Indexes, and Retrieval Routing

Learn how retrieval systems decide which indexes to build as agents search unstructured content. Covers query logs, slow-query diagnosis, payload indexes, hybrid routing, and Mixpeek MVS examples.

Read guide →](/guides/adaptive-indexing-for-agentic-search)[

Retrieval

### Payload Projection for Agentic Vector Search: Field Selection, Evidence Handles, and Context Budgets

Learn how field projection keeps agent retrieval fast, citeable, and context-efficient. Covers payload design, late materialization, evidence envelopes, select\_fields, and Mixpeek MVS examples for unstructured media search.

Read guide →](/guides/payload-projection-for-agentic-vector-search)