Why treat security monitoring as search?
Because the expensive part of finding an attacker is reading. A cloud fleet produces hundreds of millions of audit events a day, and almost all of them are routine: certificates rotating, CI jobs deploying, engineers on call. An attacker with a stolen credential produces events that look like those. Static rules catch the clumsy attempts. A careful one uses a real identity to do things that identity is allowed to do.
A frontier model can tell the two apart if it reads the right events with enough context. It cannot read all of them. At a few hundred tokens an event, one day of logs from a mid-sized fleet is tens of billions of tokens. So the job splits in two, the same way a search engine splits it:
1. Retrieval. Cheap, deterministic queries over every event narrow millions of events an hour to a few dozen actors whose behavior changed. 2. Reranking. A model investigates only those actors, pulling whatever raw events it needs, and decides which ones a human should see.
turbopuffer described running exactly this loop on its own infrastructure in Cyberdefense is a search problem (October 2026), with Claude doing the investigation. This guide shows how to build the same loop on Mixpeek, where the index lives in your object storage bucket.
What you need from the storage layer
| Requirement | Why | Mixpeek feature |
|---|---|---|
| Keep every event, cheaply | You cannot know in advance which source will matter | Namespaces stored as [.mvs datasets](https://mixpeek.com/docs/operations/storage-format) in object storage, with automatic tiering of idle data to cold and frozen classes |
| Count by actor, action and resource | "This actor did this, this many times" is the unit of analysis | The aggregate stage in a retriever, and the collection aggregate endpoint for exact counts over a whole collection |
| Compare a window with the one before | A sudden change in an actor's rate is the main signal | The temporal stage with drift detection |
| Exact and fuzzy lookup | Investigators search for a key ID, then for "anything like this" | Payload indexes for filters, BM25 keyword search and vector search in one retriever |
| A model that can only read | The investigator must never change infrastructure | The agent search stage, whose tools are retriever stages |
Step 1: normalize events and write them to a namespace
Keep the raw logs where they land, in a bucket that blocks deletes and overwrites. That copy is your evidence. Then normalize each event into a small common schema, such as a subset of OCSF: who acted, what they did, on what, from where, whether it worked and when. Add plain lookups such as the credential type and the kind of network. No model is needed for this step.
Every Mixpeek document carries at least one vector, so embed a one-line summary of each event with a small text model. That costs little and lets an investigator later search events by meaning ("service account reading a bucket it does not own").
import requests
API = "https://api.mixpeek.com/v1"
H = {"Authorization": "Bearer MIXPEEK_API_KEY", "Content-Type": "application/json"}
NS = "audit-gcp-2026-10-28" # one namespace per source per day
# One-time: create the namespace and index the fields you filter and group on.
requests.post(f"{API}/namespaces/standalone", headers=H, json={
"namespace_id": NS,
"vector_configs": [{"name": "audit_summary_v1", "dimension": 384, "metric": "cosine"}],
})
requests.patch(f"{API}/namespaces/{NS}", headers=H, json={"payload_indexes": [
{"field_name": "actor", "type": "keyword"},
{"field_name": "action", "type": "keyword"},
{"field_name": "resource", "type": "keyword"},
{"field_name": "network", "type": "keyword"},
{"field_name": "event_time", "type": "datetime"},
{"field_name": "summary", "type": "text"},
]})
# Then, continuously, in batches of up to 1,000 events:
requests.post(f"{API}/namespaces/{NS}/documents/upsert", headers=H, json={"documents": [{
"document_id": event["id"],
"vectors": {"audit_summary_v1": embed(event["summary"])},
"payload": {
"actor": event["actor"], "action": event["action"], "resource": event["resource"],
"network": event["network"], "credential_type": event["credential_type"],
"outcome": event["outcome"], "event_time": event["time"], "summary": event["summary"],
},
} for event in batch]})A namespace per source per day keeps queries small and lets old days go cold on their own. Set the idle windows with the
Every action traces back to an identity: an engineer, a service account, a CI job. A fleet has a few thousand of them, and each one has a purpose (written in your Terraform) and habits (visible in its history). The first query turns an hour of events into one row per actor, action and resource.
tiering field, or exempt a namespace under legal hold with {"tiering": {"exempt": "legal hold"}}.Step 2: count activity per actor
Every action traces back to an identity: an engineer, a service account, a CI job. A fleet has a few thousand of them, and each one has a purpose (written in your Terraform) and habits (visible in its history). The first query turns an hour of events into one row per actor, action and resource.
requests.post(f"{API}/retrievers", headers={**H, "X-Namespace": NS}, json={
"retriever_name": "iam-changes-per-actor",
"input_schema": {"start": {"type": "string", "required": True},
"end": {"type": "string", "required": True}},
"stages": [
{"stage_name": "iam_actions", "stage_type": "filter", "config": {
"stage_id": "attribute_filter",
"parameters": {"field": "action", "operator": "in",
"value": ["SetIamPolicy", "CreateServiceAccountKey", "CreateRole"],
"max_scan_documents": 100000}}},
{"stage_name": "after_start", "stage_type": "filter", "config": {
"stage_id": "attribute_filter",
"parameters": {"field": "event_time", "operator": "gte", "value": "{{INPUT.start}}"}}},
{"stage_name": "before_end", "stage_type": "filter", "config": {
"stage_id": "attribute_filter",
"parameters": {"field": "event_time", "operator": "lt", "value": "{{INPUT.end}}"}}},
{"stage_name": "per_actor", "stage_type": "reduce", "config": {
"stage_id": "aggregate",
"parameters": {
"group_by": [{"field": "actor"}, {"field": "action"}, {"field": "resource"}],
"aggregations": [{"function": "count", "alias": "events"},
{"function": "count_distinct", "field": "network", "alias": "networks"}],
"sort_by": "events", "limit": 1000}}},
],
})The groups come back in
Two kinds of change matter.
Rate changes. An actor that usually makes no IAM changes and made a dozen this hour is worth a look; a doubling of routine reads usually is not. Run the temporal stage over one actor's recent history with drift detection, and it flags windows whose count moved more than your threshold:
stage_statistics.stages.per_actor.metadata.aggregations. A retriever's first stage scans up to 100,000 documents per collection, which covers an hour of one sensitive action family on most fleets. For exact counts over a whole day or more, call POST /v1/collections/{collection}/documents/aggregate instead. It takes the same group_by and aggregations, adds filters and having, and has no row cap.Step 3: compare each actor with its own history
Two kinds of change matter.
Rate changes. An actor that usually makes no IAM changes and made a dozen this hour is worth a look; a doubling of routine reads usually is not. Run the temporal stage over one actor's recent history with drift detection, and it flags windows whose count moved more than your threshold:
{"stage_name": "actor_rate", "stage_type": "reduce", "config": {
"stage_id": "temporal",
"parameters": {
"time_field": "event_time", "window": "hour",
"aggregations": [{"function": "count", "alias": "events"}],
"drift": {"enabled": true, "metric": "events", "threshold": 300.0}}}}First-time events. An actor touching a bucket, a project or a network for the first time is one of the strongest signals of a stolen credential. Keep a small "first seen" namespace keyed by a hash of actor, action and resource. Before writing,
Add static tripwires that always go to investigation regardless of score, such as a service-account key created outside your approved pipeline.
Score each signal by how far it strayed and how sensitive it is. A new export path or a new CI branch opens a direct route out for data or code, so it scores high. A new network scores low for an engineer who travels and high for a service account. Then give each actor the score of its strongest signal. Taking the maximum stops a pile of weak routine signals from adding up to a false alarm, and keeps one serious signal from being averaged away.
This step is plain Python over the rows from steps 2 and 3. The output is a few dozen actors with their scores and the signals behind them.
Each candidate actor goes to an investigating model with the ranker's context. The model needs to pull raw events and search past activity, and it must not be able to change anything. An
GET /v1/namespaces/{ns}/documents/{document_id}: if it exists, the pair is not new; if it returns 404, write it with its first_seen time and pass it on. Upserts overwrite, so this read is what keeps the first timestamp from being replaced.Add static tripwires that always go to investigation regardless of score, such as a service-account key created outside your approved pipeline.
Step 4: score and keep the top few
Score each signal by how far it strayed and how sensitive it is. A new export path or a new CI branch opens a direct route out for data or code, so it scores high. A new network scores low for an engineer who travels and high for a service account. Then give each actor the score of its strongest signal. Taking the maximum stops a pile of weak routine signals from adding up to a false alarm, and keeps one serious signal from being averaged away.
This step is plain Python over the rows from steps 2 and 3. The output is a few dozen actors with their scores and the signals behind them.
Step 5: let a model investigate, read-only
Each candidate actor goes to an investigating model with the ranker's context. The model needs to pull raw events and search past activity, and it must not be able to change anything. An
agent_search stage gives it exactly that: its tools are other retriever stages.{"stage_name": "investigate", "stage_type": "filter", "config": {
"stage_id": "agent_search",
"parameters": {
"strategy": "custom",
"stages": ["attribute_filter", "feature_search"],
"system_prompt": "You are investigating actor {{INPUT.actor}}, flagged for: {{INPUT.signals}}. Pull the raw events behind each signal with attribute_filter, compare with the actor's earlier activity, and search past findings. Report whether this needs a human, with the event IDs that support your answer.",
"provider": "anthropic",
"max_iterations": 8,
"timeout_seconds": 120.0}}}You can also run the investigator outside Mixpeek and give it the retriever as a tool through the Mixpeek MCP server. Either way, the model reads; it holds no credentials to your cloud.
When the investigator decides a finding needs a person, post it to your incident channel. Then write every finding back, false positives included, with its severity, evidence, a full-text summary and a fingerprint of the activity. Index the summary as
Some security evidence is not a log line: the screenshot attached to a phishing report, the recording of a voice-phishing call, a badge-reader camera feed, the PDF a user was tricked into opening. Mixpeek extracts these into the same namespaces: image embeddings for screenshots, document text and layout for PDFs, audio and video for recordings, and live camera streams. An investigator can then search them alongside the events, for example every phishing report whose screenshot looks like this fake login page.
The index sits in object storage, and idle days demote to cold and frozen classes on their own. Compute goes to the hourly queries and to the few dozen investigations, which is the point of the design.
Three limits to plan around:
Cold data has to come back first. A collection in the
Retriever scans are bounded. Use the collection aggregate endpoint for exact counts beyond 100,000 documents per collection.
This finds changes in behavior. An attacker who behaves exactly like the identity they stole, from the same network at the same hours, will not stand out. Pair this with your existing rules and endpoint tooling.
Do I need a SIEM to do this? No. The raw logs stay in your bucket, the normalized events live in Mixpeek namespaces on object storage, and the queries above replace the correlation searches. You can keep a SIEM for compliance reporting and run this loop for detection.
Why not send every log to the model? Cost and attention. Even at a few hundred tokens per event, one day of logs from a fleet is tens of billions of tokens, and most of it is routine. Narrowing first means the model reads a few dozen actors with full context.
Which model should investigate? Any model that can call tools. The
Can the investigating model change my infrastructure? Not through this setup. Its only tools are retriever stages, which read from the index. Give it no cloud credentials.
turbopuffer, Cyberdefense is a search problem: finding threats in our infra with turbopuffer and Mythos, October 2026
OCSF schema
Mixpeek docs: aggregate stage, temporal stage, agent search, storage tiering, storage format
Step 6: notify a human and remember the finding
When the investigator decides a finding needs a person, post it to your incident channel. Then write every finding back, false positives included, with its severity, evidence, a full-text summary and a fingerprint of the activity. Index the summary as
text so the next investigation can run a BM25 search ("lexical": true in feature_search) over past findings and pick up where an earlier one stopped, rather than reopening the same activity every hour.Logs are not the only evidence
Some security evidence is not a log line: the screenshot attached to a phishing report, the recording of a voice-phishing call, a badge-reader camera feed, the PDF a user was tricked into opening. Mixpeek extracts these into the same namespaces: image embeddings for screenshots, document text and layout for PDFs, audio and video for recordings, and live camera streams. An investigator can then search them alongside the events, for example every phishing report whose screenshot looks like this fake login page.
What this costs, and where it does not fit
The index sits in object storage, and idle days demote to cold and frozen classes on their own. Compute goes to the hourly queries and to the few dozen investigations, which is the point of the design.
Three limits to plan around:
cold lifecycle state is rehydrated before vector search runs on it. Keep the days you query every hour active.Frequently asked questions
Do I need a SIEM to do this? No. The raw logs stay in your bucket, the normalized events live in Mixpeek namespaces on object storage, and the queries above replace the correlation searches. You can keep a SIEM for compliance reporting and run this loop for detection.
Why not send every log to the model? Cost and attention. Even at a few hundred tokens per event, one day of logs from a fleet is tens of billions of tokens, and most of it is routine. Narrowing first means the model reads a few dozen actors with full context.
Which model should investigate? Any model that can call tools. The
agent_search stage takes a provider and model_name, and an external agent can call the same retrievers over MCP.Can the investigating model change my infrastructure? Not through this setup. Its only tools are retriever stages, which read from the index. Give it no cloud credentials.