Agent Perception

18 min read

Updated 2026-06-09

# Forced Alignment for AI Agents: Word Timestamps, Diarization, and Audio Evidence Search

Learn how forced alignment turns transcripts into timestamped evidence that agents can search, cite, and inspect across audio and video archives.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Agent Perception

Audio Search

Video Search

Forced Alignment

Timestamps

Diarization

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templateFootage Intelligence at Archive ScaleMillions of scenes you can move through instead of search. Footage, ads and editor assets land in three clean collections, and scenes are clustered one partition at a time so every run stays under the sample-size caps.Deploy it](/templates/footage-intelligence)

## Why Timestamps Are a Retrieval Primitive

  
  
Audio and video RAG often starts with automatic speech recognition. That is necessary, but it is not enough.  
  
A transcript without accurate timing is just a long text file. An agent can quote it, but it cannot reliably answer the operational question: where in the source did this happen?  
  
For spoken media, useful evidence has four parts:  
  

Field

What it answers

Text

What was said

Time span

When it was said

Speaker or channel

Who said it, when available

Source handle

Which audio or video object contains it

  
  
The time span is what turns a transcript result into inspectable evidence. It lets the agent deep-link to the moment in a call, meeting, podcast, surveillance clip, lecture, deposition, or broadcast.  
  
This is the difference between:  
  
*   "The customer asked for a refund."
*   "The customer asked for a refund in call\_481 at 00:13:42.180 to 00:13:46.920."
  
  
The second answer can be verified.  
  

## What Forced Alignment Does

  
  
Forced alignment takes two inputs:  
  
1\. An audio signal. 2. A known transcript or token sequence.  
  
It returns timing boundaries for the transcript units. Those units may be phonemes, characters, words, phrases, or subtitle segments.  
  
ASR answers: "What text is in this audio?"  
  
Forced alignment answers: "Given this text, where does each part occur in the audio?"  
  
That distinction matters. A transcription model may produce segment-level timestamps, but those timestamps are often too coarse for retrieval citations. Segment boundaries can drift, especially with long audio, music, overlap, silence, accents, code-switching, or noisy recordings. Forced alignment is a second pass that snaps transcript units to acoustic evidence.  
  

## The Basic Pipeline

  
  
A production alignment pipeline usually looks like this:

audio or video object
  -> voice activity detection
  -> ASR transcript
  -> optional normalization
  -> forced alignment
  -> speaker diarization join
  -> span chunking
  -> embedding and sparse indexing
  -> agent retrieval tool

Each stage changes the retrieval contract.  
  
Voice activity detection removes silence and non-speech so the aligner does not waste work. ASR gives the text. Normalization makes transcript tokens match what the aligner expects. Forced alignment gives boundaries. Diarization attaches speaker turns. Chunking decides what span size should be searchable. Indexing makes the spans retrievable.  
  
The output should not be a plain transcript. It should be a span table:

{
  "source\_id": "call\_481",
  "source\_uri": "s3://support-calls/2026/06/09/call\_481.wav",
  "span\_id": "call\_481:000822180-000826920",
  "text": "I would like a refund because the outage affected our launch",
  "start\_ms": 822180,
  "end\_ms": 826920,
  "speaker": "customer",
  "asr\_model": "Qwen/Qwen3-ASR-1.7B",
  "aligner\_model": "Qwen/Qwen3-ForcedAligner-0.6B",
  "alignment\_confidence": 0.91
}

An agent can search this record, cite it, and request the source clip for inspection.  
  

## Alignment Algorithms

  
  
Forced alignment is an old speech problem with newer model families. The main approaches differ in how they score possible alignments between audio frames and transcript units.  
  

### 1\. HMM and Kaldi-Style Alignment

  
  
Traditional forced aligners, including the Montreal Forced Aligner family, rely on acoustic models, pronunciation dictionaries, and Hidden Markov Models. The transcript is expanded into a likely phoneme sequence. The acoustic model scores which phoneme is likely at each frame. Dynamic programming finds the most likely path through the sequence.  
  
This works well when:  
  
*   You have a pronunciation dictionary.
*   The language is supported.
*   The transcript is clean.
*   You need phoneme-level or word-level linguistic alignment.
  
  
It is less convenient for arbitrary multilingual production media because dictionaries, acoustic models, and text normalization rules become operational dependencies.  
  

### 2\. CTC Forced Alignment

  
  
Connectionist Temporal Classification models output frame-level probabilities over tokens plus a blank symbol. The aligner searches for the path through those frame probabilities that collapses to the target transcript.  
  
The common dynamic programming formulation is similar to Viterbi decoding:  
  
1\. Convert transcript tokens into an expanded sequence with blanks. 2. Score every audio frame against every possible token state. 3. Find the highest probability monotonic path. 4. Collapse blanks and repeated tokens into word or character boundaries.  
  
CTC alignment is useful because it does not need frame-level labels at training time. It is common in wav2vec2-style alignment systems and WhisperX-style pipelines.  
  
The main failure mode is tokenization mismatch. If the transcript text and the aligner vocabulary disagree, the model may force a bad path.  
  

### 3\. Dynamic Time Warping

  
  
Dynamic Time Warping aligns two sequences by stretching or compressing time while preserving order. It is often used when you have two comparable streams, such as model token timing estimates and transcript tokens.  
  
DTW is attractive because it is simple and robust to local speed changes. It is not a speech recognizer by itself. It needs features or token probabilities that already represent the audio and text.  
  
Use DTW when you need to reconcile two imperfect timing sequences, not when you need a full acoustic alignment system from scratch.  
  

### 4\. Non-Autoregressive Timestamp Prediction

  
  
Newer aligners can predict timestamps directly for transcript units. Qwen3-ForcedAligner-0.6B is an example: the model card says it aligns text-speech pairs and returns word or character-level timestamps, with support for timestamp prediction in speech windows up to five minutes.  
  
This is useful for retrieval systems because the output is closer to the data shape agents need:

\[
  {"word": "refund", "start\_ms": 823940, "end\_ms": 824310},
  {"word": "outage", "start\_ms": 824980, "end\_ms": 825430}
\]

The tradeoff is that model-specific runtime and windowing rules matter. You still need validation, confidence handling, and fallbacks.  
  

## Why Word Timestamps Are Not Always the Right Unit

  
  
Word-level timing sounds ideal, but retrieval rarely needs every word as a separate document. Very small chunks create noisy search results and brittle citations.  
  
Use word timestamps as raw material. Build higher-level searchable spans from them.  
  
Good span units include:  
  

Unit

Best for

Risk

Word

Exact subtitle sync, redaction, clip trimming

Too small for semantic search

Phrase

Spoken claim detection, quote retrieval

Needs punctuation or pause heuristics

Speaker turn

Meetings, calls, interviews

Can be too long when speakers monologue

Semantic chunk

RAG and agent answers

Requires chunker that preserves time

Scene or chapter

Podcasts, lectures, long videos

Too coarse for citations

  
  
A strong retrieval system stores multiple units:  
  
*   Word timestamps for playback and exact citation.
*   Phrase spans for quote-level retrieval.
*   Semantic chunks for answer generation.
*   Speaker turns for conversation structure.
  
  
The agent should retrieve chunks, then cite the narrower word or phrase span inside the chunk.  
  

## Joining Diarization and Alignment

  
  
Speaker diarization answers "who spoke when." Forced alignment answers "where the words occur." They should be joined by interval overlap.

aligned word:     refund     822.180s - 822.510s
speaker segment:  customer   819.000s - 828.400s
result:           refund spoken by customer

The join is simple when speakers do not overlap. Overlap makes it harder:  
  
*   Two speakers talk at the same time.
*   The diarizer changes speaker too late.
*   ASR merges speech from two speakers into one text span.
*   A speaker label covers silence or background audio.
  
  
Do not hide this uncertainty. Store overlap ratios and confidence fields.

{
  "word": "refund",
  "start\_ms": 822180,
  "end\_ms": 822510,
  "speaker": "customer",
  "speaker\_overlap": 0.96,
  "speaker\_confidence": 0.88
}

Agents should receive this uncertainty when the answer depends on attribution.  
  

## Indexing Aligned Speech

  
  
Do not embed timestamps inside the text string. Text like "\[00:13:42\] I would like a refund" adds token noise to the embedding. Store timestamps as metadata.  
  
Use two parallel representations:

{
  "embedding\_text": "I would like a refund because the outage affected our launch",
  "metadata": {
    "source\_id": "call\_481",
    "start\_ms": 822180,
    "end\_ms": 826920,
    "speaker": "customer"
  }
}

For retrieval, combine:  
  
1\. Dense embedding search for semantic meaning. 2. BM25 or sparse search for exact terms, names, and product codes. 3. Metadata filters for speaker, date, source type, language, or customer. 4. Reranking for the top candidates.  
  
This matters for agent search because spoken evidence contains both semantic and exact-match questions:  
  
*   Semantic: "Where did the customer sound frustrated about reliability?"
*   Exact: "Where did someone say SKU-447B?"
*   Filtered: "Find refund mentions by the customer, not the support agent."
*   Temporal: "What did the engineer say after the outage was mentioned?"
  
  

## Query Planning for Agents

  
  
Expose aligned speech as a bounded tool. The agent should not receive the entire transcript unless it asks for expansion.  
  
A useful tool schema:

{
  "name": "search\_spoken\_evidence",
  "description": "Search timestamped audio and video transcripts. Returns cited spans with source URI, speaker, start\_ms, end\_ms, and nearby context.",
  "input\_schema": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "speaker": {"type": "string"},
      "source\_type": {"type": "string", "enum": \["audio", "video", "any"\]},
      "date\_range": {"type": "string"},
      "top\_k": {"type": "integer", "minimum": 1, "maximum": 20}
    },
    "required": \["query"\]
  }
}

The result should include inspectable handles:

{
  "results": \[
    {
      "text": "I would like a refund because the outage affected our launch",
      "speaker": "customer",
      "source\_uri": "s3://support-calls/2026/06/09/call\_481.wav",
      "start\_ms": 822180,
      "end\_ms": 826920,
      "score": 0.84,
      "clip\_uri": "mixpeek://clips/call\_481/822180-826920"
    }
  \]
}

The agent can answer with a citation and ask for a follow-up clip only if needed.  
  

## Evaluation

  
  
Evaluate alignment separately from retrieval. A retrieval system can find the right transcript chunk while still citing the wrong second.  
  
Measure at least five things:  
  

Metric

What it catches

Boundary error

Average absolute error for word or phrase start/end times

Coverage

Percentage of transcript tokens that received usable timestamps

Speaker attribution accuracy

Whether words are assigned to the correct speaker

Citation hit rate

Whether the cited span actually contains the answer

Expansion quality

Whether adjacent context helps without pulling in unrelated speech

  
  
For agents, add task-level evals:  
  
1\. Ask the agent a question that requires spoken evidence. 2. Require an answer with source ID and timestamp. 3. Check whether the cited time span contains the supporting speech. 4. Penalize answers that cite only the file without a span. 5. Penalize answers that cite the right transcript but wrong speaker.  
  
The important metric is not only word error rate. It is whether a human can click the citation and verify the answer.  
  

## Failure Modes

  
  
**Transcript mismatch.** If ASR hallucinated, omitted, or normalized text too aggressively, the aligner may force timestamps onto words that were never spoken.  
  
**Punctuation drift.** Punctuation is not acoustic. Use it to build spans, but do not expect punctuation boundaries to align cleanly with audio.  
  
**Overlapping speech.** One transcript line may contain speech from two speakers. Store uncertainty and consider overlap-aware diarization.  
  
**Music and background audio.** Song lyrics, crowd noise, and background speech can confuse ASR and alignment. Track non-speech intervals.  
  
**Long-window drift.** Alignment windows that are too long can drift. Windows that are too short lose context. Use voice activity and sentence boundaries to choose windows.  
  
**Embedding pollution.** Inline timestamps, speaker labels, and JSON blobs can hurt semantic embeddings. Keep clean text for embedding and structured metadata for filters.  
  

## Mixpeek Example

  
  
In Mixpeek, model this as a speech evidence pipeline rather than a transcript upload.

from mixpeek import Mixpeek

mx = Mixpeek(api\_key="YOUR\_API\_KEY", namespace="support")

bucket = mx.buckets.create(
    bucket\_name="support-calls",
    bucket\_schema={"properties": {"recording": {"type": "audio"}}},
)

# Cut each call at pauses so a segment holds whole utterances, then transcribe
# and embed every segment. Each one keeps start\_time and end\_time, so the
# managed extractor aligns speech to segment-level spans. Word-level
# timestamps and speaker labels come from a forced aligner and a diarization
# model that you run yourself.
collection = mx.collections.create(
    collection\_name="support\_call\_speech",
    source={"type": "bucket", "bucket\_ids": \[bucket\["bucket\_id"\]\]},
    feature\_extractor={
        "feature\_extractor\_name": "multimodal\_extractor",
        "version": "v1",
        "parameters": {
            "split\_method": "silence",
            "silence\_db\_threshold": -40,
            "max\_segment\_duration": 30,
            "run\_transcription": True,
            "run\_transcription\_embedding": True,
        },
    },
)

mx.buckets.upload(
    bucket\["bucket\_id"\],
    blobs=\[{"property": "recording", "type": "audio", "data": "s3://support-calls/2026/06/call-0142.wav"}\],
)
mx.collections.trigger(collection\["collection\_id"\])

Then expose a retrieval tool:

results = mx.retrievers.execute(
    "your-retriever-id",
    inputs={"query": "customer requested refund after outage"},
)

If you already generated aligned spans outside Mixpeek, store them in MVS with clean text vectors and timestamp metadata:

mx.namespaces.documents.upsert(
    namespace\_id="YOUR\_STANDALONE\_NAMESPACE\_ID",
    documents=\[
        {
            "document\_id": "call\_481:822180:826920:bge\_m3",
            "vectors": {"bge\_m3": span\_embedding},
            "payload": {
                "source\_uri": "s3://support-calls/2026/06/09/call\_481.wav",
                "text": "I would like a refund because the outage affected our launch",
                "speaker": "customer",
                "start\_ms": 822180,
                "end\_ms": 826920,
                "asr\_model": "Qwen/Qwen3-ASR-1.7B",
                "aligner\_model": "Qwen/Qwen3-ForcedAligner-0.6B",
            },
        }
    \],
)

## Design Checklist

  
  
*   Store clean transcript text separately from timestamp metadata.
*   Preserve source URI, source ID, language, model IDs, extractor versions, and timing units.
*   Keep word-level timestamps even if retrieval chunks are phrase or semantic spans.
*   Join diarization and alignment by interval overlap, not by transcript line number.
*   Store confidence and overlap fields when speaker attribution is uncertain.
*   Use dense and sparse retrieval together for spoken evidence.
*   Return source URI, start time, end time, and nearby context to agents.
*   Evaluate citation hit rate, not only ASR word error rate.
*   Backfill alignment when ASR models, diarizers, or normalization rules change.
  
  

## Key Takeaways

  
  
1\. ASR makes audio searchable. Forced alignment makes it citeable.  
  
2\. Word timestamps are raw evidence, not always the retrieval chunk.  
  
3\. Keep timestamps as metadata. Do not pollute embedding text with timing strings.  
  
4\. Diarization and alignment need interval joins with uncertainty, especially during overlap.  
  
5\. Agent tools should return compact cited spans with source handles and expansion paths.  
  
6\. The eval that matters is whether a human can click the cited span and verify the answer.  
  

## Further Reading

  
  
*   [Audio Feature Extraction: How AI Agents Learn to Hear](/guides/audio-feature-extraction-how-agents-learn-to-hear)
*   [Speaker Diarization: How AI Agents Know Who Said What in Audio and Video](/guides/speaker-diarization-who-said-what)
*   [Audio-Visual Retrieval for AI Agents](/guides/audio-visual-retrieval-for-ai-agents)
*   [Why does my podcast search find the right episode but not the moment?](/guides/why-does-my-podcast-search-find-the-right-episode-but-not-the-moment) for the pain-shaped version of this problem
*   [Agent Perception Evals](/guides/agent-perception-evals)
*   [Qwen3-ForcedAligner-0.6B on Hugging Face](https://huggingface.co/Qwen/Qwen3-ForcedAligner-0.6B)
*   [WhisperX on GitHub](https://github.com/m-bain/whisperX)
*   [Montreal Forced Aligner documentation](https://montreal-forced-aligner.readthedocs.io/en/v3.2.3/)

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Agent Perception

### Audio Fingerprinting: How Agents Recognize a Specific Recording in Noise

A first-principles guide to audio fingerprinting -- the algorithm behind Shazam-style recognition that identifies an exact recording even when it is noisy, pitch-shifted, or buried in other sound. Covers spectrogram peak picking, the constellation map, combinatorial landmark hashing, inverted-index voting with time-offset alignment, and how identity-level audio search differs from semantic similarity for AI agents.

Read guide →](/guides/audio-fingerprinting-constellation-landmark-hashing)[

Agent Perception

### Object Decomposition and Layered Indexing for AI Agent Perception

A practical architecture guide for turning video, audio, images, and documents into searchable evidence that agents can inspect, cite, filter, and reason over.

Read guide →](/guides/object-decomposition-layered-indexing)[

Agent Perception

### Streaming Video Understanding: How Agents Watch an Unbounded Live Feed in Real Time

A first-principles guide to online video understanding -- how an agent perceives a live, unbounded stream it cannot store or re-watch. Covers the causal constraint, ring buffers and fixed frame budgets, token merging and KV-cache pruning, hierarchical short-term and long-term memory, entity banks for cross-time identity, event-triggered indexing, and how a streaming front end feeds a searchable retrieval index so the agent can answer questions about something that happened minutes or hours ago.

Read guide →](/guides/streaming-video-understanding-online-memory)