NEWVectors or files. Pick a path.Start →
    Back to DiagramsVideo Understanding

    Speaker Diarization: Who Said What, and When

    Transcription tells you what was said. Diarization tells you who said it and when, by clustering voices before it ever aligns them to the transcript.

    Speaker diarization answers who spoke and when, the layer transcription alone cannot provide. The same sixteen seconds of a meeting are shown at three levels: the raw waveform, voice activity detection marking speech regions, and clustered speaker turns, producing a transcript whose every line carries a speaker label and a timestamp. The standard pipeline is four stages: voice activity detection (Silero VAD, a 2MB model running 500x real-time on CPU at about 95% accuracy), speaker embedding extraction (ECAPA-TDNN, 192 or 256 dimensions over 1.5 to 3 second sliding windows, capturing pitch and timbre regardless of words), spectral clustering that reads the speaker count from the eigenvalue gap, and overlap detection that allows multiple labels on one region. Diarization is then joined with ASR word timestamps to attribute each word, and the field scores itself with diarization error rate (false alarm plus missed speech plus speaker confusion over total speech), where under 10% is production quality.
    Transcription tells you what was said. Diarization tells you who said it and when, by clustering voices before it ever aligns them to the transcript.

    Transcription tells you what was said. Diarization tells you who said it, and when. Run them together and a flat wall of text becomes a conversation: Speaker A at 00:14 moving the launch to Q3, Speaker B at 00:18 worried about the market window, Speaker C at 00:22 proposing a soft launch in May. The labels are what make the transcript answerable.

    The standard pipeline is four stages. Voice activity detection finds the speech regions first (Silero VAD is a 2MB model that runs 500x real-time on a CPU at around 95% accuracy on noisy audio). Speaker embedding extraction turns 1.5 to 3 second sliding windows into vectors (ECAPA-TDNN, 192 or 256 dimensions) that capture pitch, timbre, and speaking rate, so the same speaker looks the same regardless of the words. Spectral clustering over a cosine similarity matrix reads the number of speakers from the eigenvalue gap and assigns each window. Overlap detection then allows more than one label where people talk at once. Only after clustering does diarization join with ASR word timestamps (Whisper, Canary-Qwen): each word takes the speaker whose turn its midpoint falls into.

    The field scores itself with diarization error rate, the sum of false alarm, missed speech, and speaker confusion divided by total speech duration. Under 10% is production quality, and the benchmarks fan out by difficulty: about 5% on clean broadcast (VoxConverse), 11% on phone calls (CALLHOME), 18% on far-field 4 person meetings (AMI). One caveat worth stating up front: the labels are not names. Mapping Speaker A to a real identity needs enrolled reference audio, typically 10 to 30 seconds of clean speech per person.

    Where this diagram appears

    Run this on your own data

    Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API.

    Search your own data, free

    Related diagrams

    Object decomposition: you search inside the file, not for it. A one-hour recording.mp4 is split into scenes (visual embeddings), speech (transcript embeddings on silence and speaker turns), faces (who appeared and when), and on-screen text (OCR of slides and captions). A whole file being findable is not the same as usable: the thing you need is a moment inside it, like 23,41 where the renewal number is spoken while the pricing slide is on screen. The same move works on video, PDFs, and audio: one file becomes dozens of small, independently searchable documents.

    You search inside the file, not for it

    Feature extraction turns raw pixels into searchable signals: one video frame becomes a structured row, face, scene, spoken words (transcript), on-screen text (OCR), and a visual embedding. Every question you later ask the video is answered from these extracted signals, not from the raw pixels; if a signal was never extracted, nothing downstream can search it.

    Pixels in, signals out

    Video highlight detection pipeline: a video is decomposed into shots and segments, each segment is scored across visual, audio, and transcript signals, contiguous high-scoring intervals merge into moments, and moments are ranked per video.

    Multimodal Decomposition: One File, Many Signals