Retrieval

9 min read

Updated 2026-11-18

# How Do I Search Video or Audio for a Sound, Like Glass Breaking or Applause?

Embed the audio in short overlapping windows with a text-audio model such as CLAP and search them with a description of the sound. A runnable script found all seven sounds in a 35-second track of labeled clips from plain-text queries, with why thresholds must be per query and how window length sets timing.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Audio Search

CLAP

Sound Events

Video Search

Embeddings

[Search the sounds in every video you store. Try it as you read.Audio fingerprint docs →](https://mixpeek.com/docs/processing/extractors/audio-fingerprint)

[One-click templateFootage Intelligence at Archive ScaleMillions of scenes you can move through instead of search. Footage, ads and editor assets land in three clean collections, and scenes are clustered one partition at a time so every run stays under the sample-size caps.Deploy it](/templates/footage-intelligence)

## How do I search video or audio for a sound, like glass breaking or applause?

  
  
Embed the audio track with a text-audio model such as CLAP, in short overlapping windows, and search those window embeddings with a text description of the sound. CLAP puts audio and text in one vector space, so "glass shattering" lands near windows where glass breaks, with no labels and no transcript. Keep each window's start time, and merge neighbouring matching windows into intervals. On a 35-second test track made of seven labeled sound clips, this found all seven sounds at the right seconds from plain-text queries, in about 10 seconds on four CPU threads including loading the model.  
  

## Why does searching the transcript miss sounds?

  
  
Speech-to-text models are trained to write down words. A dog barking, a door knock or a siren produces no text, a bracketed tag at best, or an invented phrase at worst (see [why Whisper makes up words during silence](/guides/why-does-whisper-make-up-words-during-silence)). A transcript index answers "where does someone say siren" and cannot answer "where is there a siren". Sounds need an index built from the audio itself.  
  

## Which method finds which kind of sound?

  
  

Method

What it matches

Query

Good for

Limit

Audio event classifier (AudioSet models such as PANNs, AST, BEATs)

A fixed list of labels

A label name

Counting known event types across a library

Only the labels it was trained on: AudioSet labels 527 of its 632 classes

Text-audio embeddings ([CLAP](https://arxiv.org/abs/2211.06687))

Audio and text in one vector space

Any text, or a reference clip

Open-ended search: "a crowd cheering", "metal scraping"

Scores are relative; set thresholds per query

Audio fingerprinting

One exact recording

The recording itself

Finding a jingle, a sound logo or an ad's audio in longer media

Cannot find "a dog" in general, only that dog clip

  
  
Most libraries need the second method for search, and the third when the question is "where did this exact audio air".  
  

## Reproduce it: search a recording with CLAP

  
  
The test track joins seven 5-second clips from [ESC-50](https://github.com/karolpiczak/ESC-50), a public set of labeled environmental sounds:  
  

Seconds

Sound

0 to 5

clapping

5 to 10

dog

10 to 15

siren

15 to 20

glass breaking

20 to 25

wooden door knock

25 to 30

crying baby

30 to 35

church bells

  
  
The script cuts the recording into 2-second windows every second, embeds each window and each text query with [laion/clap-htsat-unfused](/model/laion/clap-htsat-unfused), and reports where each query matches best.

"""Search a recording for sounds by describing them in text, with CLAP.

    pip install torch torchaudio transformers
    python sound\_search.py track.wav
"""
import sys

import torch
import torchaudio
from transformers import ClapModel, ClapProcessor

WINDOW\_S, HOP\_S, SR = 2.0, 1.0, 48\_000          # CLAP was trained on 48 kHz audio
QUERIES = \["a dog barking", "glass shattering", "someone knocking on a door",
           "a baby crying", "people clapping", "church bells ringing", "a siren"\]

# The safetensors conversion of the same weights (the main branch has only a pickle).
model = ClapModel.from\_pretrained("laion/clap-htsat-unfused", revision="refs/pr/3").eval()
proc = ClapProcessor.from\_pretrained("laion/clap-htsat-unfused")

wav, sr = torchaudio.load(sys.argv\[1\])
wav = torchaudio.functional.resample(wav.mean(0), sr, SR)       # mono, 48 kHz

# 1. Cut the recording into overlapping windows and embed each one.
win, hop = int(WINDOW\_S \* SR), int(HOP\_S \* SR)
starts = list(range(0, len(wav) - win + 1, hop))
windows = \[wav\[s:s + win\].numpy() for s in starts\]
with torch.no\_grad():
    a = model.get\_audio\_features(\*\*proc(audios=windows, sampling\_rate=SR, return\_tensors="pt"))
    t = model.get\_text\_features(\*\*proc(text=QUERIES, return\_tensors="pt", padding=True))
a = torch.nn.functional.normalize(a, dim=-1)
t = torch.nn.functional.normalize(t, dim=-1)
sims = t @ a.T                                   # one row per query, one column per window

# 2. For each query, keep windows close to its best score and merge them into intervals.
for q, row in zip(QUERIES, sims):
    best = row.max().item()
    hits = \[i for i, s in enumerate(row.tolist()) if s >= best - 0.05\]
    spans, cur = \[\], \[hits\[0\], hits\[0\]\]
    for i in hits\[1:\]:
        if i == cur\[1\] + 1:
            cur\[1\] = i
        else:
            spans.append(cur)
            cur = \[i, i\]
    spans.append(cur)
    found = ", ".join(f"{starts\[b\] / SR:.0f}-{starts\[e\] / SR + WINDOW\_S:.0f}s" for b, e in spans)
    print(f"{q:28s} best {best:.3f}  at {found}")

Output, from running it on that track:

a dog barking                best 0.502  at 6-9s
glass shattering             best 0.638  at 15-17s
someone knocking on a door   best 0.322  at 19-21s, 21-23s
a baby crying                best 0.275  at 25-27s, 28-31s
people clapping              best 0.479  at 1-6s
church bells ringing         best 0.497  at 30-35s
a siren                      best 0.451  at 10-12s, 12-15s

Every query landed on its own clip: the dog at 6 to 9 s, glass at 15 to 17 s, the siren across 10 to 15 s, the bells across 30 to 35 s. Two things in that output matter when you build this for real:  
  
*   **Scores are not comparable across queries.** "a baby crying" topped out at 0.275 and
"glass shattering" at 0.638, and both were correct. A single global threshold would drop the baby. Rank within each query, or calibrate a threshold per query on a few labeled examples.*   **Windows decide the timing.** The knock came back as 19 to 21 s and 21 to 23 s, two spans
because the window starting at 20 s scored just under the cut. A window that straddles two sounds blends them. Shorter windows and smaller hops give sharper timestamps for more embeddings.  
  
The run took 9.9 seconds of wall time on four CPU threads for 34 windows, model loading included, with a peak of 3.1 GB of memory (torch 2.2.2, transformers 4.52.4). The model's main branch ships only a pickled checkpoint, which transformers refuses to load on torch older than 2.6, so the script loads the safetensors conversion of the same weights.  
  

## How long should each window be?

  
  
Match the window to the length of the sound you look for. A 2-second window suits short events such as a knock, a bark or a glass break, because a long window averages a short sound with whatever surrounds it. Ambience, music and sound logos read better at 5 to 10 seconds. Overlap windows by half (a 1-second hop for 2-second windows) so an event that starts mid-window still fills one window. Each window is one vector, so an hour of audio at a 1-second hop is 3,600 vectors per hour.  
  

## How does Mixpeek search sounds?

  
  
Mixpeek is a multimodal data warehouse: it indexes video, images, audio and documents where they sit in object storage and searches them with retrievers. Its [audio fingerprint extractor](https://mixpeek.com/docs/processing/extractors/audio-fingerprint) does what the script does, at scale: it pulls the audio track out of each video or audio file, resamples it to 48 kHz, embeds 5-second CLAP windows every 2.5 seconds (both configurable), and stores one document per window with its start and end time. A retriever then searches those windows with a reference clip, such as a jingle or a sound logo, and returns the files and seconds where it plays. On the rate card it is "Music & sound search" at $0.0033 per minute of video ([pricing](/pricing)). By default it covers the first 120 seconds of each file, configurable to 600, so set that limit for long recordings.  
  

## Frequently Asked Questions

  
  

### What is CLAP?

  
  
CLAP (Contrastive Language-Audio Pretraining) is a model with an audio encoder and a text encoder trained so that a sound and a description of it get nearby vectors. LAION-CLAP was trained on LAION-Audio-630K, 633,526 audio-text pairs, which lets it match a free-text query such as "glass shattering" to audio it has never seen labeled.  
  

### Can I search audio by describing the sound in words?

  
  
Yes, with a text-audio model such as CLAP: embed the audio in windows, embed the description, and rank windows by cosine similarity. In our test, seven plain-English descriptions each found the right 5-second clip in a 35-second track.  
  

### What is the difference between audio fingerprinting and sound search?

  
  
Fingerprinting finds one exact recording, such as a specific jingle or an ad's audio, even inside longer media. Sound search finds a kind of sound, such as any dog barking, from a description or an example. Fingerprinting answers "where did this audio air", and sound search answers "where is there a sound like this".  
  

### Why does my transcript search not find sounds?

  
  
A transcript contains words. Speech-to-text models output nothing, a bracketed tag or a hallucinated phrase for non-speech audio, so a dog bark or a siren never reaches the text index. Index the audio itself with audio embeddings or an event classifier.  
  

### What similarity threshold should I use for sound search?

  
  
Set it per query. In our test the correct matches scored from 0.275 ("a baby crying") to 0.638 ("glass shattering"), so one global threshold either drops the quiet queries or floods the loud ones. Rank results per query, or label a few examples and pick each query's cut from them.  
  

### How many vectors does sound search create?

  
  
One per window. With 2-second windows and a 1-second hop that is 3,600 vectors per hour of audio; with 5-second windows and a 2.5-second hop, 1,440. At 512 dimensions in float32, each vector is 2 KB.  
  

## Key Takeaways

  
  
*   Transcripts do not contain sounds; search them with embeddings of the audio itself.
*   CLAP puts audio and text in one space, so a text description finds matching audio windows
with no labels.*   On a 35-second track of seven labeled clips, seven text queries each found the right clip,
in 9.9 seconds on four CPU threads with the model load included.*   Similarity scores differ by query (0.275 to 0.638 for correct matches here), so thresholds
must be per query.*   Window length sets timing precision: about 2 seconds for short events, 5 to 10 for ambience
and music.  
  

## Sources

  
  
*   [Wu et al., Large-scale Contrastive Language-Audio Pretraining (LAION-CLAP)](https://arxiv.org/abs/2211.06687)
*   [laion/clap-htsat-unfused on Hugging Face](https://huggingface.co/laion/clap-htsat-unfused)
*   [ESC-50 environmental sound dataset](https://github.com/karolpiczak/ESC-50)
*   [AudioSet](https://research.google.com/audioset/)
*   [Mixpeek audio fingerprint extractor](https://mixpeek.com/docs/processing/extractors/audio-fingerprint)
  
  

## Where to go next

  
  
[CLAP model page](/model/laion/clap-htsat-unfused) · [CLAP 2 large](/model/laion/clap2-large-audioxl-s1B) · [why Whisper makes up words during silence](/guides/why-does-whisper-make-up-words-during-silence) · [best speech-to-text APIs](/curated-lists/best-speech-to-text-apis) · [MVS, vector search on object storage](/mvs) · [pricing](/pricing)

## Search the sounds in every video you store

Mixpeek embeds the audio track of each file in timestamped CLAP windows where it sits in your bucket, and a retriever finds the files and seconds where a sound plays.

[Audio fingerprint docs](https://mixpeek.com/docs/processing/extractors/audio-fingerprint)[See pricing](/pricing)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Retrieval

### Why Does My Search Only Work in One Language?

Search over your media library works in English, then a user types the same question in Japanese and gets nothing, or gets confident nonsense. Three separate causes produce that. Each leaves a different fingerprint in the results, and a better embedding model fixes only one of them.

Read guide →](/guides/why-does-my-search-only-work-in-one-language)[

Retrieval

### Why Does My Podcast Search Find the Right Episode but Not the Moment?

Your search returns the correct episode and you still have to scrub forty minutes to find where the guest actually said it. That is almost never a transcription problem. It is what the retrieval unit is, and this is how to tell whether you are searching episodes or moments.

Read guide →](/guides/why-does-my-podcast-search-find-the-right-episode-but-not-the-moment)[

Retrieval

### How Do You Search a 3D World You Generated?

World models hand back a scene, not a file you can watch, and the export API gives you geometry rather than pixels. This is what World Labs' Marble API actually exports, why none of it is searchable by content, and the one artifact that is.

Read guide →](/guides/how-do-you-search-a-3d-world-you-generated)