How do I search video or audio for a sound, like glass breaking or applause?
Embed the audio track with a text-audio model such as CLAP, in short overlapping windows, and search those window embeddings with a text description of the sound. CLAP puts audio and text in one vector space, so "glass shattering" lands near windows where glass breaks, with no labels and no transcript. Keep each window's start time, and merge neighbouring matching windows into intervals. On a 35-second test track made of seven labeled sound clips, this found all seven sounds at the right seconds from plain-text queries, in about 10 seconds on four CPU threads including loading the model.
Why does searching the transcript miss sounds?
Speech-to-text models are trained to write down words. A dog barking, a door knock or a siren produces no text, a bracketed tag at best, or an invented phrase at worst (see why Whisper makes up words during silence). A transcript index answers "where does someone say siren" and cannot answer "where is there a siren". Sounds need an index built from the audio itself.
Which method finds which kind of sound?
| Method | What it matches | Query | Good for | Limit |
|---|---|---|---|---|
| Audio event classifier (AudioSet models such as PANNs, AST, BEATs) | A fixed list of labels | A label name | Counting known event types across a library | Only the labels it was trained on: AudioSet labels 527 of its 632 classes |
| Text-audio embeddings (CLAP) | Audio and text in one vector space | Any text, or a reference clip | Open-ended search: "a crowd cheering", "metal scraping" | Scores are relative; set thresholds per query |
| Audio fingerprinting | One exact recording | The recording itself | Finding a jingle, a sound logo or an ad's audio in longer media | Cannot find "a dog" in general, only that dog clip |
Most libraries need the second method for search, and the third when the question is "where did this exact audio air".
Reproduce it: search a recording with CLAP
The test track joins seven 5-second clips from ESC-50, a public set of labeled environmental sounds:
| Seconds | Sound |
|---|---|
| 0 to 5 | clapping |
| 5 to 10 | dog |
| 10 to 15 | siren |
| 15 to 20 | glass breaking |
| 20 to 25 | wooden door knock |
| 25 to 30 | crying baby |
| 30 to 35 | church bells |
The script cuts the recording into 2-second windows every second, embeds each window and each text query with laion/clap-htsat-unfused, and reports where each query matches best.
"""Search a recording for sounds by describing them in text, with CLAP.
pip install torch torchaudio transformers
python sound_search.py track.wav
"""
import sys
import torch
import torchaudio
from transformers import ClapModel, ClapProcessor
WINDOW_S, HOP_S, SR = 2.0, 1.0, 48_000 # CLAP was trained on 48 kHz audio
QUERIES = ["a dog barking", "glass shattering", "someone knocking on a door",
"a baby crying", "people clapping", "church bells ringing", "a siren"]
# The safetensors conversion of the same weights (the main branch has only a pickle).
model = ClapModel.from_pretrained("laion/clap-htsat-unfused", revision="refs/pr/3").eval()
proc = ClapProcessor.from_pretrained("laion/clap-htsat-unfused")
wav, sr = torchaudio.load(sys.argv[1])
wav = torchaudio.functional.resample(wav.mean(0), sr, SR) # mono, 48 kHz
# 1. Cut the recording into overlapping windows and embed each one.
win, hop = int(WINDOW_S * SR), int(HOP_S * SR)
starts = list(range(0, len(wav) - win + 1, hop))
windows = [wav[s:s + win].numpy() for s in starts]
with torch.no_grad():
a = model.get_audio_features(**proc(audios=windows, sampling_rate=SR, return_tensors="pt"))
t = model.get_text_features(**proc(text=QUERIES, return_tensors="pt", padding=True))
a = torch.nn.functional.normalize(a, dim=-1)
t = torch.nn.functional.normalize(t, dim=-1)
sims = t @ a.T # one row per query, one column per window
# 2. For each query, keep windows close to its best score and merge them into intervals.
for q, row in zip(QUERIES, sims):
best = row.max().item()
hits = [i for i, s in enumerate(row.tolist()) if s >= best - 0.05]
spans, cur = [], [hits[0], hits[0]]
for i in hits[1:]:
if i == cur[1] + 1:
cur[1] = i
else:
spans.append(cur)
cur = [i, i]
spans.append(cur)
found = ", ".join(f"{starts[b] / SR:.0f}-{starts[e] / SR + WINDOW_S:.0f}s" for b, e in spans)
print(f"{q:28s} best {best:.3f} at {found}")a dog barking best 0.502 at 6-9s glass shattering best 0.638 at 15-17s someone knocking on a door best 0.322 at 19-21s, 21-23s a baby crying best 0.275 at 25-27s, 28-31s people clapping best 0.479 at 1-6s church bells ringing best 0.497 at 30-35s a siren best 0.451 at 10-12s, 12-15s
The run took 9.9 seconds of wall time on four CPU threads for 34 windows, model loading included, with a peak of 3.1 GB of memory (torch 2.2.2, transformers 4.52.4). The model's main branch ships only a pickled checkpoint, which transformers refuses to load on torch older than 2.6, so the script loads the safetensors conversion of the same weights.
How long should each window be?
Match the window to the length of the sound you look for. A 2-second window suits short events such as a knock, a bark or a glass break, because a long window averages a short sound with whatever surrounds it. Ambience, music and sound logos read better at 5 to 10 seconds. Overlap windows by half (a 1-second hop for 2-second windows) so an event that starts mid-window still fills one window. Each window is one vector, so an hour of audio at a 1-second hop is 3,600 vectors per hour.
How does Mixpeek search sounds?
Mixpeek is a multimodal data warehouse: it indexes video, images, audio and documents where they sit in object storage and searches them with retrievers. Its audio fingerprint extractor does what the script does, at scale: it pulls the audio track out of each video or audio file, resamples it to 48 kHz, embeds 5-second CLAP windows every 2.5 seconds (both configurable), and stores one document per window with its start and end time. A retriever then searches those windows with a reference clip, such as a jingle or a sound logo, and returns the files and seconds where it plays. On the rate card it is "Music & sound search" at $0.0033 per minute of video (pricing). By default it covers the first 120 seconds of each file, configurable to 600, so set that limit for long recordings.
Frequently Asked Questions
What is CLAP?
CLAP (Contrastive Language-Audio Pretraining) is a model with an audio encoder and a text encoder trained so that a sound and a description of it get nearby vectors. LAION-CLAP was trained on LAION-Audio-630K, 633,526 audio-text pairs, which lets it match a free-text query such as "glass shattering" to audio it has never seen labeled.
Can I search audio by describing the sound in words?
Yes, with a text-audio model such as CLAP: embed the audio in windows, embed the description, and rank windows by cosine similarity. In our test, seven plain-English descriptions each found the right 5-second clip in a 35-second track.
What is the difference between audio fingerprinting and sound search?
Fingerprinting finds one exact recording, such as a specific jingle or an ad's audio, even inside longer media. Sound search finds a kind of sound, such as any dog barking, from a description or an example. Fingerprinting answers "where did this audio air", and sound search answers "where is there a sound like this".
Why does my transcript search not find sounds?
A transcript contains words. Speech-to-text models output nothing, a bracketed tag or a hallucinated phrase for non-speech audio, so a dog bark or a siren never reaches the text index. Index the audio itself with audio embeddings or an event classifier.
What similarity threshold should I use for sound search?
Set it per query. In our test the correct matches scored from 0.275 ("a baby crying") to 0.638 ("glass shattering"), so one global threshold either drops the quiet queries or floods the loud ones. Rank results per query, or label a few examples and pick each query's cut from them.
How many vectors does sound search create?
One per window. With 2-second windows and a 1-second hop that is 3,600 vectors per hour of audio; with 5-second windows and a 2.5-second hop, 1,440. At 512 dimensions in float32, each vector is 2 KB.
Key Takeaways
Sources
Where to go next
CLAP model page · CLAP 2 large · why Whisper makes up words during silence · best speech-to-text APIs · MVS, vector search on object storage · pricing