NEWVectors or files. Pick a path.Start →
    Retrieval
    9 min read
    Updated 2026-11-18

    How Do I Search Video or Audio for a Sound, Like Glass Breaking or Applause?

    Embed the audio in short overlapping windows with a text-audio model such as CLAP and search them with a description of the sound. A runnable script found all seven sounds in a 35-second track of labeled clips from plain-text queries, with why thresholds must be per query and how window length sets timing.

    Ethan Steininger
    By , Founder & CEO, Mixpeek
    Audio Search
    CLAP
    Sound Events
    Video Search
    Embeddings

    How do I search video or audio for a sound, like glass breaking or applause?



    Embed the audio track with a text-audio model such as CLAP, in short overlapping windows, and search those window embeddings with a text description of the sound. CLAP puts audio and text in one vector space, so "glass shattering" lands near windows where glass breaks, with no labels and no transcript. Keep each window's start time, and merge neighbouring matching windows into intervals. On a 35-second test track made of seven labeled sound clips, this found all seven sounds at the right seconds from plain-text queries, in about 10 seconds on four CPU threads including loading the model.

    Why does searching the transcript miss sounds?



    Speech-to-text models are trained to write down words. A dog barking, a door knock or a siren produces no text, a bracketed tag at best, or an invented phrase at worst (see why Whisper makes up words during silence). A transcript index answers "where does someone say siren" and cannot answer "where is there a siren". Sounds need an index built from the audio itself.

    Which method finds which kind of sound?



    MethodWhat it matchesQueryGood forLimit
    Audio event classifier (AudioSet models such as PANNs, AST, BEATs)A fixed list of labelsA label nameCounting known event types across a libraryOnly the labels it was trained on: AudioSet labels 527 of its 632 classes
    Text-audio embeddings (CLAP)Audio and text in one vector spaceAny text, or a reference clipOpen-ended search: "a crowd cheering", "metal scraping"Scores are relative; set thresholds per query
    Audio fingerprintingOne exact recordingThe recording itselfFinding a jingle, a sound logo or an ad's audio in longer mediaCannot find "a dog" in general, only that dog clip


    Most libraries need the second method for search, and the third when the question is "where did this exact audio air".

    Reproduce it: search a recording with CLAP



    The test track joins seven 5-second clips from ESC-50, a public set of labeled environmental sounds:

    SecondsSound
    0 to 5clapping
    5 to 10dog
    10 to 15siren
    15 to 20glass breaking
    20 to 25wooden door knock
    25 to 30crying baby
    30 to 35church bells


    The script cuts the recording into 2-second windows every second, embeds each window and each text query with laion/clap-htsat-unfused, and reports where each query matches best.
    """Search a recording for sounds by describing them in text, with CLAP.
    
        pip install torch torchaudio transformers
        python sound_search.py track.wav
    """
    import sys
    
    import torch
    import torchaudio
    from transformers import ClapModel, ClapProcessor
    
    WINDOW_S, HOP_S, SR = 2.0, 1.0, 48_000          # CLAP was trained on 48 kHz audio
    QUERIES = ["a dog barking", "glass shattering", "someone knocking on a door",
               "a baby crying", "people clapping", "church bells ringing", "a siren"]
    
    # The safetensors conversion of the same weights (the main branch has only a pickle).
    model = ClapModel.from_pretrained("laion/clap-htsat-unfused", revision="refs/pr/3").eval()
    proc = ClapProcessor.from_pretrained("laion/clap-htsat-unfused")
    
    wav, sr = torchaudio.load(sys.argv[1])
    wav = torchaudio.functional.resample(wav.mean(0), sr, SR)       # mono, 48 kHz
    
    # 1. Cut the recording into overlapping windows and embed each one.
    win, hop = int(WINDOW_S * SR), int(HOP_S * SR)
    starts = list(range(0, len(wav) - win + 1, hop))
    windows = [wav[s:s + win].numpy() for s in starts]
    with torch.no_grad():
        a = model.get_audio_features(**proc(audios=windows, sampling_rate=SR, return_tensors="pt"))
        t = model.get_text_features(**proc(text=QUERIES, return_tensors="pt", padding=True))
    a = torch.nn.functional.normalize(a, dim=-1)
    t = torch.nn.functional.normalize(t, dim=-1)
    sims = t @ a.T                                   # one row per query, one column per window
    
    # 2. For each query, keep windows close to its best score and merge them into intervals.
    for q, row in zip(QUERIES, sims):
        best = row.max().item()
        hits = [i for i, s in enumerate(row.tolist()) if s >= best - 0.05]
        spans, cur = [], [hits[0], hits[0]]
        for i in hits[1:]:
            if i == cur[1] + 1:
                cur[1] = i
            else:
                spans.append(cur)
                cur = [i, i]
        spans.append(cur)
        found = ", ".join(f"{starts[b] / SR:.0f}-{starts[e] / SR + WINDOW_S:.0f}s" for b, e in spans)
        print(f"{q:28s} best {best:.3f}  at {found}")
    Output, from running it on that track:
    a dog barking                best 0.502  at 6-9s
    glass shattering             best 0.638  at 15-17s
    someone knocking on a door   best 0.322  at 19-21s, 21-23s
    a baby crying                best 0.275  at 25-27s, 28-31s
    people clapping              best 0.479  at 1-6s
    church bells ringing         best 0.497  at 30-35s
    a siren                      best 0.451  at 10-12s, 12-15s
    Every query landed on its own clip: the dog at 6 to 9 s, glass at 15 to 17 s, the siren across 10 to 15 s, the bells across 30 to 35 s. Two things in that output matter when you build this for real:

  1. Scores are not comparable across queries. "a baby crying" topped out at 0.275 and
  2. "glass shattering" at 0.638, and both were correct. A single global threshold would drop the baby. Rank within each query, or calibrate a threshold per query on a few labeled examples.
  3. Windows decide the timing. The knock came back as 19 to 21 s and 21 to 23 s, two spans
  4. because the window starting at 20 s scored just under the cut. A window that straddles two sounds blends them. Shorter windows and smaller hops give sharper timestamps for more embeddings.

    The run took 9.9 seconds of wall time on four CPU threads for 34 windows, model loading included, with a peak of 3.1 GB of memory (torch 2.2.2, transformers 4.52.4). The model's main branch ships only a pickled checkpoint, which transformers refuses to load on torch older than 2.6, so the script loads the safetensors conversion of the same weights.

    How long should each window be?



    Match the window to the length of the sound you look for. A 2-second window suits short events such as a knock, a bark or a glass break, because a long window averages a short sound with whatever surrounds it. Ambience, music and sound logos read better at 5 to 10 seconds. Overlap windows by half (a 1-second hop for 2-second windows) so an event that starts mid-window still fills one window. Each window is one vector, so an hour of audio at a 1-second hop is 3,600 vectors per hour.

    How does Mixpeek search sounds?



    Mixpeek is a multimodal data warehouse: it indexes video, images, audio and documents where they sit in object storage and searches them with retrievers. Its audio fingerprint extractor does what the script does, at scale: it pulls the audio track out of each video or audio file, resamples it to 48 kHz, embeds 5-second CLAP windows every 2.5 seconds (both configurable), and stores one document per window with its start and end time. A retriever then searches those windows with a reference clip, such as a jingle or a sound logo, and returns the files and seconds where it plays. On the rate card it is "Music & sound search" at $0.0033 per minute of video (pricing). By default it covers the first 120 seconds of each file, configurable to 600, so set that limit for long recordings.

    Frequently Asked Questions



    What is CLAP?



    CLAP (Contrastive Language-Audio Pretraining) is a model with an audio encoder and a text encoder trained so that a sound and a description of it get nearby vectors. LAION-CLAP was trained on LAION-Audio-630K, 633,526 audio-text pairs, which lets it match a free-text query such as "glass shattering" to audio it has never seen labeled.

    Can I search audio by describing the sound in words?



    Yes, with a text-audio model such as CLAP: embed the audio in windows, embed the description, and rank windows by cosine similarity. In our test, seven plain-English descriptions each found the right 5-second clip in a 35-second track.

    What is the difference between audio fingerprinting and sound search?



    Fingerprinting finds one exact recording, such as a specific jingle or an ad's audio, even inside longer media. Sound search finds a kind of sound, such as any dog barking, from a description or an example. Fingerprinting answers "where did this audio air", and sound search answers "where is there a sound like this".

    Why does my transcript search not find sounds?



    A transcript contains words. Speech-to-text models output nothing, a bracketed tag or a hallucinated phrase for non-speech audio, so a dog bark or a siren never reaches the text index. Index the audio itself with audio embeddings or an event classifier.

    What similarity threshold should I use for sound search?



    Set it per query. In our test the correct matches scored from 0.275 ("a baby crying") to 0.638 ("glass shattering"), so one global threshold either drops the quiet queries or floods the loud ones. Rank results per query, or label a few examples and pick each query's cut from them.

    How many vectors does sound search create?



    One per window. With 2-second windows and a 1-second hop that is 3,600 vectors per hour of audio; with 5-second windows and a 2.5-second hop, 1,440. At 512 dimensions in float32, each vector is 2 KB.

    Key Takeaways



  5. Transcripts do not contain sounds; search them with embeddings of the audio itself.
  6. CLAP puts audio and text in one space, so a text description finds matching audio windows
  7. with no labels.
  8. On a 35-second track of seven labeled clips, seven text queries each found the right clip,
  9. in 9.9 seconds on four CPU threads with the model load included.
  10. Similarity scores differ by query (0.275 to 0.638 for correct matches here), so thresholds
  11. must be per query.
  12. Window length sets timing precision: about 2 seconds for short events, 5 to 10 for ambience
  13. and music.

    Sources



  14. Wu et al., Large-scale Contrastive Language-Audio Pretraining (LAION-CLAP)
  15. laion/clap-htsat-unfused on Hugging Face
  16. ESC-50 environmental sound dataset
  17. AudioSet
  18. Mixpeek audio fingerprint extractor


  19. Where to go next



    CLAP model page · CLAP 2 large · why Whisper makes up words during silence · best speech-to-text APIs · MVS, vector search on object storage · pricing
    Managed Mixpeek

    Put multimodal search to work

    Connect a bucket and Mixpeek runs the whole multimodal search pipeline for you: extraction, indexing, and search over your own objects. No models to wire up, nothing to host.

    Start with Managed
    MVS · bring your own

    Already have vectors?

    Keep your embeddings on your own cloud and run dense, sparse, and BM25 search directly on object storage. From $25/mo.

    Start with MVS

    Search the sounds in every video you store

    Mixpeek embeds the audio track of each file in timestamped CLAP windows where it sits in your bucket, and a retriever finds the files and seconds where a sound plays.

    Audio fingerprint docsSee pricing