NEWVectors or files. Pick a path.Start →
    Ingestion
    8 min read
    Updated 2026-11-16

    Why Does Whisper Make Up Words During Silence?

    Whisper can write "Thank you for watching" or loop the last sentence over silence, and its own silence check keeps a confident invented line. Why it happens, how often (about 1% of transcriptions in one study), and the settings and a runnable segment filter that stop it.

    Ethan Steininger
    By , Founder & CEO, Mixpeek
    Speech-to-Text
    Whisper
    Transcription
    Audio
    Hallucination

    Why does Whisper make up words during silence?



    Whisper writes the most likely text for each 30-second window of audio, and when a window holds silence, music or noise, the most likely text can be a phrase nobody said. The usual ones read like the closing lines of online video captions, "Thank you for watching" or "Subtitles by the Amara.org community", or they repeat the last real sentence in a loop. Whisper's own silence check skips a window only when two signals agree, so an invented line the model is confident in stays in the transcript. Cut non-speech out before decoding with a voice activity filter, stop feeding each window's text into the next, and drop the segments Whisper itself marks as probable silence.

    How often does it happen?



    Often enough to matter in production. Koenecke and colleagues ran Whisper over recorded speech and found that roughly 1% of transcriptions contained entire hallucinated phrases or sentences with no basis in the audio, and that 38% of those hallucinations included explicit harms such as violence, invented associations or false authority. The rate was higher for speakers with aphasia, whose speech has longer pauses, which points to the pauses as the trigger.

    Two later papers dig into it. Barański and colleagues fed Whisper non-speech sounds and found a small set of phrases that come back again and again, which is why a list of known phrases works as a post-filter. Calm-Whisper masked the decoder's attention heads one at a time and found that 3 of the 20 heads in large-v3 account for over 75% of hallucinations on non-speech audio (the UrbanSound set). Fine-tuning those three heads on non-speech data cut non-speech hallucination by over 80% with under 0.1% word error degradation on LibriSpeech.

    Why Whisper's silence check keeps a made-up line



    The open-source whisper package decides per window whether the audio was silent. This is the logic in transcribe.py, with its defaults:
    should_skip = result.no_speech_prob > no_speech_threshold     # 0.6
    if logprob_threshold is not None and result.avg_logprob > logprob_threshold:   # -1.0
        # don't skip if the logprob is high enough, despite the no_speech_prob
        should_skip = False
    The model's own no-speech probability says "this window is probably silent", and the skip is then cancelled whenever the decoded text has an average log probability above -1.0. A fluent stock phrase is exactly the text a language model is confident in. A segment reading "Thank you for watching!" with a no-speech probability of 0.71 and an average log probability of -0.35 passes this check and lands in your transcript.

    Setting in transcribe()DefaultWhat it does
    no_speech_threshold0.6Skips a window as silent only when no-speech probability is above it and average log probability is below logprob_threshold
    logprob_threshold-1.0A decode below it counts as failed and is retried at a higher temperature
    compression_ratio_threshold2.4Text whose gzip compression ratio is above it counts as repetitive and is retried
    temperature0.0, 0.2, 0.4, 0.6, 0.8, 1.0The retry schedule; when every temperature fails, the last decode is kept anyway
    condition_on_previous_textTrueFeeds the previous window's text in as the prompt for the next window
    hallucination_silence_thresholdNone (off)With word_timestamps=True, skips silent stretches longer than this many seconds around a suspected hallucination


    condition_on_previous_text is how one bad window becomes ten. The package's own docstring says turning it off makes the model "less prone to getting stuck in a failure loop, such as repetition looping or timestamps going out of sync", at the cost of text that can be less consistent across windows.

    How do I stop Whisper from hallucinating?



    Work from the audio outward. Each step catches what the one before it misses.

    1. Remove non-speech before Whisper sees it. A voice activity detector (VAD) marks where someone is talking, and only those stretches get decoded. In faster-whisper, WhisperModel.transcribe() has vad_filter=False by default, so you have to turn it on. It uses Silero VAD with a speech threshold of 0.5, a minimum silence of 2,000 ms and 400 ms of padding on each side of speech. Its BatchedInferencePipeline turns the filter on by default with a 160 ms minimum silence.
    from faster_whisper import WhisperModel
    
    model = WhisperModel("large-v3", device="cuda", compute_type="float16")
    segments, info = model.transcribe(
        "call.wav",
        vad_filter=True,
        vad_parameters={"min_silence_duration_ms": 500},
        condition_on_previous_text=False,
    )
    for s in segments:
        print(f"[{s.start:.1f}-{s.end:.1f}] {s.text}")
    2. Stop the loops and skip the silences around them. With the whisper package, turn off condition_on_previous_text and set hallucination_silence_threshold, which needs word timestamps:
    import whisper
    
    model = whisper.load_model("large-v3")
    result = model.transcribe(
        "call.wav",
        condition_on_previous_text=False,
        word_timestamps=True,
        hallucination_silence_threshold=2.0,
    )
    The check scores the words in each segment: a word shorter than 0.133 seconds, longer than 2 seconds or below 0.15 probability raises the score, and a high-scoring segment that sits between silences has the silence before it skipped.

    3. Filter the segments that come out. Whisper returns no_speech_prob, avg_logprob and compression_ratio on every segment, so you can apply the silence rule it softened. This script uses only the standard library. Save the result of transcribe() with json.dump(result, open("result.json", "w")) and, if you ran a VAD, its speech regions as a list of {"start": s, "end": s}:
    """Drop Whisper segments that are probably not speech.
    
    Usage: python filter_segments.py result.json [speech.json]
    result.json: the dict openai-whisper's transcribe() returns, saved with json.dump.
    speech.json: optional list of {"start": s, "end": s} speech regions from a VAD.
    """
    import json
    import re
    import sys
    
    PHRASES = {
        "thank you", "thank you for watching", "thanks for watching",
        "please subscribe", "like and subscribe", "thank you for listening",
        "subtitles by the amara.org community", "you",
    }
    
    def norm(text):
        return re.sub(r"[^a-z0-9 .']", "", text.lower()).strip(" .")
    
    def overlap(seg, regions):
        total = sum(max(0.0, min(seg["end"], r["end"]) - max(seg["start"], r["start"]))
                    for r in regions)
        return total / max(seg["end"] - seg["start"], 1e-6)
    
    def why_drop(seg, regions):
        if regions is not None and overlap(seg, regions) < 0.5:
            return "outside detected speech"
        if seg["no_speech_prob"] > 0.6:
            return f"no_speech_prob {seg['no_speech_prob']:.2f}"
        if seg["compression_ratio"] > 2.4:
            return f"repetition, compression ratio {seg['compression_ratio']:.2f}"
        if norm(seg["text"]) in PHRASES:
            return "known filler phrase"
        return None
    
    def main():
        result = json.load(open(sys.argv[1]))
        regions = json.load(open(sys.argv[2])) if len(sys.argv) > 2 else None
        for seg in result["segments"]:
            reason = why_drop(seg, regions)
            mark = "DROP" if reason else "keep"
            print(f"{mark} {seg['start']:6.1f}-{seg['end']:6.1f} {seg['text'].strip()!r}"
                  + (f"  ({reason})" if reason else ""))
    
    if __name__ == "__main__":
        main()
    On a sample input we wrote to show each failure (six segments: three real sentences, a stock closing line marked probably silent, a repetition loop and a subtitle credit), it prints:
    keep    0.0-   4.2 'Welcome back, today we look at the Q3 numbers.'
    keep    4.2-   9.8 'Revenue grew eleven percent.'
    DROP   38.0-  41.0 'Thank you for watching!'  (no_speech_prob 0.71)
    DROP   41.0-  52.0 "I'm going to go. I'm going to go. I'm going to go. I'm going to go."  (repetition, compression ratio 3.10)
    DROP   60.0-  63.0 'Subtitles by the Amara.org community'  (known filler phrase)
    keep   70.0-  74.5 'Margins held at thirty-two percent.'
    Whisper's own rule keeps the "Thank you for watching!" segment, because its average log probability of -0.35 is above -1.0. The phrase list only drops a segment that is the phrase alone, since a real speaker can say "thank you". Given VAD speech regions, the script drops anything mostly outside them first, which is the strongest of the four checks.

    How Mixpeek handles it



    Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. When it transcribes audio with Whisper, it checks each transcript before indexing it. A transcript that is only a known filler phrase, is more than 70% one such phrase, repeats a phrase of three or more words more than three times, or has fewer than five distinct words across more than 50 characters is left out of the index with the reason recorded, and the batch still completes. The check reads a whole transcript, so one invented line inside an otherwise real transcript passes it.

    A clean transcript is what makes audio search return the right second of a recording. To choose the transcription model, see the best speech-to-text APIs; for who said each line, see speaker diarization; for the cost of a full library, see how much it costs to transcribe an hour of audio.

    Frequently Asked Questions



    Why does Whisper make up words during silence?



    Whisper writes the most likely text for each 30-second window, and over silence, music or noise the most likely text can be a stock phrase or a repeat of the last sentence. Its skip-as-silent rule needs a no-speech probability above 0.6 and an average log probability below -1.0, so a confident invented line is kept. A study found roughly 1% of Whisper transcriptions contained entire hallucinated phrases.

    Why does Whisper write "Thank you for watching" at the end of my transcript?



    The end of a recording is often silence or music, and "Thank you for watching" is one of a small set of phrases Whisper produces for non-speech audio again and again. Run a voice activity filter so the trailing silence is never decoded, or drop a final segment whose no-speech probability is above 0.6 and whose text is only that phrase.

    Does voice activity detection fix Whisper hallucinations?



    It removes the main trigger, because Whisper never decodes the silent stretches. In faster-whisper set vad_filter=True on WhisperModel.transcribe(), where it is off by default; it uses Silero VAD with a 0.5 speech threshold and a 2,000 ms minimum silence unless you pass vad_parameters. A VAD can still pass music or background talk as speech, so keep a segment filter after it.

    What does hallucination_silence_threshold do in Whisper?



    With word_timestamps=True, it tells the whisper package to skip silent stretches longer than the threshold, in seconds, around a segment whose words look anomalous: very short or very long word durations, or word probabilities below 0.15. It is off (None) by default. Start near 2 seconds and tune it on recordings that have the pauses you care about.

    Should I turn off condition_on_previous_text?



    Turn it off when you see repetition loops. With it on, each window's text is fed in as the prompt for the next window, so one hallucinated line can repeat for minutes. The package's docstring says turning it off makes the model less prone to repetition loops and timestamps going out of sync, with text that can be less consistent across windows.

    Do other speech-to-text models hallucinate too?



    The studies cited here measured Whisper. Any model that generates text from audio can write words that were not spoken, so test whichever API you use on recordings with long pauses, hold music and background noise, and check whether it returns a per-segment confidence or no-speech score you can filter on.

    Sources



  1. Koenecke et al., Careless Whisper: Speech-to-Text Hallucination Harms (arXiv 2402.08021, FAccT 2024)
  2. Barański et al., Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio (arXiv 2501.11378)
  3. Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down (arXiv 2505.12969)
  4. openai/whisper transcribe.py
  5. SYSTRAN/faster-whisper transcribe.py and vad.py
  6. Managed Mixpeek

    Put multimodal search to work

    Connect a bucket and Mixpeek runs the whole multimodal search pipeline for you: extraction, indexing, and search over your own objects. No models to wire up, nothing to host.

    Start with Managed
    MVS · bring your own

    Already have vectors?

    Keep your embeddings on your own cloud and run dense, sparse, and BM25 search directly on object storage. From $25/mo.

    Start with MVS

    Run this on your own data

    Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

    Search your own archiveRead Docs

    Related guides

    Ingestion

    Search Every Conversation You Have with GPT Live

    OpenAI's GPT-Live-1 is a full-duplex voice model that listens and speaks at the same time, priced at $0.05 a minute and reporting an 86.2% task-success score on spoken customer-service tasks, more than double the prior generation. Every session it powers is a conversation someone will need to find later. This is the three-call path to make every recording searchable by what was actually said, and the one capture step a live voice API needs that a batch generator does not.

    Read guide →
    Ingestion

    Search Every Image You Generate with ChatGPT

    OpenAI's Images 2.5 cuts generation latency up to 50% across two models, Flare for speed and Sunburst for precise editing, and ships to ChatGPT, the API and Codex on the same day. This is the three-call path to make every image you generate searchable by what is in it, plus the one step Images 2.5 needs that a URL-based generator does not: getting the bytes out of the chat app and into a bucket.

    Read guide →
    Ingestion

    Search Every World You Generate with Runway

    Runway's GWM Worlds 2 generates an interactive world live: 720p video at 24 fps with 48 kHz audio, steered by text actions and camera motion, for as long as you keep playing. Nothing about a session is repeatable, so the recording is the only record of what happened. This is the three-call path to make every session you capture searchable by what is in it, and the honest line on what a world model emits that search cannot reach.

    Read guide →