Why does Whisper make up words during silence?
Whisper writes the most likely text for each 30-second window of audio, and when a window holds silence, music or noise, the most likely text can be a phrase nobody said. The usual ones read like the closing lines of online video captions, "Thank you for watching" or "Subtitles by the Amara.org community", or they repeat the last real sentence in a loop. Whisper's own silence check skips a window only when two signals agree, so an invented line the model is confident in stays in the transcript. Cut non-speech out before decoding with a voice activity filter, stop feeding each window's text into the next, and drop the segments Whisper itself marks as probable silence.
How often does it happen?
Often enough to matter in production. Koenecke and colleagues ran Whisper over recorded speech and found that roughly 1% of transcriptions contained entire hallucinated phrases or sentences with no basis in the audio, and that 38% of those hallucinations included explicit harms such as violence, invented associations or false authority. The rate was higher for speakers with aphasia, whose speech has longer pauses, which points to the pauses as the trigger.
Two later papers dig into it. Barański and colleagues fed Whisper non-speech sounds and found a small set of phrases that come back again and again, which is why a list of known phrases works as a post-filter. Calm-Whisper masked the decoder's attention heads one at a time and found that 3 of the 20 heads in large-v3 account for over 75% of hallucinations on non-speech audio (the UrbanSound set). Fine-tuning those three heads on non-speech data cut non-speech hallucination by over 80% with under 0.1% word error degradation on LibriSpeech.
Why Whisper's silence check keeps a made-up line
The open-source
whisper package decides per window whether the audio was silent. This is the logic in transcribe.py, with its defaults:should_skip = result.no_speech_prob > no_speech_threshold # 0.6
if logprob_threshold is not None and result.avg_logprob > logprob_threshold: # -1.0
# don't skip if the logprob is high enough, despite the no_speech_prob
should_skip = FalseSetting in transcribe() | Default | What it does |
|---|---|---|
no_speech_threshold | 0.6 | Skips a window as silent only when no-speech probability is above it and average log probability is below logprob_threshold |
logprob_threshold | -1.0 | A decode below it counts as failed and is retried at a higher temperature |
compression_ratio_threshold | 2.4 | Text whose gzip compression ratio is above it counts as repetitive and is retried |
temperature | 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 | The retry schedule; when every temperature fails, the last decode is kept anyway |
condition_on_previous_text | True | Feeds the previous window's text in as the prompt for the next window |
hallucination_silence_threshold | None (off) | With word_timestamps=True, skips silent stretches longer than this many seconds around a suspected hallucination |
condition_on_previous_text is how one bad window becomes ten. The package's own docstring says turning it off makes the model "less prone to getting stuck in a failure loop, such as repetition looping or timestamps going out of sync", at the cost of text that can be less consistent across windows.How do I stop Whisper from hallucinating?
Work from the audio outward. Each step catches what the one before it misses.
1. Remove non-speech before Whisper sees it. A voice activity detector (VAD) marks where someone is talking, and only those stretches get decoded. In faster-whisper,
WhisperModel.transcribe() has vad_filter=False by default, so you have to turn it on. It uses Silero VAD with a speech threshold of 0.5, a minimum silence of 2,000 ms and 400 ms of padding on each side of speech. Its BatchedInferencePipeline turns the filter on by default with a 160 ms minimum silence.from faster_whisper import WhisperModel
model = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, info = model.transcribe(
"call.wav",
vad_filter=True,
vad_parameters={"min_silence_duration_ms": 500},
condition_on_previous_text=False,
)
for s in segments:
print(f"[{s.start:.1f}-{s.end:.1f}] {s.text}")whisper package, turn off condition_on_previous_text and set hallucination_silence_threshold, which needs word timestamps:import whisper
model = whisper.load_model("large-v3")
result = model.transcribe(
"call.wav",
condition_on_previous_text=False,
word_timestamps=True,
hallucination_silence_threshold=2.0,
)3. Filter the segments that come out. Whisper returns
no_speech_prob, avg_logprob and compression_ratio on every segment, so you can apply the silence rule it softened. This script uses only the standard library. Save the result of transcribe() with json.dump(result, open("result.json", "w")) and, if you ran a VAD, its speech regions as a list of {"start": s, "end": s}:"""Drop Whisper segments that are probably not speech.
Usage: python filter_segments.py result.json [speech.json]
result.json: the dict openai-whisper's transcribe() returns, saved with json.dump.
speech.json: optional list of {"start": s, "end": s} speech regions from a VAD.
"""
import json
import re
import sys
PHRASES = {
"thank you", "thank you for watching", "thanks for watching",
"please subscribe", "like and subscribe", "thank you for listening",
"subtitles by the amara.org community", "you",
}
def norm(text):
return re.sub(r"[^a-z0-9 .']", "", text.lower()).strip(" .")
def overlap(seg, regions):
total = sum(max(0.0, min(seg["end"], r["end"]) - max(seg["start"], r["start"]))
for r in regions)
return total / max(seg["end"] - seg["start"], 1e-6)
def why_drop(seg, regions):
if regions is not None and overlap(seg, regions) < 0.5:
return "outside detected speech"
if seg["no_speech_prob"] > 0.6:
return f"no_speech_prob {seg['no_speech_prob']:.2f}"
if seg["compression_ratio"] > 2.4:
return f"repetition, compression ratio {seg['compression_ratio']:.2f}"
if norm(seg["text"]) in PHRASES:
return "known filler phrase"
return None
def main():
result = json.load(open(sys.argv[1]))
regions = json.load(open(sys.argv[2])) if len(sys.argv) > 2 else None
for seg in result["segments"]:
reason = why_drop(seg, regions)
mark = "DROP" if reason else "keep"
print(f"{mark} {seg['start']:6.1f}-{seg['end']:6.1f} {seg['text'].strip()!r}"
+ (f" ({reason})" if reason else ""))
if __name__ == "__main__":
main()keep 0.0- 4.2 'Welcome back, today we look at the Q3 numbers.' keep 4.2- 9.8 'Revenue grew eleven percent.' DROP 38.0- 41.0 'Thank you for watching!' (no_speech_prob 0.71) DROP 41.0- 52.0 "I'm going to go. I'm going to go. I'm going to go. I'm going to go." (repetition, compression ratio 3.10) DROP 60.0- 63.0 'Subtitles by the Amara.org community' (known filler phrase) keep 70.0- 74.5 'Margins held at thirty-two percent.'
How Mixpeek handles it
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. When it transcribes audio with Whisper, it checks each transcript before indexing it. A transcript that is only a known filler phrase, is more than 70% one such phrase, repeats a phrase of three or more words more than three times, or has fewer than five distinct words across more than 50 characters is left out of the index with the reason recorded, and the batch still completes. The check reads a whole transcript, so one invented line inside an otherwise real transcript passes it.
A clean transcript is what makes audio search return the right second of a recording. To choose the transcription model, see the best speech-to-text APIs; for who said each line, see speaker diarization; for the cost of a full library, see how much it costs to transcribe an hour of audio.
Frequently Asked Questions
Why does Whisper make up words during silence?
Whisper writes the most likely text for each 30-second window, and over silence, music or noise the most likely text can be a stock phrase or a repeat of the last sentence. Its skip-as-silent rule needs a no-speech probability above 0.6 and an average log probability below -1.0, so a confident invented line is kept. A study found roughly 1% of Whisper transcriptions contained entire hallucinated phrases.
Why does Whisper write "Thank you for watching" at the end of my transcript?
The end of a recording is often silence or music, and "Thank you for watching" is one of a small set of phrases Whisper produces for non-speech audio again and again. Run a voice activity filter so the trailing silence is never decoded, or drop a final segment whose no-speech probability is above 0.6 and whose text is only that phrase.
Does voice activity detection fix Whisper hallucinations?
It removes the main trigger, because Whisper never decodes the silent stretches. In faster-whisper set
vad_filter=True on WhisperModel.transcribe(), where it is off by default; it uses Silero VAD with a 0.5 speech threshold and a 2,000 ms minimum silence unless you pass vad_parameters. A VAD can still pass music or background talk as speech, so keep a segment filter after it.What does hallucination_silence_threshold do in Whisper?
With
word_timestamps=True, it tells the whisper package to skip silent stretches longer than the threshold, in seconds, around a segment whose words look anomalous: very short or very long word durations, or word probabilities below 0.15. It is off (None) by default. Start near 2 seconds and tune it on recordings that have the pauses you care about.Should I turn off condition_on_previous_text?
Turn it off when you see repetition loops. With it on, each window's text is fed in as the prompt for the next window, so one hallucinated line can repeat for minutes. The package's docstring says turning it off makes the model less prone to repetition loops and timestamps going out of sync, with text that can be less consistent across windows.
Do other speech-to-text models hallucinate too?
The studies cited here measured Whisper. Any model that generates text from audio can write words that were not spoken, so test whichever API you use on recordings with long pauses, hold music and background noise, and check whether it returns a per-segment confidence or no-speech score you can filter on.