Why does my content moderation flag so many false positives?
Usually because one confidence threshold is doing every job. The default cutoffs in moderation APIs are low: Amazon Rekognition returns any moderation label at 50% confidence or higher unless you set a minimum. Harmful content is rare, so even an accurate classifier flags more safe items than harmful ones. And a video checked frame by frame gives each frame its own chance to misfire. The fix is a threshold per category, set from a labeled sample of your own content, a review band between "allow" and "block", and video judged by segment with the context the classifier cannot see.
What do moderation APIs actually return?
Each API returns a score per category, and the decision about what counts as a violation is left to you.
| API | What it returns | Default or scale |
| Amazon Rekognition DetectModerationLabels | Labels with a confidence from 0 to 100 | Labels at 50% or more are returned unless you set MinConfidence |
| Google Cloud Vision SafeSearch | A likelihood for adult, spoof, medical, violence and racy | Five buckets, VERY_UNLIKELY to VERY_LIKELY |
| Azure AI Content Safety | A severity for hate, sexual, violence and self-harm | Text on a 0 to 7 scale; images on 0, 2, 4 and 6 |
Why does a good classifier still flag mostly safe content?
Because harmful content is rare. Suppose 1 upload in 1,000 breaks your policy, and your classifier wrongly flags 1% of safe uploads. Out of 100,000 uploads, about 100 are harmful and 99,900 are safe, so the classifier flags about 999 safe uploads alongside at most 100 harmful ones. Roughly nine in ten flags are wrong, from a model that is right about 99% of safe content.
The rarer the violation, the worse this gets, and it differs by category. Weapons may be rare on a cooking site and common on a hunting forum, so one threshold across every category is too strict for some and too loose for others.
Why does video moderation flag even more?
Each sampled frame is a separate chance to misfire. A 10-minute video sampled at one frame per second is 600 checks. If each frame has a 0.1% chance of a false flag, the chance that at least one of the 600 fires is 1 - 0.999^600, about 45%. Nearly half of perfectly safe videos would get flagged somewhere.
Frames also lose context. A single frame of a surgery, a boxing match, a war documentary or a museum statue looks like the thing a policy bans, and the scene around it is what makes it acceptable.
Why doesn't similarity search work as a moderation filter?
A common shortcut is to embed each video segment and flag anything close to a text description such as "a gun being fired". In our own test of that approach, no similarity cutoff separated the footage: a revolver fired at the camera scored 0.33 against the description, below a folk dance at 0.34, and real logo and product footage scored 0.03 to 0.05. Similarity is good for ranking which segments to look at, and a separate judge has to decide whether each one shows the thing.
How do I cut false positives?
1. Label a sample of your own content. A few hundred items per category, drawn from what users actually upload, with the flags you got and the right answer for each. 2. Set a threshold per category from that sample. Pick the score where precision is acceptable for that category, and accept that some categories need a much higher bar than others. 3. Use two thresholds per category. Allow below the lower one, block above the upper one, and send the band between them to review. Most volume never needs a person, and the uncertain cases get one. 4. Judge video by segment. Require a label on several consecutive frames, or score the scene as a whole, before acting on it. 5. Give the review band context. A vision language model that sees the segment, its transcript and your written policy can tell a surgery from an assault far better than a single-frame score. 6. Measure precision every week. Content drifts. Re-check a sample of flags and blocks and move thresholds when the mix changes.
How do I do this with Mixpeek?
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. A retriever can find candidate segments with a feature_search stage and then pass each one to an llm_filter stage with
include_media on, so Gemini
watches the segment and keeps it only when your written criteria hold. Other providers see the segment's
thumbnail.[
{
"stage_name": "candidates",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "a weapon being fired" },
"top_k": 50
}
]
}
}
},
{
"stage_name": "judge",
"stage_type": "filter",
"config": {
"stage_id": "llm_filter",
"parameters": {
"criteria": "Keep the segment only if a real firearm is fired at a person or toward the camera. Do not keep target shooting at a range, toy guns, or footage from films or games.",
"include_media": true,
"include_reasoning": true
}
}
}
]The reasoning recorded for each decision gives your reviewers the why, and it is what you read when tuning the
criteria. Prices are on the pricing page. To compare moderation services, see the
best AI content moderation tools, the
best video moderation tools and the
best NSFW detection APIs. For advertiser rules, see
brand safety vs brand suitability.
Because one threshold does every job. Default cutoffs are low, harmful content is rare so even accurate classifiers flag more safe items than harmful ones, and frame-by-frame video checks give every frame a chance to misfire. Set thresholds per category from your own labeled content and send a middle band to review.
There is no single right number. Label a few hundred of your own items per category, then pick the score where precision is acceptable for that category. Use two thresholds: allow below the lower one, block above the upper one, and review what falls between.
A video is checked many times. At one frame per second a 10-minute video is 600 checks, and with a 0.1% false-flag rate per frame about 45% of safe videos get flagged at least once. Requiring a label across consecutive frames, or scoring the whole scene, removes most of those.
Yes, when it judges the uncertain middle band rather than everything. A vision language model that sees the segment, its transcript and your written policy can use context a single-frame classifier cannot, such as telling medical footage from violence. It is slower and costs more per item, so run it after a cheaper first pass.
Frequently Asked Questions
Why does my content moderation flag so many false positives?
Because one threshold does every job. Default cutoffs are low, harmful content is rare so even accurate classifiers flag more safe items than harmful ones, and frame-by-frame video checks give every frame a chance to misfire. Set thresholds per category from your own labeled content and send a middle band to review.
What confidence threshold should I use for content moderation?
There is no single right number. Label a few hundred of your own items per category, then pick the score where precision is acceptable for that category. Use two thresholds: allow below the lower one, block above the upper one, and review what falls between.
Why does moderation flag safe videos more than safe images?
A video is checked many times. At one frame per second a 10-minute video is 600 checks, and with a 0.1% false-flag rate per frame about 45% of safe videos get flagged at least once. Requiring a label across consecutive frames, or scoring the whole scene, removes most of those.
Can an LLM reduce moderation false positives?
Yes, when it judges the uncertain middle band rather than everything. A vision language model that sees the segment, its transcript and your written policy can use context a single-frame classifier cannot, such as telling medical footage from violence. It is slower and costs more per item, so run it after a cheaper first pass.