What is ROC Curve and AUC

How It Works

• True positive rate (recall) = true positives / all actual positives
• False positive rate = false positives / all actual negatives
• Sweep the threshold from high to low: each value gives one (FPR, TPR) point, and joining them draws the curve
• A curve bowing toward the top-left corner means the classifier finds positives while admitting few negatives

Worked Example

• Four positives score 0.9, 0.8, 0.6 and 0.55; four negatives score 0.7, 0.4, 0.3 and 0.2
• Of the 16 positive-negative pairs, the positive scores higher in 14, so AUC = 14 / 16 = 0.875
• At threshold 0.5: TPR = 4/4 = 1.0 and FPR = 1/4 = 0.25
• At threshold 0.75: TPR = 2/4 = 0.5 and FPR = 0/4 = 0

ROC vs Precision-Recall Curves

• When positives are rare, for example 1 unsafe video in 1,000, FPR stays tiny even with many false alarms, so ROC AUC can look excellent while precision is poor
• For rare classes, plot precision and recall against the threshold as well
• AUC compares rankings; the threshold you deploy still has to be chosen for the cost of each error type

Choosing Thresholds in Mixpeek

Mixpeek taxonomies assign a label when a document's similarity to it reaches `min_score` (0.0 to 1.0) in the taxonomy enrich stage; the docs use 0.8 in a high-confidence-only example and 0.3 in a multiple-classifications example. Label a sample of a few hundred documents, compute TPR and FPR at several `min_score` values, and pick the point whose error mix fits the use case, such as a low false negative rate for brand safety.

Common Pitfalls

• Reporting AUC alone for a heavily imbalanced problem
• Choosing a threshold on the test set and then reporting results on the same set
• Comparing AUC values measured on different datasets
• Assuming a high AUC means the scores are calibrated probabilities