AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox

METAL LAB

No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios

arXiv:2608.217752026-08-25

Testing 53 AI safety filters shows no single model catches every kind of harmful content

Researchers benchmarked 53 AI moderation models, both general-purpose LLMs like GPT and specialized safety models like Llama Guard, across 11 datasets covering four types of harm. They tested each model in two modes: seeing only the user's prompt, or seeing the prompt plus the model's actual response. Large frontier models led in some categories but fell behind smaller specialized models in others, and real-world conversational safety remained unsolved for every model family tested.

METAL LAB explanatory visual

How 53 models were tested across 11 datasets and 4 harm categories

Evidence statusMeasured results reported

  1. Input mode: Q vs QAQ feeds only the user prompt to the moderator; QA feeds the prompt plus the model's actual response before judging safe/unsafe
  2. Harm categories C1-C4C1 adversarial jailbreaks, C2 standard policy violations, C3 over-refusal on benign prompts, C4 real-world conversational safety
  3. Model families comparedSpecialized moderators (Llama Guard, WildGuard, BingoGuard, PolyGuard) use their native input formats; general-purpose LLMs (GPT, Gemini, Command-A, etc.) use the authors' optimized 14-category safety prompt
  4. Scoring methodDiscrete safe/unsafe verdicts are scored with macro F1, and ranking gaps are checked with paired-bootstrap confidence intervals
  5. Outcome patternWinners shift by category: large commercial models lead in Q, small specialized moderators lead in QA, and every model struggles on C4 conversational safety
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

  1. 53 models (31 open-source, plus commercial ones) were evaluated on 11 safety datasets under two settings: prompt-only (Q) and prompt-with-response (QA).
  2. Harms were grouped into four categories: C1 adversarial jailbreak resistance, C2 standard policy enforcement (hate, harassment, etc.), C3 over-refusal on benign prompts, and C4 conversational and real-world safety.
  3. Since every model outputs only a discrete safe/unsafe verdict rather than a probability score, performance was measured with macro F1 (precision and recall averaged across classes), and paired-bootstrap confidence intervals found a statistically significant gap between the top-1 and rank-2 model in only 4 of 18 tested cases.
  4. The team built an optimized safety prompt (14 violation categories plus guidance for handling borderline content) for general-purpose LLMs, which beat the standard Llama Guard prompt by 4-15% and a simple safe/unsafe instruction by 18-28% in macro F1.
Figure 1: Average F1 over datasets for Q setting.
Figure 1: Average F1 over datasets for Q setting.
Table 2: Prompt template ablation: average macro F1 (%) across three prompt designs.
SimpleLlamaGuardOurs
ModelQQAQQAQQA
Gemma-3-4B42.852.760.960.964.767.5
Llama-3.2-3B37.131.057.462.564.869.9
Phi-4-mini46.458.951.952.866.466.2
Figure 2: Average F1 over datasets for QA setting.
Figure 2: Average F1 over datasets for QA setting.
Table 4: Significance summary per dataset and mode (Q/QA). Δ1​-​2: paired-bootstrap difference in macro F1 between rank-1 and rank-2 over shared samples, with 95% CI (1,000 resamples). Sig.: Yes if the 95% CI excludes zero (4/18 cases). Tie-tier: number of models whose macro-F1 point estimate falls inside the rank-1 95% CI, i.e. statistically indistinguishable from the leader.
DatasetSet.Δ1​-​2 [95% CI] (%)Sig.Tie-tier
harmaugQ+2.3 [+0.3, +4.4]Yes2
harmbenchQ+0.6 [−2.4, +3.8]–10
xrtestQ+1.0 [−4.1, +5.7]–14
aegisQ+0.8 [−3.0, +4.4]–8
oaiQ+0.2 [−2.0, +2.4]–11
toxicchatQ+0.0 [−4.2, +4.1]–6
wildguardQ+0.2 [−0.9, +1.4]–5
simplesafetyQ+0.0 [+0.0, +0.0]–28
xstestQ+0.7 [−0.7, +2.0]–3
beavertailsQ+0.3 [−2.6, +3.6]–16
bingoQ+0.6 [+0.3, +0.9]Yes1
harmaugQA+9.2 [+6.8, +11.6]Yes1
harmbenchQA+1.3 [−1.1, +3.7]–6
xrtestQA+0.5 [−2.3, +3.6]–2
wildguardQA+1.5 [−0.7, +3.9]–3
xstestQA+0.4 [−1.4, +2.2]–4
beavertailsQA+0.7 [−1.4, +2.7]–3
bingoQA+1.5 [+1.0, +2.0]Yes1
Figure 3: Our framework: 11 benchmarks in four challenge categories (C1–C4), evaluated on 53 models under Q and QA.
Figure 3: Our framework: 11 benchmarks in four challenge categories (C1–C4), evaluated on 53 models under Q and QA.

Findings

  • In the Q setting, large commercial models -- GPT-4o-mini, GPT-4.1, GPT-4.1-mini, GPT-5, Gemini-2.5-pro, and Command-A -- achieved the highest average F1, in the 75-77% range.
  • In the QA setting, classification got easier once the model's actual response was visible, with Llama-Guard-3-8B leading at 78.6% F1, closely followed by BingoGuard-Llama-8b and Gemma-2-27b-it.
  • C1 (adversarial/jailbreak) was hardest in Q mode, with average F1 only 52.7-61.2%; Llama-Guard-3-8B, Llama-Guard-4-12B, Phi-4-reasoning, and GPT-4.1 topped out around 67%.
  • In C2 (standard policy enforcement), BingoGuard-Llama-8B (87.1%) and BingoGuard-Phi3-3B (82.2%) consistently outperformed all other models in both Q and QA settings.
  • C3 (over-refusal on benign prompts) was easiest, with 15 models exceeding 95% F1 in Q mode (GPT-4.1 at 98.4%, GPT-5 at 98.2%), while C4 (conversational/real-world safety) stayed the hardest category, averaging only about 52% F1 in both Q and QA.
Figure 4: Macro F1 (averaged across all datasets) vs. per-sample latency for the 31 open-source models, under prompt-only (Q) and prompt-with-response (QA) settings. Marker shape encodes model family; color identifies individual models (see legend). The gold star marks the best macro F1 in each setting; the green diamond marks the “fast & accurate” pick (within 2% of the best F1 at the lowest latency). Gemma-2-27B-it, Gemma-3-27B-it, and Phi-3.5-MoE-instruct were evaluated on 4× A100 40GB; all other models on 1× A100 40GB.
Figure 4: Macro F1 (averaged across all datasets) vs. per-sample latency for the 31 open-source models, under prompt-only (Q) and prompt-with-response (QA) settings. Marker shape encodes model family; color identifies individual models (see legend). The gold star marks the best macro F1 in each setting; the green diamond marks the “fast & accurate” pick (within 2% of the best F1 at the lowest latency). Gemma-2-27B-it, Gemma-3-27B-it, and Phi-3.5-MoE-instruct were evaluated on 4× A100 40GB; all other models on 1× A100 40GB.

Where it can be used

  • Services needing to catch adversarial jailbreak attempts might prioritize large general-purpose LLMs, which led in that category.
  • Deployments that can afford to inspect the model's actual output might consider smaller specialized safety models like Llama Guard or BingoGuard, which were strongest once a response was visible.
  • Teams worried about over-blocking benign requests could use datasets like XSTest and SimpleSafetyTests as a sanity-check baseline before deploying a moderator.
  • Latency-sensitive real-time systems could look at compact models such as BingoGuard-Phi3-3B or BingoGuard-Llama-8B, which the authors found near the fast-and-accurate frontier.

Limits and open work

  • The evaluation covers only English, single-turn interactions with binary safe/unsafe labels, so generalization to multilingual, code-switched, multi-turn, or severity-graded settings is untested.
  • Each dataset was capped at 1,000 samples, which may not capture long-tail patterns of harmful content seen in real deployments.
  • QA evaluation relies on responses from a limited set of source models included in the original datasets, so moderators trained on similar distributions may have an unfair advantage.
  • The optimized-prompt improvement was tested on only three open-source models; the authors expect but have not empirically verified that gains transfer to closed-source frontier models.
  • Only 4 of 18 tested top-1 vs. rank-2 comparisons showed a statistically significant gap, meaning many observed ranking differences could reflect sample noise rather than a true performance difference.

Why it matters

Anyone deploying an AI safety layer gets evidence that picking 'the best model' overall is the wrong question; the right model depends on the type of harm and whether a response is available to inspect. It also challenges the common assumption that bigger models are automatically safer.

Terms in this paper

  • Q / QA setting · Q means the moderator sees only the user's prompt; QA means it also sees the model's actual response before judging safety
  • macro F1 · A performance score combining precision and recall, averaged per class, used because models only output a discrete safe/unsafe label
  • over-refusal · When a model wrongly flags a harmless request as dangerous and refuses to answer it
  • specialized moderators (Llama Guard, WildGuard, BingoGuard, PolyGuard) · Models trained specifically to classify content as safe or unsafe, evaluated here in their own native input format
  • paired-bootstrap confidence interval · A statistical resampling method used to check whether a performance gap between two models is real or just sample noise

Original abstract (English)

Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adversarial jailbreaks that bypass safety filters to implicit hate that evades detection, the range of risks these models pose continues to grow. While both specialized content moderators and general-purpose LLMs are being used as safety layers, the question of which model is best suited for which type of harmful content remains unanswered. We present the most comprehensive evaluation of LLM safety capabilities to date, systematically testing \textbf{53} models across \textbf{11} datasets that we organize into four distinct categories. Our evaluation under both prompt-only and prompt-response settings uncovers critical blind spots: large frontier models that lead on one category fall significantly behind smaller, specialized alternatives on others, and real-world conversational safety remains largely unsolved across all model families. These findings challenge the assumption that scale alone ensures safety, and provide the community with a structured framework for informed model selection.

Authors · Afshin Orojlooyjadid, Hitesh Patel

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Afshin Orojlooyjadid et al., arXiv:2608.21775, cc-by-nc-sa-4.0