Frontier model tier
The main leaderboard is the local, open-weight, byte-reproducible benchmark — that's the primary result this project publishes. This page is a separate comparison tier: paid, hosted frontier models called over OpenRouter at reps=1. Because these are remote API calls to closed models, the output is not byte-reproducible the way the local runs are — a re-run against the same provider can return a different answer even at temperature 0. Grading is otherwise identical: the same 147-item regulator-cited corpus, the same three prompts (P0/P1/P2), the same exact-match verdict scoring against the regulatory ground truth.
Models are listed flat, ranked by accuracy — there is no "winner" callout on this page.
Leaderboard
| # | Model | Accuracy | F1 | False-pos rate | Claim-ID | Quarantined |
|---|---|---|---|---|---|---|
| 1 | moonshotai/kimi-k2.6 | P0 | 96.5% | 5.0% | 95.2% | 0 / 147 |
| 2 | moonshotai/kimi-k3 | P0 | 90.1% | 11.7% | 97.4% | 0 / 147 |
| 3 | google/gemini-3.5-flash | P1 | 89.8% | 8.3% | 96.0% | 0 / 147 |
| 4 | qwen/qwen3.7-max | P0 | 86.3% | 0.0% | 98.5% | 3 / 147 |
| 5 | z-ai/glm-5.2 | P0 | 85.2% | 3.3% | 97.0% | 0 / 147 |
| 6 | google/gemini-3.1-pro-preview | P2 | 84.8% | 13.3% | 98.6% | 1 / 147 |
| 7 | openai/gpt-5.5 | P0 | 86.9% | 36.8% | 98.8% | 3 / 147 |
| 8 | anthropic/claude-fable-5 | P2 | 86.9% | 41.7% | 97.7% | 0 / 147 |
| 9 | anthropic/claude-opus-4.8 | P0 | 84.6% | 30.0% | 96.1% | 3 / 147 |
| 10 | anthropic/claude-haiku-4.5 | P0 | 83.9% | 23.3% | 94.5% | 0 / 147 |
| 11 | anthropic/claude-sonnet-5 | P1 | 84.0% | 36.7% | 98.7% | 0 / 147 |
| 12 | openai/gpt-5.6-sol | P1 | 83.9% | 41.7% | 97.5% | 0 / 147 |
| 13 | x-ai/grok-4.5 | P1 | 81.6% | 31.7% | 97.3% | 0 / 147 |
| 14 | deepseek/deepseek-v3.2 | P2 | 80.0% | 25.0% | 97.1% | 0 / 147 |
| Majority-class baseline (always "deceptive") | 59% | – | 100% | – | – |
Each model's row is its best prompt by accuracy (tiebreak: higher F1, then the simpler prompt). Accuracy and F1 are overall verdict correctness, higher is better — the thin tick on the accuracy bar marks the 59% always-"deceptive" baseline. False-pos rate is how often honest content got wrongly flagged as deceptive, lower is better. Claim-ID is, on items correctly flagged deceptive, whether the model also identified which claim was the deceptive one. Quarantined is how many of the 147 items the API returned empty/unparseable content for and the harness scored as an abstention rather than crashing — most models are 0. There is no speed column: hosted API calls aren't timed the way the local RX 7900 XTX runs are.
Full model × prompt grid
| Model | Prompt | Accuracy | F1 | Claim-ID accuracy | False-positive rate |
|---|---|---|---|---|---|
| anthropic/claude-fable-5 | P0 | 76% | 83% | 96% | 52% |
| anthropic/claude-fable-5 | P1 | 80% | 85% | 98% | 43% |
| anthropic/claude-fable-5 | P2 | 82% | 87% | 98% | 42% |
| anthropic/claude-haiku-4.5 | P0 | 81% | 84% | 95% | 23% |
| anthropic/claude-haiku-4.5 | P1 | 81% | 84% | 95% | 23% |
| anthropic/claude-haiku-4.5 | P2 | 78% | 82% | 95% | 30% |
| anthropic/claude-opus-4.8 | P0 | 81% | 85% | 96% | 30% |
| anthropic/claude-opus-4.8 | P1 | 80% | 84% | 95% | 36% |
| anthropic/claude-opus-4.8 | P2 | 81% | 85% | 96% | 33% |
| anthropic/claude-sonnet-5 | P0 | 76% | 81% | 97% | 40% |
| anthropic/claude-sonnet-5 | P1 | 80% | 84% | 99% | 37% |
| anthropic/claude-sonnet-5 | P2 | 77% | 82% | 99% | 42% |
| deepseek/deepseek-v3.2 | P0 | 76% | 81% | 97% | 42% |
| deepseek/deepseek-v3.2 | P1 | 73% | 79% | 96% | 45% |
| deepseek/deepseek-v3.2 | P2 | 77% | 80% | 97% | 25% |
| google/gemini-3.1-pro-preview | P0 | 75% | 82% | 96% | 55% |
| google/gemini-3.1-pro-preview | P1 | 79% | 83% | 97% | 30% |
| google/gemini-3.1-pro-preview | P2 | 83% | 85% | 99% | 13% |
| google/gemini-3.5-flash | P0 | 81% | 84% | 96% | 23% |
| google/gemini-3.5-flash | P1 | 88% | 90% | 96% | 8% |
| google/gemini-3.5-flash | P2 | 86% | 88% | 96% | 7% |
| moonshotai/kimi-k2.6 | P0 | 96% | 97% | 95% | 5% |
| moonshotai/kimi-k2.6 | P1 | 95% | 96% | 96% | 2% |
| moonshotai/kimi-k2.6 | P2 | 93% | 94% | 95% | 12% |
| moonshotai/kimi-k3 | P0 | 88% | 90% | 97% | 12% |
| moonshotai/kimi-k3 | P1 | 83% | 85% | 99% | 17% |
| moonshotai/kimi-k3 | P2 | 84% | 86% | 97% | 15% |
| openai/gpt-5.5 | P0 | 83% | 87% | 99% | 37% |
| openai/gpt-5.5 | P1 | 79% | 84% | 99% | 37% |
| openai/gpt-5.5 | P2 | 77% | 83% | 97% | 43% |
| openai/gpt-5.6-sol | P0 | 76% | 82% | 98% | 47% |
| openai/gpt-5.6-sol | P1 | 79% | 84% | 98% | 42% |
| openai/gpt-5.6-sol | P2 | 76% | 81% | 97% | 42% |
| qwen/qwen3.7-max | P0 | 86% | 86% | 98% | 0% |
| qwen/qwen3.7-max | P1 | 82% | 83% | 95% | 4% |
| qwen/qwen3.7-max | P2 | 84% | 85% | 97% | 5% |
| x-ai/grok-4.5 | P0 | 76% | 82% | 97% | 45% |
| x-ai/grok-4.5 | P1 | 78% | 82% | 97% | 32% |
| x-ai/grok-4.5 | P2 | 75% | 77% | 97% | 18% |
| z-ai/glm-5.2 | P0 | 84% | 85% | 97% | 3% |
| z-ai/glm-5.2 | P1 | 82% | 82% | 100% | 2% |
| z-ai/glm-5.2 | P2 | 83% | 84% | 98% | 5% |