Can a local LLM spot deceptive advertising?

147 real marketing claims — each one ruled deceptive or cleared by a regulator or court (FTC, ASA, NAD, ACCC, SEC), with the citation to prove it. Small open-weight models judge every claim locally at temperature 0 with schema-constrained output. Grading is exact-match against the regulatory verdict. Every result is byte-reproducible and hash-verified.

BEST DETECTOR

nemotron-3-nano:4b

92.5%accuracy (best prompt)

vs 59% for a model that calls everything deceptive

F1 93.6% · 157.5 tok/s median

Leaderboard

#	Model	Accuracy	F1	False-pos rate	Claim-ID	tok/s	s/item
1	nemotron-3-nano:4b	92.5% P0	93.6%	6.7%	97.5%	157.5	0.40
2	cogito:32b	89.0% P0	91.0%	18.3%	92.6%	33.2	0.87
3	glm-4.7-flash	88.1% P0	89.8%	12.1%	89.3%	92.5	0.49
4	granite3.3:8b	83.7% P0	85.0%	8.3%	92.7%	102.3	0.28
5	deepseek-r1:8b	83.7% P0	84.6%	5.0%	98.5%	96.1	0.40
6	falcon3:10b	83.6% P0	84.4%	5.0%	92.3%	91.7	0.38
7	lfm2:24b	82.1% P2	85.6%	27.6%	84.4%	183.3	0.27
8	mistral-small3.2:24b	81.6% P1	83.2%	11.7%	95.5%	47.5	0.71
9	phi4:14b	81.4% P2	84.2%	22.0%	94.4%	72.9	0.43
10	gemma4:26b	80.3% P0	80.8%	5.0%	91.8%	100.6	0.70
11	deepseek-r1:32b	79.7% P0	84.3%	37.9%	94.9%	33.2	0.79
12	qwen2.5:14b	79.5% P0	83.3%	30.5%	93.3%	67.0	0.50
13	qwen3.5:9b	78.9% P0	78.3%	0.0%	98.2%	17.4	1.43
14	aya-expanse:32b	77.5% P0	79.2%	15.0%	96.8%	34.1	1.00
15	exaone3.5:32b	77.5% P0	78.4%	10.0%	93.3%	34.6	0.88
16	qwen3.6:27b	77.5% P0	76.6%	0.0%	94.4%	36.6	1.01
17	qwen2.5:7b	76.9% P0	81.9%	40.0%	93.5%	119.4	0.34
18	phi4-reasoning:14b	76.7% P0	77.9%	13.3%	95.0%	63.9	0.48
19	gemma4:12b	74.7% P2	74.1%	6.7%	90.6%	66.6	0.84
20	mistral-nemo:12b	73.3% P0	71.9%	5.0%	96.0%	93.9	0.43
21	gemma2:9b	70.3% P0	80.0%	72.9%	90.7%	86.4	0.53
22	phi4-mini:3.8b	69.9% P0	68.1%	8.3%	85.1%	167.0	0.34
23	deepseek-r1:14b	67.3% P2	78.4%	80.0%	94.3%	61.6	0.52
24	gemma3n:e4b	67.1% P0	76.5%	66.1%	94.9%	74.5	0.78
	Majority-class baseline (always "deceptive")	59%	–	100%	–	–	–

Each model's row is its best prompt by accuracy (tiebreak: higher F1, then the simpler prompt). Accuracy and F1 are overall verdict correctness, higher is better — the thin tick on the accuracy bar marks the 59% always-"deceptive" baseline. False-pos rate is how often honest content got wrongly flagged as deceptive, lower is better. Claim-ID is, on items correctly flagged deceptive, whether the model also identified which claim was the deceptive one. tok/s and s/item are median generation speed on an RX 7900 XTX — see the methodology page for how timing is measured and why it's excluded from the determinism hash. Click a column header to sort; the baseline row always stays last.

Accuracy vs. speed

Best-prompt accuracy against median generation throughput on an RX 7900 XTX — up and to the right wins.

Qwen
Gemma
Phi
Mistral
Deepseek
Granite
Other

Prompt sensitivity

Accuracy for each model at each of the three prompts (P0 naïve, P1 instructed, P2 rubric).

Full model × prompt grid

Model	Prompt	Accuracy	F1	Claim-ID accuracy	False-positive rate
aya-expanse:32b	P0	78%	79%	97%	15%
aya-expanse:32b	P1	71%	73%	93%	18%
aya-expanse:32b	P2	73%	75%	92%	22%
cogito:32b	P0	89%	91%	93%	18%
cogito:32b	P1	81%	84%	91%	27%
cogito:32b	P2	80%	83%	95%	28%
deepseek-r1:14b	P0	64%	78%	95%	96%
deepseek-r1:14b	P1	61%	76%	94%	97%
deepseek-r1:14b	P2	67%	78%	94%	80%
deepseek-r1:32b	P0	80%	84%	95%	38%
deepseek-r1:32b	P1	72%	79%	93%	53%
deepseek-r1:32b	P2	79%	84%	94%	37%
deepseek-r1:8b	P0	84%	85%	98%	5%
deepseek-r1:8b	P1	71%	77%	93%	47%
deepseek-r1:8b	P2	71%	78%	92%	49%
exaone3.5:32b	P0	78%	78%	93%	10%
exaone3.5:32b	P1	73%	78%	94%	37%
exaone3.5:32b	P2	74%	78%	95%	28%
falcon3:10b	P0	84%	84%	92%	5%
falcon3:10b	P1	67%	71%	95%	35%
falcon3:10b	P2	70%	74%	93%	32%
gemma2:9b	P0	70%	80%	91%	73%
gemma2:9b	P1	66%	77%	92%	80%
gemma2:9b	P2	65%	77%	90%	81%
gemma3n:e4b	P0	67%	76%	95%	66%
gemma3n:e4b	P1	65%	74%	96%	60%
gemma3n:e4b	P2	67%	74%	94%	52%
gemma4:12b	P0	68%	64%	90%	3%
gemma4:12b	P1	73%	73%	90%	7%
gemma4:12b	P2	75%	74%	91%	7%
gemma4:26b	P0	80%	81%	92%	5%
gemma4:26b	P1	73%	74%	98%	12%
gemma4:26b	P2	74%	74%	96%	10%
glm-4.7-flash	P0	88%	90%	89%	12%
glm-4.7-flash	P1	82%	85%	90%	29%
glm-4.7-flash	P2	78%	83%	91%	41%
granite3.3:8b	P0	84%	85%	93%	8%
granite3.3:8b	P1	79%	82%	90%	25%
granite3.3:8b	P2	79%	81%	89%	15%
lfm2:24b	P0	73%	82%	9%	64%
lfm2:24b	P1	62%	76%	75%	93%
lfm2:24b	P2	82%	86%	84%	28%
mistral-nemo:12b	P0	73%	72%	96%	5%
mistral-nemo:12b	P1	72%	74%	89%	22%
mistral-nemo:12b	P2	73%	74%	89%	17%
mistral-small3.2:24b	P0	80%	80%	93%	3%
mistral-small3.2:24b	P1	82%	83%	96%	12%
mistral-small3.2:24b	P2	78%	78%	95%	10%
nemotron-3-nano:4b	P0	93%	94%	98%	7%
nemotron-3-nano:4b	P1	72%	81%	92%	65%
nemotron-3-nano:4b	P2	72%	81%	93%	65%
phi4-mini:3.8b	P0	70%	68%	85%	8%
phi4-mini:3.8b	P1	66%	70%	93%	33%
phi4-mini:3.8b	P2	66%	67%	90%	25%
phi4-reasoning:14b	P0	77%	78%	95%	13%
phi4-reasoning:14b	P1	76%	78%	95%	18%
phi4-reasoning:14b	P2	73%	75%	97%	20%
phi4:14b	P0	77%	80%	92%	22%
phi4:14b	P1	73%	77%	97%	35%
phi4:14b	P2	81%	84%	94%	22%
qwen2.5:14b	P0	79%	83%	93%	31%
qwen2.5:14b	P1	76%	79%	94%	27%
qwen2.5:14b	P2	72%	77%	94%	38%
qwen2.5:7b	P0	77%	82%	94%	40%
qwen2.5:7b	P1	72%	78%	92%	47%
qwen2.5:7b	P2	76%	79%	94%	23%
qwen3.5:9b	P0	79%	78%	98%	0%
qwen3.5:9b	P1	74%	72%	96%	0%
qwen3.5:9b	P2	74%	72%	96%	0%
qwen3.6:27b	P0	78%	77%	94%	0%
qwen3.6:27b	P1	75%	74%	94%	3%
qwen3.6:27b	P2	72%	70%	96%	2%

Key findings

False-positive rate on honest content spans from 0% (best-calibrated) to 97% (worst over-flagger) across the (model × prompt) grid.
The core failure mode is over-flagging honest-but-aggressive ads, not missing lies — claim identification on correctly-flagged deceptive items stays in the 9%-98% range across every model and prompt.
The "instructed" P1 prompt (a definition + tactics taxonomy) is the weakest performer on average F1 in this pilot (77% avg), not the strongest — more instruction does not reliably mean better detection.
Which prompt wins is model-dependent: a plain naïve prompt or the full per-tactic rubric each take the top average-F1 spot on different models — there is no single prompt that dominates across every model tested (P0 currently averages highest overall at 80%).