I recently watched a panel of AI agents disprove a sentence I had written, verified to my own satisfaction, and shipped into the documentation of a system I had just built. The sentence asserted an absence: that a certain set of terms appeared nowhere in a large published archive. I believed it. I had checked it, or rather, I had run the kind of check that produces belief: I looked for the terms in the places I expected them, found nothing, and wrote “verified absent.”
The panel was four AI agents I had configured with one unusual instruction: do not review this work; try to prove it wrong. One of them treated my absence claim as a target, went hunting for counterexamples instead of confirmation, and produced them in minutes. The claim was false. My verification had been a search for agreement wearing a lab coat.
That correction never reaches me under the standard prompt. Ask an AI to “review” something and you get what review-as-assessment produces: plausibility checks, style observations, gentle suggestions, and, because these models are tuned on human approval, agreement with whatever frame you handed over. The failure is not that the model is weak. It is that assessment accepts your premises, and your premises are the one place it will never look.
The fix is not asking harder for honesty. Everyone has tried “be brutally honest” and received the same agreeable review in a gruffer voice. Sincerity is not the missing ingredient. Structure is.
Review asks “is this good?” and inherits your framing. Refutation asks “prove this wrong” and must construct its own.
The protocol
Here is the structure I now use for anything that matters: code, designs, factual claims, plans, drafts. Four rules, each closing a specific hole that assessment leaves open.
Rule one: instruct refutation, not assessment. Each agent’s task is to disprove the work: find the input that breaks the function, the counterexample to the claim, the scenario where the design corrupts data. The verb changes everything downstream. A reviewer scans for things worth commenting on; a refuter constructs attacks. An attack that fails to land is information. A compliment is not.
Rule two: many agents, distinct hostile lenses. Not one agent told to be thorough, but several, each assigned a different way the work could be wrong: one attacks correctness, one attacks security, one attacks the failure modes (“what happens when this cannot run?”), one attacks the factual claims’ provenance (“where is the evidence this was ever true?”). Five identical skeptics find the same bug five times. The hole in the failure handling of a pre-commit guard I maintain was invisible to the correctness lens and obvious to the what-if-it-breaks lens. Lenses are cheap; assign them deliberately.
Rule three: uncertainty counts against the work. Each refuter’s instruction ends with: if you construct an attack and cannot resolve whether it lands, report the work as refuted and say why. This is the inversion that kills flattery at the root. A reviewer’s uncertainty becomes hedged praise; a refuter’s uncertainty becomes a failed survival. The work must earn its pass, not merely collect favorable-to-mixed reviews.
Rule four: findings must survive their own panel. Refuters produce false positives: confident, plausible, wrong. So each claimed defect goes to independent verifiers asked to refute the finding, and a finding that cannot be reproduced or demonstrated dies there. Note the deliberate asymmetry between the two stages. At the panel stage, uncertainty is charged against the work; at the verification stage, it is charged against the finding. The first kills flattery. The second kills paranoia. In one recent run, a panel surfaced twenty-nine candidate defects; verification separated the five that were real, exploitable holes from the plausible-but-wrong findings that would have cost me a morning of phantom fixes.
The prompt shape, concretely:
You are one lens on a hostile review panel. Your lens: [failure modes / security / correctness / provenance]. Your task is to REFUTE the work below: construct the strongest case that it is wrong, unsafe, or unsupported, and demonstrate it concretely with a failing input, a counterexample, or missing evidence. If you cannot construct a credible attack, say the work survived your lens; do not manufacture objections. If you construct an attack but cannot resolve it, default to refuted and say why.
And for verifying findings:
Try to refute this claimed defect. Reproduce it or demonstrate it cannot occur. If you cannot decide, the finding is unproven; reject it.
What the panels have actually caught
The receipts, from my own recent work. They are the reason I trust this enough to write it down.
The absence claim I opened with: disproved by the provenance lens, minutes into its first run, in documentation I had already merged.
Five real, exploitable fail-open holes in protective automation that had passed conventional review, mine and my agents’ both, over months of use. Each came from a lens constructing attacks no assessing reviewer had reason to attempt: what if the filename carries a quote character; what if the change arrives as a rename; what if the exclusion pattern is invalid. Every one now exists as a regression test, and the holes themselves, with the fail-closed contract that fixes their class, have their own article.
And the negative catch, which I value nearly as much: panels killing their own findings, the plausible-sounding defect reports that independent verification could not reproduce, before I spent a morning fixing problems that did not exist.
Once you have watched all three happen, single-prompt “please review this critically” stops feeling like diligence. It feels like asking the work’s admirer to officiate its trial. This is the same structural humility I argued for in treating a green test suite as a warning rather than a clearance: the signal you wanted is exactly the signal to distrust, and the fix is structure that makes agreement expensive.
Costs, limits, and where not to use it
A refute panel spends several times the tokens and minutes of a single review, so it is not the tool for routine work: a linting pass, a small refactor, a draft you will rewrite anyway. I reserve panels for work where being wrong is expensive: protective systems, factual claims headed for publication, architectural decisions, anything irreversible. Below that threshold, one reviewer with a specific brief beats a panel on cost and usually suffices.
The protocol also has failure modes of its own, and honesty about them is part of the method. Refuters instructed to attack will sometimes manufacture objections to have something to report; that is what the finding-verification stage exists to absorb, and why “say the work survived your lens” belongs in the prompt. Lens assignment is judgment: a panel without the right lens misses the defect that lens would have owned, and no process guarantees you chose well. And a panel’s silence means only that the work survived the attacks you thought to commission. That is much stronger evidence than agreement. It is still not a certificate.
One more boundary: the panel attacks the work, not the direction. Whether the thing is worth building, whether the article is worth publishing, whose judgment sets the bar: that stays with me. I have written before about making AI a thinking partner rather than an answer machine; refutation panels are that idea operationalized for verification, not a delegation of taste.
The pattern extends past code with no modification. The topic list for my own writing now goes through hostile lenses (a jaded reader who assumes every AI-adjacent post is empty; a skeptic who kills anything without a transferable pattern), and article drafts face the same panel structure before publication. Prose findings need the two-stage asymmetry even more than code findings do, because in prose, plausible-but-wrong is the native failure mode of both the writing and the reviewing.
Your AI wants to tell you the work is good, and the work is sometimes good. The only way to know is to make “good” the verdict that survives when several well-aimed attempts to destroy it have failed. Stop asking for reviews. Commission refutations, arm them with different weapons, and let the work earn its way past them.