Your Coding Agent Says It's Done. It Isn't.
Asking the agent to double-check is the same model grading its own homework. The fix sits outside the model and touches the outside world before the turn is allowed to end.
Your coding agent says the deploy is live, the tests pass, the file is written. You check. It isn't, they don't, it isn't. Asking it to double-check does not fix this — that is the same model grading its own homework. The fix has to sit outside the model and touch the outside world before the turn is allowed to end. Here is what that looks like in practice, with the commands.
Why “are you sure?” never works
A completion claim is generated the same way every other sentence is: as the most plausible continuation. After a long, mostly-successful sequence of edits, “Done — the page is live” is an extremely plausible continuation. It is produced by the same process whether or not the page is live, because nothing in that process involves looking at the page.
So when you reply “are you sure?”, you are asking the model to sample again from a context that already contains its own confident claim. Sometimes it flips — and now you have two unreliable answers instead of one. Self-review catches sloppiness. It cannot catch a belief the model has no way to test.
You cannot fix a hallucinated fact with a better prompt. You fix it with a process that goes and looks.
The four false-done modes, from most to least common
1. It never ran the thing
The agent wrote the test, described what the test would prove, and reported the result it expected. No test runner was invoked. This is the plain case and it is caught by one question: which tool call produced that output? If there is no command in the transcript, there is no result.
2. It ran the thing and misread it
Subtler and much more expensive. The command ran, exited 0, and the agent read that as success — but the command did not do what the agent thought.
We lost most of a morning to exactly this. A build wrapper printed a tidy Errors: 0 | Warnings: 0 summary and exited clean, so the agent concluded the fix had shipped. The wrapper had short-circuited: it never regenerated the static HTML. We nearly signed off a fix by checking a page that had been prerendered seven hours earlier. The lesson went straight into our failure log: for build output, verify the artifact — its timestamp and its contents — never the summary line.
# did the build actually write new files? $ find .next -name '*.html' -newermt '-5 minutes' | head $ ls -lT .next/server/app/index.html # and does the served output contain the change? $ curl -s https://example.com/ | grep -c 'the-string-i-just-added' 1
3. It verified against the wrong thing
The agent curls localhost:3000and gets a 200 — from a dev server another session started, running last week's branch. Or it checks a URL that redirects to a cached edge copy. The check was real; the target was wrong. This one gets much worse the moment you run more than one agent on a box, which is its own subject: running multiple coding agents on one machine.
4. The gate itself was satisfied by the wrong file
The one nobody predicts. You build a quality gate, the agent passes it, and the gate was reading a checklist file rather than the deliverable. Ours did exactly that: the gate scanned the quality_check.md the agent had just written and never opened the package it described. A hand-off nearly went out with seventeen instances of retired internal shorthand in it, caught by a manual sweep and nothing else. A gate that reads the agent's summary is part of the agent's summary.
The principle: a claim is a testable proposition
Every completion claim contains nouns you can go look at. “I pushed the branch and opened a PR” contains a branch name and a PR number. “The docs page is live” contains a URL. “Published to npm” contains a package and a version. Each of those is a one-line shell command away from being settled.
# a URL is live and returns what it should
$ curl -s -o /dev/null -w '%{http_code}\n' https://example.com/blog/my-post
200
# a repo / PR / release exists (and is not a hopeful invention)
$ gh api repos/OWNER/REPO --jq '.full_name'
$ gh api repos/OWNER/REPO/pulls/42 --jq '.state'
# a package version is actually on the registry
$ npm view my-package version
0.1.4
# the file the agent says it wrote exists and is not empty
$ wc -c path/to/file && ls -lT path/to/fileMaking it structural instead of aspirational
Everything above works if you remember to do it, which means it works for two weeks. To make it stick, the check has to be something other than your own discipline. Three layers, in increasing order of effort:
Layer 1: a claim-checking CLI
Write one command that takes a claim in plain text, extracts every URL, repo, and package name in it, hits each one for real, and exits non-zero if any of them fails. Then the review step is one line instead of five, and it composes in a shell:
$ verify "shipped /blog/my-post, live on example.com, repo OWNER/REPO updated" URL https://example.com/blog/my-post 200 OK REPO OWNER/REPO ok pushed 4m ago ✗ URL https://example.com/blog/my-other-post 404 FAIL $ echo $? 1 # and because it exits properly, it gates other things $ verify "$CLAIM" && git push
Layer 2: a stop hook that refuses the turn
Claude Code fires hooks on real events — before and after tool use, on session start, on prompt submit, and when the agent tries to end its turn. That last one is the leverage point. A Stop hook is a shell command, so it can run your checker; if it exits non-zero, the turn does not end and your message goes back to the model as the reason.
{
"hooks": {
"Stop": [
{
"hooks": [
{
"type": "command",
"command": ".claude/hooks/done_gate.py"
}
]
}
]
}
}The gate itself is small: read the transcript, find the completion language, extract the checkable nouns, run the checks, and block with a specific reason if any fail. The important design decision is that the gate has no opinion about quality. It only decides whether the claims made are true. Gates that try to judge good work get argued with and disabled; gates that only check existence stay installed.
Layer 3: an adversarial second read, used sparingly
For work with real blast radius — anything public, anything that spends money — a second model prompted to attack the claim (not to review the work) catches a different class of error than either the author model or a shell check. Keep it narrow: “here is the claim and here is the evidence; name every way this could be false.” Run it on the few things that matter, not on every turn, or it becomes noise you skim.
What to change today, before you build anything
- Ban unaccompanied completion language. Put it in your
CLAUDE.md: no “done”, “fixed”, “deployed”, or “verified” without the command and its output pasted immediately after. Vague success is the tell. (More on making that file actually get followed: how to write a CLAUDE.md that gets followed.) - Require the last mile, not the middle.“The build succeeded” is the middle. “The apex domain returns 200 and contains the new string” is the last mile. Ask for the end of the chain, in production, by URL.
- Make the agent state its own falsifier.Before it starts: “what command will prove this worked, and what output would mean it did not?” Deciding the test in advance is much harder to rationalize around than judging your own result afterwards.
- Check one claim per day at random. Not to police the agent — to calibrate yourself. You will find the shape of failure that is specific to your stack within a week, and that shape is what you automate first.
Honest limits
Existence checks prove existence. A page can return 200 and be wrong; a test can pass and assert nothing; a PR can be open and be garbage. This whole approach kills the category of failure where the agent asserts something that is simply not there — which is the majority of wasted time — and does nothing about code that is real and bad. That still needs review.
Second: a gate you route around is worse than no gate, because it produces false confidence. If your checker is slow, or flaky, or occasionally blocks legitimate work, you will start bypassing it and eventually stop reading its output. Keep it fast, keep it strictly about facts, and let it fail open with a loud message when the network is down rather than blocking on your own outage.
Third: none of this makes an agent trustworthy in the human sense. It makes its claims cheap to check, which is a different and more achievable thing. The goal is not an agent you believe. It is an agent whose statements cost you two seconds to settle.
The bottom line
Agents report success because success is the likely next sentence, not because they looked. Put something outside the model between the claim and your acceptance of it: a CLI that hits the URL, a hook that refuses to end the turn, an artifact check instead of a summary line. Start with the single most common lie in your stack — for most people it is “it's deployed” — and automate the check for that one thing this afternoon.
Free ground truth for the machine-state half of this problem is whats-running-mcp (MIT, one install command). The verification layer we run — Agent Reliability Kit ($29) — and everything else developer-facing is on our agent tools page.