Why AI Automations Break in Production: Six Failure Modes From Our Own Log

Your automation ran fine for weeks. Then a customer mentioned they never got the email, or you noticed the dashboard had been showing Tuesday's numbers since Friday, and the uncomfortable question arrived: how long has this been broken, and what else is? We run a one-person company on AI agents and automations, and we log every failure in an append-only file — 83 entries at the time of writing. Read enough of them in one sitting and a pattern emerges: production automations almost never fail loudly. They fail while reporting success. Here are the six shapes that keep recurring in our own log, each with the real incident and the check that would have caught it.

1. Success theater: the automation reports work it did not do

The worst entry in our log, by damage per line of code: an email-capture endpoint on this very site shipped with the philosophy “never block the reader.” Somewhere in implementation, that became “always return success.” The endpoint answered {ok: true} whether or not it had actually stored the address — and since no email provider was configured on launch day, it never had. For three weeks, every single person who typed their email into that form saw a success screen, and every single address was silently discarded. The UI was happy. The logs were quiet. The failure was invisible from both sides of the glass.

This is the signature failure of AI-built automations specifically, because a coding agent optimizes for the request you made — “handle errors gracefully” — and graceful error handling, naively implemented, is indistinguishable from lying. The subtle version shows up everywhere: the scraper that returns an empty list instead of raising, the sync job that catches the exception and moves on, the poster that counts an HTTP 200 as “posted.”

An automation that cannot fail loudly will eventually fail silently, and silence in production reads as success.

The check:verify writes by reading them back from the destination system, never by the endpoint's own response. After the fix, our capture flow is only allowed to say “saved” when a storage failure would have made the response itself fail — and the deploy test reads the address back out of the provider. If your automation claims it wrote something, your test is a read, on a different rail than the write.

2. The surface you verified is not the surface users see

Twice in one week, our log records the same shape at two different companies' edges. First: a production deploy of this site reported Ready, Production, done — and the live domain kept serving a 44-minute-old build. New pages returned 200 on the deployment's internal URL and 404 on the domain customers actually visit. The platform even confirmed, when asked to promote the build, that it was “already the current production deployment.” It was — and the domain alias was still pinned to the old one.

Second: documentation fixes were committed, pushed, and visible on GitHub — while the package registry, where effectively all of the real traffic lands, served the week-old copy. The publish step had silently stopped working (an expired auth token that produced no local error), so the repo and the registry disagreed for days. Everything we checked said done. Everything users touched said otherwise.

The check: verify on the surface of record — the URL a stranger would type, the registry page a stranger would install from — with a cache-buster, and grep for the actual changed string, not for a status code. Our rule now has a second half that matters just as much: also grep for a fabricated control string and confirm it is absent. If your check cannot fail, it is not a check. We wrote more about making an agent prove its own claims in Your Coding Agent Says It's Done. It Isn't.

3. Data rot: the input aged and nothing noticed

An automation is only as current as its most stale input, and stale inputs do not announce themselves. Our log has a painful one: a selection job that picked “newest items” from an exported catalog snapshot. The export parsed cleanly, had plausible dates, and looked current. It was four months old — every record newer than its last refresh simply did not exist as far as the automation was concerned, so the “newest” picks were silently four months behind, every run, with zero errors. Worse, on inspection its date column turned out to be the upload date, not the release date — so even the rows it had were sorted by the wrong clock.

The check:every snapshot file an automation reads needs either a refresh step at the top of the run or an explicit staleness gate — “abort if the newest row is older than N days.” Ours now re-derives the manifest from the source system as step one, and the refresh script refuses to proceed if the new data is somehow older or smaller than the old. A file that parses is not a file that is true.

4. Resurrection: the thing you killed comes back

At automation scale you will eventually kill a job — and discover that something else was keeping it alive. Our log's most dramatic entry: we disabled a set of background jobs, confirmed they were gone, and moved on. A watchdog cron — installed months earlier, forgotten — noticed the “missing” jobs every two minutes and helpfully re-registered all of them. The machine ground down to 137 MB of free disk and a load average of 208 before the real cause surfaced: not the jobs, the healer.

The check:when decommissioning any automated process, hunt for its supervisors first — cron entries, watchdogs, “keep-alive” scripts, restart policies — and disable those before the process. Then make resurrection structurally impossible (rename or delete the job definition so a re-register fails), and re-verify after a delay longer than the healer's interval. “It's gone right now” and “it stays gone” are different claims.

5. Self-expiring truth: copy that becomes a lie on a schedule

Automations do not just perform actions — they publish claims, and some claims carry an expiry date. We once shipped “price ends tonight at midnight” on a product page, with the price change left as a manual step for later that night. The machine that would have done it is deliberately not always on; the step never ran; and the countdown copy sat on a live product page, false, for 21 days. Nobody wrote a bug. The system did exactly what it was configured to do, which was nothing.

The check: no time-bound promise goes live without its enforcement shipping in the same change — the price change scheduled and verified, or the copy written without a deadline. Our publish gate now fails any deliverable containing time-bound language unless the enforcement receipt is attached. If your stack has a human step between a promise and its fulfillment, the promise is already broken; you just have not hit the date yet.

6. The audit that grades against its own denominator

The most dangerous failures survive becauseyou checked. We once audited every buy button on our site: all eight routed to the correct checkout pages, test passed, surface declared healthy. The store itself had nine products. The audit's denominator came from the site's own product list — so the one product missing from the site was invisible to the audit by construction, and it happened to be the cheapest, lowest-friction item on the roster. Eight of eight was true. It was also the wrong question.

Agents make this failure constantly when asked to verify their own work, because they inherit the same worldview that produced the bug. The fix is structural, not motivational:

• Pull the denominator from the other system.Audit the site against the platform's live product list, the mailing list against the provider's export, the cron jobs against the OS — then diff. The finding lives in the set difference.
• Run a negative control.Feed the check something fabricated and confirm it fails. We have logged multiple incidents where a “verification” returned identical success for real and made-up inputs — which means it had verified nothing.
• Separate the checker from the doer. The process (or agent) that performed the work never gets to be the only one confirming it.

The practice underneath all six: write the failures down

The individual checks matter, but the compounding asset is the log itself. Ours is one append-only markdown file. Every entry has the same anatomy: date, what happened, what it cost, and the rule it produced. Three properties make it work:

1. 1It is append-only. Nobody tidies it, nobody deletes embarrassing entries, and old entries keep earning — several of the incidents above were caught the second time they almost happened, by an agent that read the log first.
2. 2Agents read it before similar work. The file is wired into the instructions of every agent we run: before touching a deploy, a store surface, or a data export, read the entries about deploys, store surfaces, and data exports. A failure log nobody consults is a diary; one that gates work is infrastructure.
3. 3Every entry must end in a rule.“The deploy didn't propagate” is a complaint. “Never claim a site change is live without fetching the custom domain and grepping for the changed string” is a rule an agent can execute. If you cannot phrase the lesson as an instruction, you have not finished the incident.

This costs about four minutes per failure and it is the closest thing we have found to compound interest on pain. The alternative is paying full price for the same incident twice.

If yours already broke: the 20-minute triage

Working backwards from the six modes, in the order that finds the most damage fastest:

1. 1Establish the blast window. Find the last provably good output — not the last success message, the last output a human confirmed — and assume everything since is suspect. In our email-capture incident the success messages ran three weeks past the last real save.
2. 2Check the surface of record, not your dashboard. Load what a stranger loads. If customers see something different from what your tooling reports, you are in failure mode 2 and your tooling is part of the incident.
3. 3Date every input. For each file, export, or feed the automation reads, find the newest record and ask whether that date is plausible. A clean parse of stale data is mode 3, and it never throws.
4. 4List the supervisors.What restarts, retries, or “heals” this system? Each one is a way your fix gets silently reverted.
5. 5Only then read the code. Most production automation failures we have logged were not code bugs — they were wrong assumptions about state, surfaces, and time. The code usually did what it said.

The bottom line

Production is where your automation meets the two things no demo contains: time and other systems. Time rots inputs, expires promises, and resurrects the dead; other systems cache, alias, and lie politely with 200s. None of the six failures above announced themselves, and all six would have been caught by the same discipline — verify on the surface of record, read your writes back, control your checks with fabricated inputs, and write every miss into a log your agents are forced to read. The operation that does this does not have fewer incidents. It has fewer second incidents — and at automation scale, that is the entire difference between compounding and cleaning up. For how the whole system fits together day to day, start with how one person runs a whole company on AI agents.