Claude wrote my build a quality gate, ran it, and reported it passing. The gate had never looked at the thing it was guarding. Here is the fault injection.

The colleague you don't have

✓ sitemap parity: 198 service x city URLs in the emitted sitemap, all index:true, and all 198 indexedServiceCityPairs() entries present.

That is the build-time guard on my site, reporting success, and the build it belongs to exited 0. I got it to print that line by breaking the site on purpose first. At the moment it printed, 574 pages on disk were carrying <meta name="robots" content="noindex, follow">, including every one of the 198 pages the sitemap had just marked index:true, and that same sitemap was listing 238 URLs asking Google to index them. Nothing was deployed, so nothing broke in production. What the build produced was a dead site, and the guard called it clean.

Claude wrote that guard. Claude ran it. Claude reported it passing, and the report was accurate. The distance between an accurate report and a useful one is what this article is about.

Metallic Media Group is one person and a fleet of Claude Code agents. The SOPs are written as a four-role process covering dev, SEO, content, and PM, and I run all four. Between a green build and production there is nobody. There is no reviewer, no second engineer, and no colleague who looks at a passing check and asks whether anything was actually checked.

The site is a programmatic SEO build. The full service by city matrix is 184 services against 387 cities, which is 71,208 pages, and a new domain cannot get 71,208 thin pages indexed, so it indexes deep instead of wide. A small set of pages ships index, follow and enters the sitemap, and everything else ships noindex. One function, indexedServiceCityPairs(), decides which is which, and the sitemap, generateStaticParams(), and the robots meta tag in lib/seo.ts all read it. When those three disagree, the site either begs Google to index pages that tell it not to, or it orphans the pages it meant to rank.

I asked Claude to build a guard that fails the build when they disagree. It built one that never looked at the robots tag.

What I am not claiming

I have no throughput ratio for you. I am not going to tell you that three people now do the work of twelve, or that output went up by some multiple, or that a client's traffic did anything in particular. Metallic Media Group is early. I do not have those numbers, and I am not going to manufacture them.

I have seen a great many of those numbers published in the last year by people who are in roughly my position, and I notice that almost none of them come with a method. A clean before-and-after multiple is very easy to assert and nearly impossible to check, which is precisely why it gets asserted so often.

What I have instead is a repository, a commit hash, two versions of one script, and the terminal output from a controlled experiment that anyone can rerun in about twenty seconds. Everything below traces to one of those. Where I am extrapolating past what I actually demonstrated, I say so in the sentence where I do it.

The spec is the artifact

The roles in my SOPs are not staffed, they are specified. What a colleague would carry in their head, I carry in a prompt that gets used the same way every time. The prompt that governs data/tiers.ts, which is the file at the center of this entire story, reads in full:

Review data/tiers.ts and audit what is currently indexed:

List every service slug in TIER1_SERVICE_SLUGS

List every city slug in TIER1_CITY_SLUGS

Calculate the current total indexed service/city pages (services × cities)

Show me any TIER1_TARGET_CITY_SLUGS and TIER1_TARGET_SERVICE_SLUGS if they exist

Then propose additions for the following new client markets:

Home market cities: [list cities with state]

Target market cities (limited services only): [list cities with state]

Services to index for home markets: [all Tier 1 services, or specify]

Services to index for target markets: [3 max — typically digital-marketing-agency, seo-agency, and one vertical]

Before applying:

Show the full list of currently-indexed pages so nothing accidentally flips to noindex

Show the projected total page count after changes

Confirm the noindex conditional logic in seo.ts is NOT being changed — only tiers.ts membership lists

Confirm nothing currently indexed becomes noindex

Do not apply until I approve. Do not touch robots.ts.

Read the second half of that again. Every line under "Before applying" is a reject criterion wearing a prompt's clothing. The prompt does not ask for good work. It names the specific ways this operation goes wrong, it demands the evidence that would reveal each one, and it withholds permission to write until that evidence is on the table. A second prompt in the same SOP, the one that audits the GEO layer, ends the same way: "Do not guess at services or contact info, read data/business.ts and data/services.ts first before writing anything," and then, "Show diff and wait for my approval before pushing."

The same posture appears in the hard rules of the onboarding SOP, and the first one is the one that matters here. Real data only. Never fabricate stats, populations, review counts, credentials, or outcome claims. Omit rather than invent. Elsewhere in the same SOP, in the data-build phase, it gets stated again at the field level: omit a field rather than fabricate it.

Hold onto that rule. It is about to reappear somewhere I did not expect it.

The gate that guarded nothing

Claude built the guard. It ran the build against the repository as it stood, with nothing broken and nothing injected. The build exited 0, the guard printed its checkmark, and the work was reported done.

The guard passed. The work was done. Every fact in that report was true, and it is worth being precise about why it was also useless. A passing check on a working system is the one result that carries no information at all. It is what you get from a rigorous check, and it is equally what you get from a check that does nothing.

I did not accept it. Not because I suspected anything specific, and not because I had read the code closely enough to have an objection. I did not accept it because a passing check tells you nothing on its own. A check that passes and a check that cannot fail produce identical output. The only way to tell them apart is to break the code on purpose and watch what the check does. So I asked for the fault injection. Show me it fails.

It did not fail. The guard, as first written, compared the emitted sitemap against indexedServiceCityPairs(), which is the same function the sitemap is generated from. It was comparing the rule to itself. I called it close to a tautology at the time and that is still the right phrase, because it was not entirely vacuous. The first version already carried negative controls, a pair of service and city slugs known to sit outside every tier, plus a check that the predicate returns false for a slug pair that does not exist at all. That is real anti-degeneracy work and it was there from the beginning. What none of it did was look at lib/seo.ts. Nothing in the guard ever read the HTML the build had actually rendered. The robots logic could be deleted outright and the guard would still go green, because the guard was never looking at it.

Claude then rewrote it, adding a fourth assertion that reads the robots meta tag out of the prerendered HTML on disk. That assertion is 42 lines. It is the entire difference between a guard that works and a guard that decorates.

Here is the proof, rerun fresh for this article rather than recalled from the session.

I reconstructed the original version by removing the fourth assertion from the committed guard at b7da191. Then I broke the site in one place. In lib/seo.ts, the robots tag is set by a conditional:

robots: isIndexedServiceCity(service.slug, city.slug) ? INDEX : NOINDEX,

I replaced it with a constant:

robots: NOINDEX,

That is a synthetic break, and I want to be plain about it rather than dress it up. I did not wait for this to happen in the wild. It stands in for a real and unremarkable class of drift: somebody refactors the metadata helper, an agent edits seo.ts while working on something adjacent, a merge resolves badly, and the robots tag quietly stops honouring the predicate. That is the exact failure the guard exists to catch, and it is the reason the file is in the repository at all.

With the break in place and the original guard running, the build exited 0 and printed the line this article opens with.

Look closely at what that line claims, because the guard was not confused and it was not lying. It reported that the 198 URLs in the sitemap matched the 198 entries in the tier data, and they did. That comparison was correct. It was also the comparison of a function against itself, and it had no bearing whatsoever on the 574 pages the build had just written to disk carrying noindex, follow. The guard was accurate about data/tiers.ts and silent about the only artifact Google would ever see. all index:true was a true statement about the rule and a false statement about the site.

Then I changed one thing. Not the break, which I left exactly where it was. I put the fourth assertion back:

  SITEMAP PARITY GUARD FAILED

198 page(s) are in the sitemap and index:true per the tier data,
but the HTML the build rendered ships a non-index robots tag. lib/seo.ts is no
longer honouring isIndexedServiceCity():
  /digital-marketing-agency/jacksonville-fl  ->  "noindex, follow"
  /digital-marketing-agency/miami-fl  ->  "noindex, follow"
  /digital-marketing-agency/tampa-fl  ->  "noindex, follow"
  ...

That listing runs to 198 lines, one for every page the sitemap had marked index:true, and the build exited 1.

Same break. Same repository. Same command. The only variable in the experiment is one assertion, and it is the difference between shipping a dead site and failing the build.

Fabricated data and fabricated verification are the same bug

Now go back to the hard rule. Never fabricate stats, populations, review counts, credentials, or outcome claims. Omit rather than invent.

I wrote that rule to stop a model from inventing a review count when the real one was not available to it. What the guard incident showed me is that the same rule has a second application I had not written down. The model did not invent a statistic. The model invented a verification. It produced a check that had the shape of a check, that ran like a check, that reported like a check, and that could not see the thing it was guarding.

Both are the same underlying behaviour. Producing the shape of an artifact is cheap. Producing the real thing is expensive. Under any pressure toward completion, and a coding agent is under constant pressure toward completion, the shape is the path of least resistance, and from the outside the shape and the real thing are indistinguishable. A fabricated review count looks exactly like a real one until you check the source. A vacuous test looks exactly like a rigorous one until you make it fail.

I want to mark clearly what is demonstrated here and what is not. I have one documented case, reproduced above. The generalisation from "a model will fabricate verification" to "this is a systematic property of agentic coding" is reasoning, not evidence. I have not run a study. I believe it because the incentive structure is obvious once you see it, and because a green build is the shortest available path to "done," but you should treat that paragraph as a hypothesis I am asking you to test against your own work rather than a finding I am reporting.

The practical consequence stands regardless. Adding a second AI pass to review the first does not solve this. A reviewer agent looking at the original guard would very likely have called it correct, because the guard was correct as a comparison. It was just comparing the wrong two things. The reviewer would have read passing code, seen a passing build, and agreed. Vacuous verification survives inspection far more reliably than it survives fault injection. Reading it can catch it, and later in this article I describe a case where reading and reasoning did catch exactly this, in my own work rather than the model's. Breaking the thing the check claims to guard catches it far more often, and it does not depend on anybody being sharp that day.

Making the discipline mechanical

The point of the guard is not that it checks the sitemap. The point is that it converts a habit into infrastructure. "Remember to be sceptical about whether the robots tag survived that refactor" is a thing I have to remember. next build && npm run verify:sitemap is a thing that happens whether I remember or not.

The fourth assertion checks four things against the HTML the build actually emitted. It confirms the page was prerendered at all, and if the file is missing it reports that generateStaticParams() has stopped reading indexedServiceCityPairs(). It confirms the rendered robots tag starts with index, which is the assertion that fired above, and which the code comment describes as closing the loop from data to lib/seo.ts to shipped markup. It treats an absent robots tag as a failure rather than a pass. And then it does the thing I think is the most important line in the file. It checks that it checked something at all:

if (htmlChecked === 0 && notPrerendered.length === 0) {
  errors.push(`Checked zero rendered pages. The guard is broken, not the site.`);
}

That is an assertion whose only purpose is to stop the guard from passing vacuously. The lesson from the incident is now written into the code that the incident produced.

It is not the only place. The function that locates the emitted sitemap refuses to skip when it cannot find its input, and the comment explains why: a guard that skips when it cannot see its input is worse than no guard at all. The comparison step fails when either side comes back empty, on the grounds that this "almost certainly means the guard is broken rather than the site being correct." The guard is written throughout on the assumption that the least trustworthy component in the system is the guard.

That assumption is the whole method. Everything else is implementation.

Where this still fails

The obvious limit is that the guard checks the signals it knows about. It verifies that the robots tag matches the tier data and that the sitemap matches both. It has nothing to say about whether the tier data is right, whether those 198 pages deserve to be indexed, or whether the content on them is any good. A green build is not a quality signal. It is a consistency signal, and consistency is the floor.

The subtler limit is that my own scepticism does not generalise on its own, and I can prove that with something that happened while this article was being written.

I told Claude how to run the fault injections for this piece. My instructions specified two: break lib/seo.ts and show the fixed guard failing, and separately, demote a city called trinity-fl in data/tiers.ts and show the original guard "exiting 0 despite the break." Claude ran the second one and told me there was no break. Demoting a city is a legitimate operation. The sitemap drops the pages, generateStaticParams() stops prerendering them, and the robots tag correctly flips them to noindex, all three moving together, because all three read the same function. Both guard versions pass it. Both guard versions should pass it.

I had written an exhibit that presented a guard working correctly as evidence of a guard failing. Had it gone out that way, this article would have committed, in its own evidence section, the precise sin it exists to condemn. The vacuous-evidence generator on that occasion was not the model. It was me, assembling the exhibits, reaching for a demonstration I already believed in.

There was a second defect in the same instruction, and Claude caught that one too. My plan had run two different breaks against two different guards, the trinity demotion against the original and the lib/seo.ts break against the rewrite, and then invited the reader to compare the results. That is a confounded experiment. If one guard passes one break while a different guard catches a different break, nothing has been isolated and the comparison proves nothing. The clean A and B you read earlier, one break held constant while only the guard changes, exists because that design was refused.

That is the third catch of the same failure mode on this project, and the discipline only worked because it was applied to the person applying the discipline. I do not have a mechanism for that one. There is no build step that fails when the author is credulous about their own argument. The best I have is the same question, asked in the same tone, of my own work: what would it look like if this were wrong, and have I actually looked?

For the record, the trinity-fl case earns its place after all, once it is described accurately. It proves that no edit to tiers.ts can ever fail the first two assertions, because both sides of that comparison derive from the function being edited. The check never verified that the rule was right. It verified that the rule equalled itself.

Take the artifacts

Both versions of the guard, the diff between them, which is a 42-line assertion plus a two-line reporting change, and the terminal logs from all six runs are published at github.com/MetallicMedia/exit-0-is-not-evidence.

The site repository itself stays private, because data/tiers.ts is a map of the markets my clients care about and that is not mine to publish. This means the commit hash, b7da191, is not a checkout target for you. It is an integrity fingerprint for the guard I have published.

So the reproduction I am asking for is not a reproduction of my build. It is the same experiment against yours. Find a check in your own pipeline that you have never watched fail. Break the thing it claims to protect. If it still passes, you have learned something worth more than anything in this article, and it cost you one build.

Exit 0 Is Not Evidence