The config was mine and the fix on paper was still wrong

A delivery and configuration audit asks a narrow question: does the running system behave the way its configuration says it does? The interesting answer is usually no, and the reason this class of fault survives is that nothing reports it. There is no error, no failed build and no alert. The config is readable and looks correct. The site returns 200.

The worked examples below are this site, audited on 16 August 2026. They are used here instead of a client engagement for one reason: the findings are that the auditor was wrong, three times in one day, and that is not something worth demonstrating with somebody else's system. The config was mine, I had written it, and reading it still produced a remedy that would have caused the fault it was meant to prevent — then two of the fixes for that shipped with the same defect in a different form.

What reading the config concluded

This site sets HTTP headers in two places: netlify.toml and a _headers file. The _headers file states three times, in comments, that it deliberately mirrors netlify.toml — the two agree so that whichever one wins a conflict, the private pages stay uncached and unindexed.

They had drifted. netlify.toml carried rules caching every .js and .css file for a year; _headers had no equivalent. So the audit wrote the obvious remediation item: add the missing rules to _headers so the two files agree again. It was filed with a severity, a file list, and a note about why it mattered.

Every part of that reasoning is defensible from the source. It is also wrong.

What probing the live system showed

Three requests, against production rather than the repository:

The service workers were already correct. Their own more specific rules were winning over the broad one, exactly as intended. There was no defect. The two files had diverged and the divergence was harmless, which is a state a config diff cannot distinguish from the other kind.

Why the written remedy was the dangerous part

Adding a blanket .js rule to the second file would have brought both service workers into scope of a year-long immutable cache.

A service worker cached for a year cannot be fixed by deploying. The browser keeps running the copy it already has, so the next deploy does not replace it, and neither does the one after that. The bug would have outlived every subsequent fix, on the two pages where it is least visible, and the symptom — a page that works for new visitors and is stale for returning ones — is the kind that gets attributed to the user's browser.

So the audit item, written from a correct reading of the configuration, would have manufactured an outage-class fault in a system that was working. The only thing that separated those two outcomes was making three requests.

What the finding actually was

Something real was underneath it, and it was smaller and duller than the item claimed: nothing in either file recorded why the asymmetry existed. The mirroring comments were scoped to the private-page blocks, and asset caching was never mirrored on purpose — but that had never been written down, so the next person to read the two files would reach the same conclusion and might act on it.

The fix was documentation, in both files, naming the service-worker hazard so the tempting change is refused by whoever considers it next. No rule moved. Every changed line begins with a comment marker.

The general shape

This is not an argument that configuration review is useless. It found the asymmetry; nothing else would have. It is an argument about ordering:

• Config review finds candidates. It cannot grade them. “These two files disagree” is a fact about the files. Whether that disagreement reaches production is a fact about the running system, and the two are not related by inspection.
• A remediation item is a proposed change, and proposed changes need testing too. An audit that ships a backlog without probing has shipped untested changes with a delay fuse. They are more dangerous than ordinary untested changes, because they arrive with the authority of a review attached.
• The failure is invisible from inside the audit. Nothing in reading the config told me the item was wrong. It read as one of the clearer findings in the set.

The same pass found three faults the live system did confirm: a form posting to an endpoint that had never existed and returned 404 on every submission, a published figure that a generated file was faithfully reproducing from a stale source, and a script cached for a year with no version token, so a fix would have reached first-time visitors only. Those were real. The one that was written most confidently was the one that was wrong.

Two more, on the same day, and they are not the same kind

Writing the fixes produced two further instances. Taken together with the one above they separate into two species, and the distinction is the practical part — because it decides which check would have caught them.

Species one: the file says one thing and does another.

• A rule that never ran. Replacing the name list with something class-based produced /*.md and /package*.json. Netlify ignores both: a splat may only end a path segment, so a splat followed by literal characters matches nothing. No error, no warning. The rules deployed, read correctly in the file, and the documents kept returning 200.
• A fail-open path that only worked when nothing was wrong. The gate that replaced them was written to fail open, because a gate on every request that fails closed is a sitewide outage caused by its own bug. Its catch block called the downstream chain again — the exact call that had just thrown — so the second exception escaped anyway. It read as fail-open and was not.

The first of those is the more dangerous, and the reason is review. A reader sees /package*.json, recognises the intent, and stops. The pattern is legible, the intent is correct, and nothing about it invites a second look; it is a lie in the shape of a fix. The fail-open bug at least required someone to follow the call to notice, so tracing it was already an act of suspicion. A defect that survives careful reading is worse than one that merely survives casual reading.

Species two: the file is honest, the reading is correct, and the prescription is wrong. That is the header case at the top of this page. Nothing in that configuration misrepresented itself. The remedy was derived correctly from an accurate reading and would still have caused an outage.

This is why the distinction matters rather than being a taxonomy for its own sake:

• Species one is caught by knowing the platform. Someone who knows Netlify's splat rule sees the dead glob immediately. Someone who traces the exception path sees the fail-open bug. Both are, in principle, catchable from the source — by a sufficiently expert reader.
• Species two cannot be caught by reading at all. Reading was never the failing step. No degree of care applied to the document reaches it, because the document was right. It is only reachable by leaving the document and asking the running system.

An audit that only reads will find some of species one and none of species two. Which means the expertise that makes a reviewer good at the first kind is exactly what makes the second kind invisible: the better the reading, the more confident the wrong prescription.

Both of the day's species-one defects were caught the same way — by requesting the paths afterwards, and by a test written to drive the gate before it deployed. Neither was caught by re-reading the file that contained it, and both were written by the same person who had just written the page arguing that reading is not enough.

What a live probe cannot tell you

Stated plainly, because an audit method that only lists its strengths is the thing this page is arguing against:

• It sees the current deploy, not intent. A header that is correct today because of a rule someone added by accident still reads as correct.
• It cannot see what a crawler or a cache holds. What Google fetched last week is answerable only from that provider's own tooling.
• It cannot prove a negative about paths it did not request. Coverage is a list you have to build deliberately.
• Probing a production system is itself an action. Anything that writes — a form submission, a test enquiry — leaves records that have to be cleaned up, and that has to be planned before the first request rather than after.

How this gets run

Config first, to build the candidate list and know what the system claims. Then requests against the running system for every claim that can be checked with one. Then the two are compared, and only findings that survive both become items. Anything that cannot be settled from outside is written down as unverified with the specific check that would settle it, rather than being graded on likelihood.

And then the same treatment for the fix. That is the part this day added. A remedy is a claim about behaviour like any other, so it gets probed after deploying rather than assumed from the diff — and where a check can be written instead, the check gets deliberately broken first to prove it fails, because a check that passes for the wrong reason is another file saying one thing and doing another. Both species-one defects here were found that way and neither was found by re-reading.

The full report from this pass is published, including the corrections trail and the section recording that the remedy above was wrong. Most of its verdicts are refutations, which is the ordinary outcome of checking claims rather than collecting them.

Read the full audit report →