The 404 that moved every deploy

A bulk-enrolment dialog stopped opening. No error, no spinner, nothing — the button simply did not respond. Everything around it worked. It worked in development. It worked against a local preview of the production build. It worked on a colleague's machine.

Then we shipped a fix for something unrelated, and the dialog came back on its own. The search and filter bar stopped working instead.

What it looked like

The symptom was always the same shape, and the shape is what made it hard.

One feature, in one part of the app, doing nothing. Not failing — nothing. The surrounding page was fine. Reloading did not help. And the feature that was broken changed between deploys, which is the detail that sent us looking in the wrong place for a week.

A broken feature that moves reads as a race condition, or as state getting corrupted, or as a caching problem. It does not read as a deployment problem, because deployments are supposed to fail in all-or-nothing ways.

The network tab

One 404. A single JavaScript file:

GET /assets/BulkEnrollCohortDialog-DhoHOVP-.js   404
GET /assets/CohortDetailPage-a7f31c2e.js         200
GET /assets/EnrollStudentsDialog-9b04dd1a.js     200

Every neighbouring file was fine. The file that 404'd was present in the build output on disk, had been uploaded, and was listed in the deployment.

Look at its name.

Cloudflare Pages will not serve a file whose name ends in a hyphen

That is the whole bug.

BulkEnrollCohortDialog-DhoHOVP-.js — note the hyphen immediately before the extension. Every file named like that answers 404 from the edge. Every file that does not is served normally.

Vite hashes chunk filenames so that a changed file gets a new name and can be cached forever. Rollup's default alphabet for those hashes is base64url, which includes - and _ alongside the letters and digits. So roughly one chunk in thirty ends up with a hash whose last character is one of those two.

Which chunk that happens to is decided by the content hash. Change anything in the application and the hashes shuffle, the affected files change, and the broken feature moves. It moves on every deploy, which is exactly what we saw and exactly why we did not believe it was the build.

The fix

One line in each app's Vite config:

build: {
  rollupOptions: {
    output: {
      hashCharacters: "hex",
    },
  },
}

Hex is [0-9a-f]. It cannot produce a hyphen or an underscore, so it cannot produce a filename the edge refuses.

It is in all five applications now, and the comment beside it names the two chunks that actually broke, because a bare hashCharacters: "hex" is the kind of line somebody removes while tidying.

Checking a build for it

Cheap enough to run anywhere:

ls dist/assets | grep -E '[-_]\.(js|css)$'   # must print nothing

If that prints anything, that deployment has a feature which will 404 for every user, and nothing else in the build will tell you.

What we took from it

A symptom that moves between releases is evidence about the build, not against it. We reasoned the opposite way for several days. Application bugs are usually stable; it is the build that is different every time, so a wandering symptom points at the layer that changes on every deploy.

"Works in preview" was a false negative. A local preview server serves whatever is on disk. The file was on disk. Only the edge refused it, so the one environment that could reproduce the problem was the one we could not easily poke at.

Defaults inherit assumptions. Nobody chose base64url; it is Rollup's default, and it is a perfectly good default in most places. It only becomes a bug where it meets a host with a rule about filenames. Two reasonable decisions, made by people who never spoke, producing something that neither of them is wrong about.

That last one keeps coming up. It is the same shape as the chat panel nobody opened, where two syntax highlighters arrived two years apart and neither had any reason to notice the other.