A missing 404 page made robots.txt return HTML
Without a 404.html, Cloudflare Pages answers every unknown path with the home page and a 200. Here is what that broke.
The blog’s robots.txt was once a complete HTML document. A crawler asking
for the rules got the home page instead, with a status of 200 OK.
Single-page fallback
Cloudflare Pages has a helpful default. If a site has no 404.html, Pages
assumes it is a single-page application and serves index.html for every
path it cannot find, with a 200, so that client-side routing can take over.
For a static site that is exactly wrong:
- Every typo is a success.
/does-not-existreturns the home page with a 200, so search engines index duplicate copies of it. - Missing files are invisible. If a file you meant to publish did not make it into the build, nothing tells you; the path still “works”.
That second point is how robots.txt became HTML. The file was not in the
build output, so Pages served the fallback, and the fallback looked like
success.
The fix, twice over
- Both the website and the blog ship a real
404.html. With one present, Pages returns it with a 404 status for unknown paths. - Both sites now assert in CI that
404.htmlexists in the build output. The page is small and easy to delete by accident in a refactor, so a missing one fails the build instead of quietly turning every unknown path into a 200.
The lesson
Defaults that are helpful for one kind of site are harmful for another. When a platform does something “for you”, find out what it does when a file is missing, because that is when you will find out whether you agree.
AlzyWelzy
Building PaceStreak. GitHub
Related
InfrastructureWhat the free tier actually buys, checked against the account
The API runs on a GCP e2-micro because that's what Always Free covers - one instance, 30 GB of standard disk, an ephemeral IP. It turns out to be sitting exactly at that ceiling, which explains more of the outage two posts back than any single bug did.
3 min
PostmortemsA CORS error that was actually a masked 500
Login worked, /me failed, and the browser blamed CORS. The real cause was three layers down: Swarm silently ignoring a short-form tmpfs key, which meant Starlette's outermost error middleware - the one layer that runs before CORS headers exist - was the one answering.
4 min
InfrastructureA status page that watches the whole stack, finally
app and api had no DNS records for most of their existence, so there was nothing to monitor - which meant status.pacestreak.com kept quietly not telling the whole story even after both went live. Fixed, plus a lesson in why a 200 from the app proves nothing about the API behind it.
2 min