A CORS error that was actually a masked 500
Login worked, /me failed, and the browser blamed CORS. The real cause was three layers down: Swarm silently ignoring a short-form tmpfs key, which meant Starlette's outermost error middleware - the one layer that runs before CORS headers exist - was the one answering.
The symptom was clean and specific, which made it easy to misdiagnose:
logging in worked, the token came back, and the very next request - GET /v1/me - failed with a CORS error in the browser console. No response body,
no status code visible to the frontend, just Firefox refusing to let the
response through because it claimed to be missing
Access-Control-Allow-Origin.
The instinctive fix is to go look at the CORS configuration. That would have been wrong, and it’s worth writing down why, because the actual chain runs through three unrelated layers and the error message points at none of them.
What a browser actually means by “CORS error”
A CORS error in the console almost always means one of two very different
things: the server genuinely didn’t send the right header, or the server
crashed before it had the chance to send any headers from the app’s own
middleware stack, and the browser’s only honest way to describe “the
response didn’t have what I needed” is to blame CORS. The second case is far
more common in practice than people expect, because in Starlette and
FastAPI, CORSMiddleware is one of several layers wrapped around your
actual route handler - and it is not the outermost one.
ServerErrorMiddleware sits outside every middleware you register,
including CORSMiddleware. It exists specifically to catch anything your
own code didn’t - an unhandled exception in a route, in a dependency, in
another middleware - and turn it into a 500 instead of an unhandled crash.
But because it sits outside CORSMiddleware, a request that dies there
never passes back through the layer that would have added the
Access-Control-Allow-Origin header. The browser sees a response with no
CORS headers and reports exactly that, truthfully, which is why the message
is misleading rather than wrong.
So: /v1/me was throwing an unhandled exception, not failing a CORS check.
What was actually throwing
/v1/me reads push_public_key, which calls into the VAPID key handling in
app/notifications/service.py. That function needs a genuinely writable
temporary directory - tempfile.mkdtemp() - and that call was failing with
FileNotFoundError.
The container itself was configured to be read-only with a tmpfs mount at
/tmp, which is the right way to run it: read_only: true plus a tmpfs
exception is a small, deliberate hardening pattern, not an accident. The bug
was in how that tmpfs mount was declared:
# what was there - silently ignored under Swarm
tmpfs: ["/tmp"]
Docker Compose’s short-form tmpfs: key is valid syntax under plain
docker compose up. It is silently ignored under docker stack deploy -
Swarm’s compose parser accepts the file, accepts the key, and drops it on
the floor without a warning. read_only: true from the same hardening block
still applied, because that key is honored under Swarm. The net result:
a container that’s read-only everywhere, including the one path
(/tmp) an exception was supposed to be carved out for it not to be.
tempfile.mkdtemp() tried to create a directory on a read-only filesystem,
got FileNotFoundError (Python’s error for a failed mkdir in this case, not
the more obvious PermissionError one might expect), and that exception
propagated straight up to ServerErrorMiddleware - outside CORSMiddleware,
so no CORS header ever went out.
The fix is the long-form key, which Swarm does parse correctly:
x-hardening: &hardening
read_only: true
volumes:
- type: tmpfs
target: /tmp
security_opt: ["no-new-privileges:true"]
The chain, end to end
- Swarm ignores
tmpfs: ["/tmp"](short-form), but still appliesread_only: truefrom the same block. /tmpis read-only with no exception, contrary to what the compose file said.tempfile.mkdtemp()inside VAPID key handling fails withFileNotFoundError.- The exception is unhandled at the route level, so it reaches
ServerErrorMiddleware- the one layer that sits outside every user-registered middleware, including CORS. - The response that reaches the browser has no
Access-Control-Allow-Originheader, because the layer that would have added it never got the chance to run. - The browser reports a CORS error, because from its side, that’s exactly what happened - the header really is missing. It’s just not the cause.
Nothing here was a bad guess. Login worked, so the token flow and CORS
config for that endpoint were clearly fine, which made “why would /me
specifically be different” the right question. The answer needed container
logs, not the network tab - the browser was reporting a true symptom of a
cause three layers removed from anything CORS-related.
AlzyWelzy
Building PaceStreak. GitHub
Related
InfrastructureOne outage, four root causes
The API went down behind a Cloudflare Tunnel error and stayed down through a reboot, an interrupted OS upgrade, a stale connection pool, and a box too small for its own workload. None of the four were the same bug.
9 min
Security & PrivacyFixing the dead ends in sign-up and verification
Three real bugs found by using the OTP flow the way a person actually would: a stuck login, a false 'already registered' error, and a confirmation screen that didn't lead anywhere. Plus a status page that finally watches the whole stack.
3 min
Security & PrivacyThe API is live, and deploys itself
The API's hosting decision is made: a free-tier GCP VM, Neon and Upstash, zero-downtime rolling updates on a single-node Swarm - and why the VM pulls its own updates instead of GitHub Actions pushing them.
5 min