Skip to content

Written up in October 2026. Dates are when I actually built it.

Reviewing your own code like you’re attacking it is humbling. I did that to my events collector and found a good one.

The bug

/admin/send-digest emails every subscriber, and it sat behind HTTP Basic Auth. The check was shaped like this:

1
2
if cfg.basic_auth_user:
    # verify the Authorization header against configured credentials

Read that again. If BASIC_AUTH_USER isn’t set, the whole check is skipped, and the endpoint is open. Forget one environment variable on a fresh deploy and anyone who finds the URL can send mail to the whole list. That’s the classic “secure if configured” mistake: missing config silently turns into no security.

The fix

Fail closed. If the credentials aren’t configured, return 401 and refuse to run. If they are, require a well-formed Authorization: Basic ... header and compare it against them, with any decode error also meaning 401. Now an unconfigured deployment can’t send anything to anyone, which is exactly what you want from a deployment you forgot to configure.

The rule I took away: a security control that depends on configuration needs a default of “no”, not “skip”.

Months later: the same lesson, but for retries

The scraper base class retried any HTTP error three times with zero backoff. That includes 403, 404 and 410, which are the server saying “never”. So a dead page got hit three times, back to back, every run, for no possible benefit.

I added a NO_RETRY_STATUS_CODES = {403, 404, 410} set that fails on the first attempt. Everything else keeps real exponential backoff (2 ** attempt), and 429s are handled separately.

While in there I documented why three venues return zero events: two are behind a SiteGround “Robot Challenge” and one is behind Cloudflare Turnstile. I noted that priming cookies with Playwright could probably get past it, and decided not to bother: one of them had only ever contributed about two events, and another never got a successful run before the block showed up. Knowing when a fix isn’t worth the complexity is part of the job too, and writing the reason in the code means future me won’t relitigate it.

Both fixes are the same idea: say what happens when things go wrong, out loud, in the code.