We shipped a public feature that calls a paid model on behalf of anonymous visitors. The abuse design was layered on purpose: honeypot field, minimum-fill-time, IP throttles, per-person limits, a monthly budget, and, sitting in front of the expensive part, a bot challenge (a CAPTCHA-style widget) that every submit had to pass. The design doc called that challenge mandatory, in bold, and explained why: without it, most of the other layers can be automated or distributed across proxies, leaving the monthly budget as the last backstop between an anonymous script and our API bill.
The challenge was wired through a global feature flag we already had for a few
other forms, CHALLENGE_ENABLED, read from the environment. On the backend the
verifier was a before-action that becomes a no-op when the flag is false. On
the frontend the widget renders only when its build-time twin is true.
A reviewer read the deploy configuration and pointed out what we had not: the
flag defaulted to false. In the production compose file. In the frontend
build. In the deploy script. Everywhere.
So the actual behaviour of an ordinary deploy was: the new route ships, the navigation link to it ships, the model calls work, and the challenge is off. The doc says mandatory. The system says optional, default no.
How this happens
Nothing here was careless in isolation. The flag existed before this feature and defaulted off for a sensible reason: the challenge needs a site key and a secret, and a fresh environment without them should not break every form. Each form that used the flag was one where "off" degraded gracefully to "no challenge, still works". That was fine for a login form behind other protections.
Then we reused the flag for a surface where "off" is not graceful. It is the difference between a free feature and a vending machine that dispenses our API budget. The flag's meaning did not change. The cost of its default did, and nobody re-examined the default because it was already there.
This is the general shape: a global switch whose safe default was chosen for its first consumer, silently inherited by a later consumer with a different risk profile.
The fix: refuse to run without the gate
The right response is not "remember to set the flag" in a launch checklist, though it is on one. Checklists get skipped on the third deploy. The right response is to make the feature unable to run in production without its gate:
before_action :ensure_abuse_gate_active, only: :create
def ensure_abuse_gate_active
return unless Rails.env.production?
return if Challenge.enabled?
Rails.logger.error("[public_scan] REFUSED — challenge is disabled in production")
render_error("This feature is temporarily unavailable.",
status: :service_unavailable, code: "service_unavailable")
Now the failure mode of a mis-configured deploy is a visible outage of one feature, with a log line that names the flag to set. That is a failure mode we can live with. The previous one was an invisible open door with a meter running.
What "mandatory" should mean in code
Security requirements should be invariants, not configuration conventions: if a control is described as mandatory, there should be a place in the code where its absence is an error, not a no-op. A few forms that worked for us:
- A guard on the feature, not just on the control. The challenge verifier no-opping when disabled is correct for the forms it was designed for; the new feature needed its own check that says "I specifically require this."
- Fail closed wherever the endpoint can spend real money, not just where
the environment name says so. We gated on
Rails.env.production?because that is the only environment we deploy this to today; the real invariant is broader — a staging or demo tier wired to real credentials would need the same gate. A boot-time assertion is stronger still when you can afford one; we chose request-time so one feature's missing key can't take the whole API down. - Log the remedy, not just the refusal. The error line should name the env var to set, so the operator at 2 a.m. doesn't need the design doc.
The habit
When you reuse an existing feature flag for something new, re-read its default through the new feature's eyes. Ask: if this defaults the way it does today, and someone deploys without touching it, what runs? For most flags the answer is "a slightly degraded page". For a few it is "an unmetered public endpoint". The flag will not tell you which. You have to look.
We had written the word mandatory. A reviewer read the compose file. The gap between those two took eleven lines to close, and it should have been the first eleven lines we wrote.