Kill Switches and Incident Response

Most of this Academy is about how feature flags help you ship faster. This chapter is about how they help you stop, quickly, calmly, and without a deploy, when something goes wrong.

A kill switch is a flag wrapped around a piece of behavior that can fail in ways your code can’t recover from on its own. Third-party APIs. Expensive AI calls. Recommendations that suddenly look like nonsense. Anything where “turn that off, now” is a sentence you’d rather not say while typing into a CI/CD console at 2am.

What deserves a kill switch?

Not everything. The temptation, after you’ve built your first one and felt how good it feels to flip a switch instead of revert a commit, is to wrap everything in flags. That’s the path to the flag-debt problem from Chapter 6. The discipline is to wrap only the things that can plausibly need to be turned off, fast, with no notice.

A working filter:

• External dependencies that can degrade unexpectedly. Payment providers, email services, SMS, third-party search, embeddings APIs, LLM calls. Anything where their bad day becomes your bad day.
• Expensive code paths. A new recommendations engine that’s beautiful at 1% CPU and unaffordable at 100%. A feature that hits the DB three times instead of once.
• Features that interact with money. Anything that creates, modifies, or refunds a charge. Anything that issues a coupon. You want a single switch you can flip if the coupon math is wrong.
• AI-shaped features. Especially anything generative. The blast radius of “the model started saying weird things” is real. Feature flags for AI rollouts goes deeper.

A working anti-filter:

• Internal-only refactors. If userService.findById now uses a different cache, you probably don’t need a flag. You need tests.
• Pure UI changes that don’t talk to anything. A color change is not a kill-switch candidate.
• Anything where “off” is meaningless. A login flow can’t be turned off. Don’t pretend otherwise.

The shape of a good kill switch

A kill switch is not just if (flag) { newPath() } else { oldPath() }. The “off” branch has to actually be a viable production behavior, on its own, indefinitely. That sounds obvious. It is routinely violated.

Off must be safe

If the kill switch’s “off” position calls a service that’s already broken, you don’t have a kill switch, you have a second failure mode. The “off” branch should rely on the smallest, most boring, longest-proven code path you have. Often that means keeping the old code around for the duration of the flag’s life.

Yes, even after the new feature is at 100%. The kill switch is the contract that says: this is the safe path. Delete it the day you delete the flag, not before.

Off must be cheap

A 200ms eval on the kill-switch flag during an incident is enough to make engineers reach for the env-var override. The flag SDK has to be local-eval, cached, and resilient to the flag service itself being slow. If the only fast path is “set an env var and redeploy”, you don’t have a kill switch, you have a slow rollback.

Off must be reachable by the right person

The on-call engineer at 3am needs:

• A direct link to the flag in the dashboard.
• Permission to flip it without a peer review.
• Confidence the flip will take effect in seconds.
• A clear answer to “is this the right switch?”, which means good naming.

This last point is the most underrated. A kill switch named feature_v2_enabled will, in a moment of stress, be confused with feature_v2_beta_enabled and v2_feature_kill and disable_feature_v2_legacy. Pick a naming convention and enforce it:

ops.<system>.<verb>

So: ops.payments.disable_apple_pay, ops.search.disable_personalization, ops.ai.disable_summarizer. Easy to grep, hard to misread under stress.

Permanent flags need permanent owners

A release flag dies when the feature ships. A kill switch never dies. That means it needs an owner, a team or person whose name is on the flag, who answers when someone says “is this safe to flip?”, and who reviews the flag the same way they’d review a service in their on-call rotation.

The audit log of a kill switch is part of your incident review. Who flipped it. When. Why. The flag system should record this without anyone having to remember to. (And the on-call engineer should not have to type a justification before flipping, that’s friction at exactly the wrong moment. Record the flip; ask for the why later.)

The kill-switch drill

Once a quarter, run the lever. Pick a kill switch, flip it during business hours, observe what happens, flip it back. Two things almost always come out of this:

1. The dashboard link is stale or hard to find. (Fix that.)
2. The “off” branch has bit-rotted. Someone refactored something. (Fix that.)

You do not want to discover either of those at 3am. The blog post Kill switches and incident response has a longer write-up of the drill format.

The control loop you actually want

The fully-formed control loop is:

1. Monitoring detects an anomaly.
2. On-call engineer identifies the affected feature.
3. Engineer flips the kill switch in the dashboard.
4. SDK on every server picks up the change within seconds (push, not poll).
5. Anomaly disappears.
6. Engineer investigates root cause without time pressure.
7. Fix lands, flag is flipped back on.

If your flag system can’t do steps 4 and 5 in under ten seconds, your incident response is bottlenecked on the flag system rather than on the human. That’s a bug. Fix it before you trust the kill switches.

What this buys you

Engineers stop dreading on-call. Product stops fearing the launch. The CTO stops calculating MTTR by stopwatch. The number on the incident retro, “time from detection to mitigation”, collapses from “a deploy” to “a click”.

That single change is most of why feature flag systems exist in mature companies. The release flags are nice. The kill switches are non-negotiable.

---

Previous: Chapter 3, Targeting and rollouts  ·  Next: Chapter 5, Experimentation vs feature flagging