Experimentation vs Feature Flagging
When a flag becomes an A/B test, what changes the moment you start measuring, and the three patterns that look like experiments but really aren't.
In code, an A/B test and a feature flag look identical. Both are a conditional that returns one of several values for a given user. The flag system serves both. The dashboards often share a tab.
Operationally, they could not be more different. Conflating them is one of the most common, and most expensive, mistakes a team can make with their flag system.
This chapter is about the line between them, and why it matters.
The single-bit difference
A feature flag is a delivery mechanism. A given variant is the right answer; the flag controls who gets it, when.
An A/B test is a measurement mechanism. You do not yet know which variant is the right answer. The whole point is that the data will tell you.
That difference cascades into everything:
| Feature flag | A/B test | |
|---|---|---|
| Goal | Roll out the right thing safely | Discover which thing is right |
| Lifetime | Until 100% then delete | Until statistical significance |
| Variant assignment | Whatever’s most useful (gradual %) | Random, balanced, locked |
| Exposure tracking | Nice-to-have | Required |
| Statistical math | None | All of it |
| Owner | Engineering | Product / Growth |
| Premature decision cost | Low | Catastrophic |
A flag system that pretends these are the same thing produces “experiments” that aren’t really experiments and “rollouts” that get peeked at as if they were.
What changes the moment you start measuring
The instant a flag has a metric attached, three things become non-negotiable.
1. Locked variant assignment
A rollout can move from 5% to 25% to 100%, that’s the point. An experiment cannot. The moment you start a test, the assignment is frozen until you either ship a variant or kill the test. If you increase the treatment from 5% to 50% midway, your new users are systematically different from your old users (they joined later, in a different season, after marketing changed copy) and your results are garbage.
2. Exposure tracking
You need to log, for every user who evaluated the flag, which variant they got and when. Not “who saw the feature”, who triggered the assignment. The exposure event is what lets you join flag data to outcome data later.
This is also why “did they actually see it?” matters. A user who got variantA but never reached the screen that uses the flag is not a meaningful data point. Good experimentation systems emit exposure events at the point of use, not the point of assignment.
3. A pre-registered hypothesis
What metric will this move, by how much, on what timeline, with what minimum detectable effect? Pick the numbers before the experiment starts. Otherwise you’re stopping the test the day the numbers happen to look good and calling that science.
If this is starting to sound like more work than turning a flag on, yes. That’s the whole point. Experimentation is the expensive version. Use it when the answer actually matters.
Three patterns that look like experiments but aren’t
1. The “rollout disguised as a test”
You’re 99% sure the new feature is better. You set it up as a 50/50 test “for data”. Two weeks later you ship it because the data was positive. Nothing is learned. The data was always going to be positive because the team would’ve shipped either way.
This is fine, it’s just a release flag, but stop calling it an experiment. Save the experimentation tooling for genuine uncertainty.
2. The “vanity A/B test”
A surface-level redesign tested against the old version. The metric is “clicks” or “engagement” with no specific hypothesis. You’ll find a directional result, ship the winner, and discover six months later that you had no idea what the test was actually measuring.
The fix: a real hypothesis with a real metric (“redesign reduces drop-off in the third step of checkout by ≥ 2pp”). If you can’t write that sentence, you don’t have a test.
3. The “infinite experiment”
The test never ends because no variant has reached significance. After three months of waiting, the team makes a decision based on the trend and moves on. The flag stays on. Five more flags like this accumulate. Now you have a config-by-experiment problem and nobody knows what’s “live”.
The fix: every experiment has a kill date. If it hasn’t reached significance by then, pick a default (the variant the PM would have shipped without the test) and turn the test off.
When not to A/B test
Plenty of decisions don’t warrant an experiment, even though you could.
- Anything with strong qualitative signal. Five user interviews will tell you more than three weeks of stats on a feature nobody wants.
- Anything with tiny sample sizes. If you don’t have enough traffic to reach significance in a reasonable time, you’re going to make a decision on noise. Just ship the variant the team prefers.
- Anything involving brand or trust. “Should we A/B test the privacy policy?” No. Some things are decisions, not measurements.
- Anything tied to a customer commitment. If you’ve told an enterprise customer feature X works a certain way, you cannot put 50% of them on a variant that doesn’t.
The bias of any experimentation platform is toward more tests. The discipline is toward fewer, better ones.
The handoff between flag and experiment
In practice, a single piece of behavior often passes through both modes:
- Build the variants. Both behind a flag, default off.
- Internal + beta rollout. Smoke-test the new variant for bugs.
- A/B test. Lock the assignment, fire exposure events, wait for the metric.
- Pick a winner. Set the flag to 100% of the winning variant.
- Delete the losing variant from code. Delete the flag.
Notice that 2 and 3 are separate phases. The internal rollout finds bugs; the A/B test measures impact. Doing them simultaneously corrupts both.
If your flag platform forces you to do these as one thing, to “experiment” by gradually ramping a variant, you have the wrong platform. The right platform supports both modes and is clear about which one a given flag is in.
The mental model
Treat experiments as statistical objects with deadlines. Treat feature flags as operational levers without deadlines (well, with a different kind of deadline, see Chapter 6). They share infrastructure. They do not share rules.
When in doubt: are you trying to deliver something, or learn something? If you can answer that question, you know which tool to reach for.
Previous: Chapter 4, Kill switches and incident response · Next: Chapter 6, Managing flag debt