A team ships a redesigned onboarding flow behind a flag, sets it to 50%, watches the signup dashboard for a day, and decides the number “feels” better. They flip it to 100% and tell the next standup they “A/B tested it.” They didn’t. They ran a rollout with an opinion attached. The infrastructure was identical to a real experiment; the rigor wasn’t.

This mix-up is common because feature flags and A/B tests genuinely do share a mechanism: a percentage of traffic gets bucketed into one path or another, based on a rule your SDK evaluates at request time. Same primitive, same dashboard, same evaluate() call. But the two things layered on top of that primitive are answering different questions, and confusing them is how teams end up making irreversible calls on data that was never designed to support them.

Same switch, two different jobs

A feature flag is a deployment control. Its question is narrow: does this break anything? You wrap new code in a condition, expose it to a small slice of traffic, and watch for errors, latency regressions, or support tickets. If something goes wrong, you flip the flag off. Nothing to revert, and no waiting on a build while production is on fire.

An A/B test is a measurement instrument. Its question is different: is this actually better? You split comparable users into groups, expose each to a different version, and use statistics (not a Tuesday-afternoon glance at a dashboard) to determine whether a metric moved because of the change or because of noise.

Flags ask about safety. Experiments ask about truth. A rollout percentage can serve either purpose, but it can’t serve both at once without you being explicit about which one you’re running.

Where the confusion comes from

The confusion isn’t accidental. It’s structural. Both patterns use:

  • The same targeting rules (percentage-based, attribute-based, or both)
  • The same SDK call site in your code
  • The same-looking dashboard, showing a metric split by variant

But the two use cases diverge sharply the moment something looks off. A canary rollout is deliberately biased toward speed: the instant error rates tick up, you kill it, because fast asymmetric response to bad signal is the entire point. An experiment requires the opposite discipline. Random assignment has to stay consistent for the life of the test (the same user sees the same variant every session, which is what sticky bucketing is for), and you can’t kill the losing variant the moment it looks bad.

Peeking at results early and stopping when the number you want appears is one of the most common ways experiments produce false positives. A test that gets killed on day one because “it’s clearly losing” was never a test. It was a rollout wearing a lab coat.

If you don’t decide up front which mode you’re in, you’ll default to canary behavior, because that’s what your instincts and your on-call training both tell you to do, and then call the result an experiment after the fact.

What each one can’t do

A flag can’t tell you whether a change is good. It can only tell you whether it’s stable. Teams that treat “we rolled it out to 50% and nothing broke” as evidence the feature is an improvement are answering a question nobody asked. Stability is necessary for a launch; it says nothing about impact.

An experiment, meanwhile, isn’t built for kill-switch speed. Proper experiments need a pre-registered sample size, a defined success metric, and the patience to let the test run to completion. Bolting “let’s just watch it for a few hours” onto a real experiment doesn’t make it faster. It makes it underpowered. If you need instant rollback semantics, that’s a flag concern, and it should be handled by the deployment layer, not by cutting an experiment short.

The pattern that works: flag for safety, instrument for truth

The teams that avoid this trap don’t pick one tool over the other. They sequence both, deliberately, for the same change.

  1. Flag the change behind a kill switch first. Ship the new onboarding flow to 5% of traffic under a descriptively named flag (enable-new-onboarding-flow, not experiment-1 or flag-284). Watch error rate, latency, and crash reports for a day or two. This step has nothing to do with whether the flow is better. It’s asking whether it’s safe.
  2. Once it’s stable, run the actual experiment. Allocate a proper randomized split with sticky bucketing, define the metric you’re trying to move (signup completion rate, not “vibes”), calculate the sample size you need before you start, and don’t touch it until you hit that number.
  3. Ship the winner through the same flag. Once the experiment concludes, the flag that protected the rollout is also the mechanism that ships the result to everyone, with no separate deploy.

The flag and the experiment aren’t competing tools fighting for the same slot in your stack. They’re sequential phases of the same change, and the reason it feels like one tool when done well is that good feature flag infrastructure supports both phases without you having to bolt on a second system.

What phase one and phase two actually look like

The call site doesn’t change between the two phases. Only what you do with the result does:

const variant = shipSilently.evaluate("enable-new-onboarding-flow", { userId });

if (variant === "on") {
  // Phase one (safety): just watch error rate and latency.
  // Phase two (truth): also log the variant to your analytics
  // pipeline against the same userId, so the experiment can
  // attribute signup completions back to the assignment.
  renderNewOnboarding();
}

Nothing about the flag definition changes when you move from “is this safe” to “is this better.” What changes is whether a second system (your event pipeline, not your flag SDK) is recording the assignment against an outcome metric. That’s the tell for whether you’re running a rollout or an experiment: if nobody downstream is logging variant-to-outcome, there’s no experiment happening, no matter how the traffic is split.

A decision framework for your next rollout

Before you reach for either tool, answer one question honestly: what are you actually trying to find out?

  • “Will this break something?” → You need a flag. You don’t need a hypothesis, a control group, or a sample size. You need a fast off switch and someone watching a dashboard.
  • “Which version performs better?” → You need a flag and an experiment. The flag handles the safe rollout; the experiment tracking handles the measurement. Skipping the second half and reading the first half’s dashboard as if it answers the second half’s question is the mistake this whole post is about.
  • “I don’t have a metric or a hypothesis, but I want to see what happens” → You don’t have an experiment. You have an opinion with a rollout percentage attached, and that’s fine. Just don’t present it as data.

Most teams, most of the time, only need the first case. Reserve the second for changes where being wrong is expensive enough to justify the discipline a real experiment demands. The flag protects the deploy; the experiment settles the argument. If you can’t say out loud which of the two you’re running right now, you’re running the first one.

For more on drawing sharper lines around what a flag actually is (and isn’t), see Not Everything Is a Feature Flag and Progressive Rollouts: How to Ship Without the Big Bang.

ShipSilently provides sub-millisecond feature flag evaluation at the edge. Get started for free.