When you ship a CRUD feature behind a flag, a 1% rollout means 1% of users see the new button. When you ship an LLM feature behind a flag, a 1% rollout means, what, exactly?

1% of users see Opus 4.7 instead of Sonnet 4.6? But what happens when they reload the page mid-conversation and re-bucket into the other variant? What if the new model is four times more expensive? What if it’s slower, and now 1% of your users are staring at a spinner for eight seconds?

AI features are different. They’re stochastic. They cost real money per call. They’re latency-sensitive at the user-perceived level. And they’re stateful at the conversation level in ways most product features aren’t. The standard “ramp the flag from 1% to 100%” playbook doesn’t translate cleanly. You need a different set of patterns.

This post covers three of them, model swaps, prompt rollouts, and tool gating, and why each one needs its own approach.

Three rollouts hiding inside one AI feature

When teams talk about “rolling out an AI feature,” they usually mean one of three things, and they conflate them at their peril.

  1. Model swaps. Same prompt, same tools, different underlying model. Sonnet 4.5 → Sonnet 4.6, in-house fine-tune → frontier model.
  2. Prompt updates. Same model, different system prompt, instructions, or few-shot examples. By volume, the most common kind of change in production AI systems, and the one with the highest defect rate.
  3. Tool and capability rollouts. Same model, same prompt, but the agent now has access to a new function (search, database query, side-effecting action).

These look similar from the outside, they all change what the model produces, but they have very different operational characteristics. Treat them all the same and you’ll get bitten by whichever one’s failure mode you ignored.

Pattern 1: model swaps need sticky bucketing

A user opens your assistant, sends three messages, refreshes the page, sends a fourth. If your flag re-buckets on every request, that fourth message can land on a different model than the previous three. The model has no memory of the prior context, you’re feeding it the transcript regardless, but its style, tone, refusal behavior, and capability profile shift mid-conversation. Users notice.

The fix is sticky bucketing: the flag’s value is derived from a stable identifier, hashed deterministically, so the same user (or the same conversation) always resolves to the same bucket for the lifetime of the rollout.

const variant = await client.getFlag('llm-model', {
  userId: user.id,
  conversationId: conv.id,
}, { default: 'sonnet-4-6' });

const model = {
  'sonnet-4-6': 'claude-sonnet-4-6',
  'opus-4-7':   'claude-opus-4-7',
}[variant];

The bucketing key matters. For chat assistants, conversationId is usually the right granularity, let a single conversation stay on one model from start to finish, but allow the next conversation to bucket fresh. For multi-session agents, use userId.

Sticky bucketing also matters for caching. Prompt caches, KV-stored eval results, response caches, their hit rates collapse if a user flip-flops between variants. If you’ve spent any effort tuning cache hit rate, sticky bucketing is the difference between keeping that work and silently throwing it away.

Pattern 2: prompt changes need shadow evaluation

Prompt changes are where most production AI regressions come from. A reasonable-looking edit to the system prompt can quietly change tool-calling rates, refusal behavior, or output format in ways that don’t surface until thousands of calls later, when a downstream parser starts erroring or a user complaint trickles in.

Standard percentage rollouts don’t catch this well. By the time you’re at 10%, you’ve shipped a regression to a meaningful slice of users.

The pattern that works is shadow evaluation: run the new prompt in parallel with the old one, return the old one’s output to the user, and log both for offline comparison.

const shadow = await client.getFlag('shadow-prompt-v3', context);
const live = await llm.complete(promptV2, input);

if (shadow) {
  // fire-and-forget, never block the user response
  void llm.complete(promptV3, input).then(shadowOutput => {
    logShadowComparison({ live, shadow: shadowOutput, input });
  });
}

return live;

The shadow path costs 2× tokens for whatever percentage is shadowing, but it lets you build an eval set from real production traffic without any user-visible risk. After 10,000 shadow comparisons you have empirical data on whether the new prompt is better, worse, or substantively unchanged, before a single user sees it.

Pair this with offline evals (golden set plus LLM-as-judge) and you have two independent signals before flipping the prompt live. Skip the shadow phase and you’re flying blind.

Pattern 3: tool rollouts need capability gating

When you give an agent a new tool, say, a search_internal_docs function, you’ve expanded the surface area in two directions: the agent now has new things it can do, and new things it can do wrong.

Tool rollouts behave less like prompt changes and more like permission grants. The right pattern is per-tool, per-tier capability gating:

  • Internal users get the tool first
  • Then power users on a paid tier
  • Then a percentage rollout gated on guardrails, rate limits, usage caps, audit logging
  • Then everyone
const tools = [
  ...baseTools,
  ...(await flagOn('tool-search-docs', ctx) ? [searchDocs] : []),
  ...(await flagOn('tool-write-file',  ctx) ? [writeFile]  : []),
  ...(await flagOn('tool-send-email',  ctx) ? [sendEmail]  : []),
];

const response = await llm.complete({ ...promptArgs, tools });

One subtle but critical point: the flag isn’t gating whether the tool gets executed when called. It’s gating whether the tool is in the list passed to the model in the first place. If you only enforce at the execution layer, a hallucinated tool name (or one leaked from a previous turn) can still trigger a real call. Capability gating has to happen at the boundary, before the prompt is constructed, not after the model decides to invoke something.

The cost dimension nobody talks about

Most feature-flag advice glosses over cost because most features cost roughly the same to serve to one user as to another. AI features don’t.

A 1% rollout of Opus 4.7 over Sonnet 4.6 is not a 1% cost increase. Depending on token mix and price ratios, it might be a 4–6% cost increase, concentrated in your highest-traffic users if your bucketing accidentally over-samples them.

Two practical implications:

  • Rollout percentages should be priced before they’re flipped. Multiply daily token volume by the price ratio of the variants. If the answer is “we just doubled spend,” that needs to be a deliberate decision, not a side effect of a rollout dial.
  • Watch hot users, not just user counts. A handful of power users can dominate token consumption. Bucket on userId for stickiness, but track aggregate cost per bucket, not just user count per bucket, so you don’t get surprised when 1% of users turn out to be 30% of the bill.

A flag system that lets you express segment rules (“only users on the Pro tier with conversation count under 100/day”) is more useful here than one that only does percentage splits.

Latency budgets are tighter, not looser

A typical product feature can absorb a 5ms flag check without anyone noticing. An LLM feature is already burning 800ms to 3s on time-to-first-token, so 5ms sounds like a rounding error.

It’s the opposite. Every millisecond above the perceived-instant threshold compounds with the streaming latency users are already tolerating. The flag check is your cheapest operation by far in the call stack, anything that pushes it onto a network round-trip changes that. Edge-evaluated flags (no round-trip on the hot path) matter more for AI features, not less.

Eval-driven rollouts, not metrics-driven

For non-AI features, you ramp up while watching ops metrics, error rate, p99 latency, conversion. For AI features, those metrics often look fine while output quality silently degrades. The model is still returning 200s. Latency is unchanged. Users just trust it less.

Roll out AI changes against an evaluation set, not just against ops dashboards. Define the bar before you flip the flag:

  • A scored eval set of representative prompts with expected behaviors
  • Output diff metrics from shadow-evaluation traffic
  • Human spot-checks on a sample of live outputs
  • Optional: LLM-as-judge scoring on output pairs

If the eval bar isn’t met, the rollout doesn’t advance, even if your error rate is 0.0%. The flag dashboard and the eval dashboard need to be looked at together, not separately.

Kill switches still apply, with one wrinkle

Last week’s piece on kill switches still applies, with one important adjustment: when you flip an AI feature off, in-flight conversations see the change at the next turn, not the current one. Design the off-state to be graceful, fall back to the previous model or prompt rather than erroring, and make sure tools you’ve removed degrade cleanly when the agent tries to call them after the flag flips. A tool_not_available response is fine; a 500 from the tool layer is not.

The same primitive, different patterns

The TL;DR: AI rollouts are ordinary feature rollouts plus three new dimensions, statefulness, real cost per call, and quality-without-errors. The flag system you already have can do all of it. You just need different patterns: stickiness for models, shadow evaluation for prompts, capability gating for tools.

Use the right one for the right kind of change and you’ll catch regressions before users do, without paying for it twice.

ShipSilently flags evaluate locally with sub-millisecond latency, support sticky bucketing on any attribute, and let you ship variants of any shape, including model identifiers and prompt templates as JSON values. Try it free.