On July 10, 2026, LaunchDarkly had a bad day. Three overlapping incidents hit their public status page in a single morning: mobile and server SDKs intermittently down (from 9:00am PDT), the LaunchDarkly UI down with flag evaluation errors (9:16–10:48), and the web application unavailable with elevated flag-delivery failure rates (10:10–11:19).
Bad days happen to every vendor, and they’ll happen to us. This one’s worth writing up because of what their customers had to do about it.
Here is the remediation from LaunchDarkly’s own status page, in their words:
Affected SDKs emit messages—varying by language—that contain any of the following: “giving up permanently,” “Invalid SDK key,” “unauthorized,” “not authorized,” or “401.” Any service that logged these messages may be unable to receive flag updates or send event data until it is restarted.
Here are the recommended actions for each affected application:
- If you use server-side SDKs, please restart the application to re-establish its connection to LaunchDarkly.
- If you use the Relay Proxy, please restart the Relay Proxy to re-establish connections. It should not be necessary to restart the applications behind it. …
- If this does not resolve the issue for an affected application (e.g. you still see the log messages from above), please log a support ticket and we will follow up with you directly to assist with remediation.
So the vendor’s control plane returned spurious 401s for a while. The SDKs running inside customers’ production services concluded the credentials were dead, logged “giving up permanently,” and stopped trying. Once the incident ended, none of them came back. The fix was for every affected customer to restart their own fleets in the right order (Relay Proxy first, then the applications behind it if needed) and to open a support ticket if that didn’t do it.
Their incident, and every affected customer got handed the cleanup.
”Giving up permanently” is a design decision
Let’s steelman it, because the engineers who wrote that behavior weren’t stupid.
The reasoning goes: a 401 means the SDK key is invalid. Keys become invalid when someone revokes them, and a revoked key will never work again, so retrying is pointless. It burns requests and hammers an auth service with traffic that can’t succeed. Better to fail fast and log loudly. For a key that really has been revoked, that logic holds up.
The problem is the premise: the SDK cannot actually know why it got a 401.
From inside a customer’s process, these two situations are byte-for-byte identical:
- An admin revoked this SDK key an hour ago. It will 401 forever.
- The vendor’s control plane is having an incident and rejecting valid credentials. It will 401 for the next eleven minutes.
Same status code, same response body, no header that says “this one’s on us.” And situation 2 isn’t exotic. Key rotation produces it routinely: you rotate a credential, and for a window bounded by somebody’s cache TTL, some servers still reject the new key. We know because we have that window ourselves. Our API-key validation is cached for up to five minutes, so a freshly rotated ShipSilently key can be rejected by a perfectly healthy control plane. Every distributed auth system has a window like this somewhere.
So the real question isn’t “should an SDK retry a revoked key?” It’s: when you can’t distinguish a permanent failure from a transient one, which mistake do you choose?
If you treat 401 as transient and you’re wrong, a service with a genuinely revoked key sends one small, jittered request a minute until someone fixes the config. If you treat 401 as permanent and you’re wrong, a vendor’s control-plane blip disconnects every SDK in every customer’s fleet, and the remediation section of the status page starts with the word “restart.”
One of those mistakes costs a few bytes. The other pages your customers’ on-call engineers to clean up your incident. On July 10, an entire customer base got the second one.
The policy: no terminal states
We rebuilt our SDKs’ connection handling around a single rule: there is no failure the SDK won’t try to recover from. There is no “giving up permanently” branch in the codebase to reach. Every error a ShipSilently SDK can encounter falls into one of four classes:
| Error | Class | Behavior |
|---|---|---|
| Network error, 5xx, 429 | Transient | Retry with backoff |
| 401 / 403 | Transient until proven otherwise | Same backoff. Never fatal. |
| 402 (streaming not in plan) | Feature gate | Poll instead; don’t retry a door that’s closed on purpose |
| 404 (flag doesn’t exist) | Definitive answer | Return your default. No retry. |
The retry loop itself is the boring, well-known shape, full-jitter exponential backoff:
attempt = 0
loop forever:
connect()
if healthy for 30s: attempt = 0 // recovered systems get fast reconnects
on failure:
start polling // flag data stays fresh THROUGHOUT
delay = random(0, min(60s, 1s × 2^attempt))
attempt += 1
sleep(delay), then try again // no attempt limitA few things in that loop do the real work of making restarts unnecessary.
The SDK polls while it reconnects, so losing the stream never means losing freshness. It drops to interval polling immediately, keeps attempting the stream in the background, and stops polling once the stream is back. Your kill switches work the entire time.
The cache survives everything. In-memory flag state is replaced only by a successful response; a 401 can’t clear it, and neither can a week of network partition. The worst case during any outage is stale flags, never no flags, and never a process that needs a human before it can evaluate again.
Dead connections can’t hide, because our servers send a heartbeat event every 30 seconds. An SDK that hears nothing for 90 assumes the socket was silently dropped by some middlebox and reconnects on its own. That catches the streams that die without the courtesy of an error event.
And the backoff is jittered because when a provider recovers from an incident, every disconnected SDK on earth tries to come back at the same moment. Full jitter spreads that stampede across the backoff window instead of turning recovery into a second outage. Repeated failures log once per minute per failure class, so the retry loop doesn’t relocate the incident into your logging bill.
All of this shipped this week in the Node, React, and Go SDKs. The 401-then-recover scenario, the exact shape of the July 10 incident, is a named test case in the Node and Go suites (React inherits the Node client unchanged). If our control plane has a terrible morning and rejects your valid key for ten minutes, your remediation checklist is: nothing. Your services keep serving last-known-good flags, reconnect with backoff, and log one warning line a minute until they quietly recover.
What we’re not claiming
House rules: we tell you where the comparison cuts against us.
- LaunchDarkly ships 25+ SDKs, including mobile. We ship three today (Node, React, Go), and more are on the way. If the one you need isn’t there yet and you’re considering a switch, email us. SDK ordering on our roadmap gets decided by exactly these emails.
- They have a decade of scale behind them and a mature experimentation platform. We’re a small team with a much narrower product.
- Retrying a genuinely revoked key does waste a request a minute per service. We think trading a trickle of doomed requests for never having to restart a fleet is obviously correct, but it is a trade.
- And the honest meta-point: we will have incidents too. The claim is not “we won’t go down.” The claim is that when we do, recovery is the SDK’s job, not yours, and that’s now enforced by tests rather than by a paragraph on a status page.
The part where the invoice makes it worse
Design flaws sting more at enterprise prices. Vendr’s purchasing data puts the median LaunchDarkly contract at roughly $72,000 a year, built from per-service-connection fees and per-MAU metering (full breakdown, with sources). At that price, “please restart your applications, Relay Proxy first, and file a ticket if that doesn’t help” is a remarkable sentence to ship to customers.
ShipSilently is $49 a month, flat. Unlimited seats, no connection metering, and SDKs whose failure-recovery plan doesn’t include you.
If July 10 had your team restarting fleets, the migration guide covers the SDK swap. Most teams finish in under an hour, and flags run in parallel while you switch. We also keep a plain-English log of LaunchDarkly’s recent incident history on our LaunchDarkly status page, including this one.
Sources
- LaunchDarkly’s public status page, status.launchdarkly.com: incident titles, time windows, and remediation text for July 10, 2026 (“Mobile and Server SDKs are intermittently down,” from 9:00am PDT; “LaunchDarkly UI is down and flag evaluation errors are occurring,” 9:16–10:48; “Web application unavailable and flag delivery evaluations have elevated failure rate,” 10:10–11:19). Quotes are verbatim from the 16:32 PDT update; screenshots captured July 10–11, 2026. All times PDT as shown on their page.
- The flag delivery and event ingest incident LaunchDarkly linked from its own UI-incident update: status.launchdarkly.com/incidents/chywwz01ptb0.
- Contract pricing: Vendr purchasing data (median ≈ $72k/yr across 196 purchases), compiled with dates and source links on our LaunchDarkly comparison page.
- A capture of their May–July 2026 incident history, with per-incident notes, lives on our LaunchDarkly status tracker.
ShipSilently flags evaluate at the edge in under 5ms and recover from anything without a restart. Try it free.