So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.
This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)
BobbyTables2 18 hours ago [-]
I had a former colleague who would put a “retry 10x with sleeps” in each of A, B, C, D.
None of these were even expected to fail. But the code was buggy as well, so after D being retried 10000 times, it’d eventually give up. Managed to convert a sub-second operation into a half hour affair.
prologic 16 hours ago [-]
LOL
nirui 8 hours ago [-]
I think the example is still too generic.
First thing is, if your service has 7 call layers, maybe it's just too deep.
Second thing is, I found that retry strategy works the best if you define it based on what the nodes are actually doing (instead of treating them as generic nodes). For example, if node D is a database failing a transaction, you may just configure it to retry the transaction instead of doing an application-initialized request resubmit, because the database probably knows better about why the transaction has failed than the application connected to it.
Third thing is, retry is worth it only when progress has been and/or can still be made. If the resources is no longer available forever, then there's no point of retrying.
Scoundreller 20 hours ago [-]
Meanwhile Google keeps giving me “please wait, do not reload page” walls, so I ctrl-r as rapidly as possible. Or is that the human test and response?
tgrowazay 16 hours ago [-]
It is anti-bot proof-of-work defence.
mitxela 13 hours ago [-]
When did Google replace their CAPTCHA with Anubis? I've never seen Anubis on Google.
16 hours ago [-]
maxchisto 22 hours ago [-]
I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point
hxtk 18 hours ago [-]
I’d like to mention it since Uber has a really cool load shedder [1], also implemented similarly by Netflix [2] and failsafe-go [3]. It basically looks for points where more requests per second suddenly cause a significant increase in latency and calls that the concurrency limit.
429 (and sometimes 503) errors returned by servers might well be a symptom of intentional load shedding. Perhaps it's just not explicitly called out as a server behavior that induces client retries.
aftbit 22 hours ago [-]
I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
CBLT 22 hours ago [-]
There's a good amount of literature about this (check the other comments), but you can vastly simplify this into two things you need to do:
1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.
2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.
sandeepkd 16 hours ago [-]
I get a feeling of dejavu for this one. Most of my experience has been in Java and in most places I worked in the past we had this hierarchy of exception classification which gets reflected into the http status codes as well. On high level the HTTP status codes in case of errors are already classified as re-tryable or not, the convention varies globally but can be adopted in a standard manner within a company.
The reason why I brought up the exception propagation is cause within a large enough service with multiple layers of depth the exception hierarchy provides with similar context.
The hard part is not implementing something like this, its about maintaining it consistently across every new change. With small product teams this architecture concept/convention/constraint can easily get lost/forgotten and what you are left with is a theoretical system which does not works as desired when the storm comes
otterley 21 hours ago [-]
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Ooh, I like the idea of propagating "no retries" hints in the responses back upstream. Have you seen it implemented in the wild, or in public discussions about the practice?
hxtk 17 hours ago [-]
In gRPC, the statuses it returns in trailers can include arbitrary details, and Google has a well-known proto for common ones in `google/rpc/error_details.proto`. One such detail is RetryInfo [1].
When we implement retries where I work, the general rule is that if a request is suitable for retry, it should include the RetryInfo in the error status and use it as the base delay for the exponential backoff. The absence of that detail means don’t retry, and we have a client interceptor that parses the response status and retries according to that logic.
I've only seen it in bigcorp cross-service typedefs, or in startup's code that re-implements the checks in every service.
jeffbee 17 hours ago [-]
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
This can't really be tolerated in practice, though, because it means that one bad component somewhere in your stack, one that is able to accept and respond to requests but for whatever reason isn't able to make requests to its backends, poisons the whole stack. You can't take one backend's word for it that the failure is not localized and therefore retryable.
So many variables, but the simple thing is to set things up like normal rate limiting (which you would want to do anyways). The one generating the errors passes back a retry time. You can add jitter here, tell low priority requests to wait longer, etc.
BTW: do keep track of priority. It’s like having a database that gets flooded with connections and won’t allow new ones in—but will for admin users (btw, it did not used to be that way in the early days of MySQL).
mandevil 20 hours ago [-]
At $previousJob we implemented circuit breakers: centralize all requests to the foreign service (every call to service theta went through the service theta client which had some shared state so everything so we could keep track of requests) and then monitor, when error % got above a certain limit start to dump requests to a text file for sending in the future instead of now. And the centralized caller will send one message every time gap (we started at 30s) and as long as that errors out we keep writing.
We did that because otherwise we would get 2x30 second timeouts to a dead service on every user interaction and it made for a terrible user experience. Keeping track and handling it smartly made the average user experience a lot better.
Microsoft breaks all Old New Thing links every few years so it's necessary to post the title so the right post can still be found.
applfanboysbgon 22 hours ago [-]
This is trading a good developer experience for a bad user experience. There are situations where it makes sense to force manual retry, but there's no reason to apply one universal rule to all possible situations. Lack of considering nuance for your situation is just intellectual laziness.
anonymars 21 hours ago [-]
My point was that just throwing exponential backoffs at the retry problem is not a magic solution
I don't follow how being cautious about avoiding multiplicative layers of backoffs is trading a good developer experience for a bad user experience. The described situation is an awful user experience. Simply adding a retry and calling it a day sounds like the easy developer experience at the expense of the user experience
applfanboysbgon 21 hours ago [-]
> The described situation is an awful user experience
Sure, the worst case scenario is. 99.9999% of the time, a transient error will actually just work on the first or second auto-retry and save your users the effort of paying attention and manually retrying things. This is especially prudent for background tasks where the failure may not be noticed right away; coming back to something fire-and-forget 30m later to see it never tried to finish is not a good user experience.
> My point was that just throwing exponential backoffs at the retry problem is not a magic solution
Nobody said it was. In fact, I suggested the exact opposite - a proper solution takes dev effort. Adhering to an iron rule of "just make them manually retry" is throwing your hands up and not even trying to solve the problem because laziness is convenient.
anonymars 20 hours ago [-]
> Nobody said it was
I responded to a post that merely linked to the Wikipedia article for exponential backoff (in response to "I'd be interested to hear other strategies in [protecting against retry storms]")
The submitted article is precisely about the degenerate case and the difficult work of dealing with it
> This approach works for transient or low-rate failures. However, during moderate or severe degradation, it becomes counterproductive. Aggressively retrying against an already struggling service increases load, accelerates failure, and amplifies retry traffic across upstream dependencies. What begins as a localized outage can quickly escalate into a stack-wide incident—ultimately degrading, or in the worst case, completely breaking, the end user experience.
applfanboysbgon 18 hours ago [-]
Yes, and then responds to that degenerate case by suggesting that you never automatically retry. It's like saying "you should never drive a car/take a flight/ride public transportation because it's gone wrong so many times". Things go wrong. You should absolutely consider the impact and what will happen when they go wrong, but the end takeaway to just never engage with them because they can go wrong is, frankly speaking, lazy and bad advice.
sroussey 22 hours ago [-]
Yes, exponential back off and jitter are the first things to work on, and good if you don’t have a better signal (like loss of network).
Also, a simple signal status server or queue system helps to keep global state such that everyone doesn’t retry all at once.
If you have a central error rate server you can skip your retry based on the error rate (100% error rate, don’t retry, etc).
konaraddi 20 hours ago [-]
Depending on the context, circuit breakers
jeffbee 17 hours ago [-]
Just limiting your retry budget to 1% of normal rates using a client-local token bucket with no distributed coordination will eliminate the possibility of long-lived retry storms.
cyberax 20 hours ago [-]
One good option that is not (yet?) mentioned here is a deadline for retries. You can cap the request duration by, say, 500ms and pass the remaining time budget to downstream services.
This can be done via an HTTP header and enforced by the middleware.
sisuo30390 6 hours ago [-]
[flagged]
nirmeetimthebes 17 hours ago [-]
[flagged]
sfraxo 22 hours ago [-]
[flagged]
penguin_booze 13 hours ago [-]
It's not related to the article content, but I can't help but notice the "diversity" in authorship grades: a senior staff, a (regular) senior, and a principal. I suppose grades are a key mechanism by which corporates keep their hamster wheels running.
I'm thankful that, at my $WORK, we're all software engineers by title, from the lowly interns to the CTO. Still, everybody knows who's who and whom to talk to, depending on the gravity of the issue.
orsenthil 7 hours ago [-]
It is similar writing an academic paper and getting it published. These blog post reflects on technical strength of the company, so it has to go through the vetting and approval before it gets out of the door, and intermediaries are added as authors.
In academic world, I hear, folks simply add the name of their professors due to their reputation. Of course, professors are putting their reputation at stake by giving a chance to the right students.
orf 12 hours ago [-]
> I'm thankful that, at my $WORK, we're all software engineers by title, from the lowly interns to the CTO
All engineers are equal, however some engineers are more equal than others?
ahoka 10 hours ago [-]
Ah, the classic tyranny of structurelessness.
OJFord 11 hours ago [-]
> at my $WORK, we're all software engineers by title, from the lowly interns to the CTO. Still, everybody knows who's who and whom to talk to, depending on the gravity of the issue.
It's not exactly conducive to knowing that if you're a new starter though, is it. I imagine it takes quite a while to get to know, especially outside of your immediate remit.
tancop 10 hours ago [-]
Positions are better than titles for that. Compare "Principal Engineer" with no other info to "Software Engineer, Lead at Mobile UX".
You know if they are your boss, not how big of a boss they are. That's way more important unless your organization is a chaotic mess where managers feel like they can order around people from a different team. And then you got bigger problems.
OJFord 9 hours ago [-]
Sure but that's just adding more rather than different information really?
Like if the title is Tech Lead, Staff, Principal or whatever there's an (often) unstated focus area. You're saying if you also state that, that's helpful. Yes it is! But if you only state that – Software Engineer, Mobile UX – it's helpful but you're back to having no idea of relative experience or decision-making power.
(If the org is truly flat and the intention is no inequality in 'decision-making power' etc. then fine, I suppose, but that's kind of a different conversation, pretty unusual, and I'm not sure how much I really believe it works (as in, is borne that way) in practice.)
high_na_euv 12 hours ago [-]
Title should indicate work and responsibility, right?
CTOs work is totally different from SE, so why s/he is SE?
Lets make your janitor can be SE too
zyngaro 2 hours ago [-]
TLDR: retry only if the error is owned by the immediate next hop (assuming you can determine ownership).
The engineering is sound, but the article misses a important connection to queueing theory which shows in much simpler terms how retries can increase the instability of a system already under stress.
whatever1 15 hours ago [-]
Easy. Take a larger cut from the driver for each retry.
siscia 12 hours ago [-]
It seems VERY cooperative.
If you can afford that, with all the coordination costs that it comes from it, good.
An alternative is just to let the downstream service own the retry logic. Too many requests? Just error out as soon as possible.
Each team manages its budget and each other team adapts.
UltraSane 19 hours ago [-]
This feels like trying to reinvent Fibre Channel's flow control mechanism.
whoevercares 20 hours ago [-]
Token bucket is all you need
prologic 20 hours ago [-]
Not quite. A token. bucket alone would not prevent a retry storm if you have a chain of services A -> B -> C -> D with a failure in D, you'd end up still having A, B & C all performing retries as they cascade through the chain, albeit (yes) at your configured rate-limits (using token bucket), but you'd still end up with an amplification effect.
This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)
None of these were even expected to fail. But the code was buggy as well, so after D being retried 10000 times, it’d eventually give up. Managed to convert a sub-second operation into a half hour affair.
First thing is, if your service has 7 call layers, maybe it's just too deep.
Second thing is, I found that retry strategy works the best if you define it based on what the nodes are actually doing (instead of treating them as generic nodes). For example, if node D is a database failing a transaction, you may just configure it to retry the transaction instead of doing an application-initialized request resubmit, because the database probably knows better about why the transaction has failed than the application connected to it.
Third thing is, retry is worth it only when progress has been and/or can still be made. If the resources is no longer available forever, then there's no point of retrying.
1: https://www.uber.com/us/en/blog/cinnamon-using-century-old-t...
2: https://github.com/Netflix/concurrency-limits
3: https://failsafe-go.dev/adaptive-limiter/
1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.
2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.
The reason why I brought up the exception propagation is cause within a large enough service with multiple layers of depth the exception hierarchy provides with similar context.
The hard part is not implementing something like this, its about maintaining it consistently across every new change. With small product teams this architecture concept/convention/constraint can easily get lost/forgotten and what you are left with is a theoretical system which does not works as desired when the storm comes
Ooh, I like the idea of propagating "no retries" hints in the responses back upstream. Have you seen it implemented in the wild, or in public discussions about the practice?
When we implement retries where I work, the general rule is that if a request is suitable for retry, it should include the RetryInfo in the error status and use it as the base delay for the exponential backoff. The absence of that detail means don’t retry, and we have a client interceptor that parses the response status and retries according to that logic.
1: https://github.com/googleapis/googleapis/blob/bba4c646b1f85a...
This can't really be tolerated in practice, though, because it means that one bad component somewhere in your stack, one that is able to accept and respond to requests but for whatever reason isn't able to make requests to its backends, poisons the whole stack. You can't take one backend's word for it that the failure is not localized and therefore retryable.
https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/... (originally published 2020, republished 2026)
https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...
https://aws.amazon.com/blogs/developer/announcing-updated-re... (2026)
BTW: do keep track of priority. It’s like having a database that gets flooded with connections and won’t allow new ones in—but will for admin users (btw, it did not used to be that way in the early days of MySQL).
We did that because otherwise we would get 2x30 second timeouts to a dead service on every user interaction and it made for a terrible user experience. Keeping track and handling it smartly made the average user experience a lot better.
Microsoft breaks all Old New Thing links every few years so it's necessary to post the title so the right post can still be found.
I don't follow how being cautious about avoiding multiplicative layers of backoffs is trading a good developer experience for a bad user experience. The described situation is an awful user experience. Simply adding a retry and calling it a day sounds like the easy developer experience at the expense of the user experience
Sure, the worst case scenario is. 99.9999% of the time, a transient error will actually just work on the first or second auto-retry and save your users the effort of paying attention and manually retrying things. This is especially prudent for background tasks where the failure may not be noticed right away; coming back to something fire-and-forget 30m later to see it never tried to finish is not a good user experience.
> My point was that just throwing exponential backoffs at the retry problem is not a magic solution
Nobody said it was. In fact, I suggested the exact opposite - a proper solution takes dev effort. Adhering to an iron rule of "just make them manually retry" is throwing your hands up and not even trying to solve the problem because laziness is convenient.
I responded to a post that merely linked to the Wikipedia article for exponential backoff (in response to "I'd be interested to hear other strategies in [protecting against retry storms]")
The submitted article is precisely about the degenerate case and the difficult work of dealing with it
> This approach works for transient or low-rate failures. However, during moderate or severe degradation, it becomes counterproductive. Aggressively retrying against an already struggling service increases load, accelerates failure, and amplifies retry traffic across upstream dependencies. What begins as a localized outage can quickly escalate into a stack-wide incident—ultimately degrading, or in the worst case, completely breaking, the end user experience.
Also, a simple signal status server or queue system helps to keep global state such that everyone doesn’t retry all at once.
If you have a central error rate server you can skip your retry based on the error rate (100% error rate, don’t retry, etc).
This can be done via an HTTP header and enforced by the middleware.
I'm thankful that, at my $WORK, we're all software engineers by title, from the lowly interns to the CTO. Still, everybody knows who's who and whom to talk to, depending on the gravity of the issue.
In academic world, I hear, folks simply add the name of their professors due to their reputation. Of course, professors are putting their reputation at stake by giving a chance to the right students.
All engineers are equal, however some engineers are more equal than others?
It's not exactly conducive to knowing that if you're a new starter though, is it. I imagine it takes quite a while to get to know, especially outside of your immediate remit.
You know if they are your boss, not how big of a boss they are. That's way more important unless your organization is a chaotic mess where managers feel like they can order around people from a different team. And then you got bigger problems.
Like if the title is Tech Lead, Staff, Principal or whatever there's an (often) unstated focus area. You're saying if you also state that, that's helpful. Yes it is! But if you only state that – Software Engineer, Mobile UX – it's helpful but you're back to having no idea of relative experience or decision-making power.
(If the org is truly flat and the intention is no inequality in 'decision-making power' etc. then fine, I suppose, but that's kind of a different conversation, pretty unusual, and I'm not sure how much I really believe it works (as in, is borne that way) in practice.)
CTOs work is totally different from SE, so why s/he is SE?
Lets make your janitor can be SE too
If you can afford that, with all the coordination costs that it comes from it, good.
An alternative is just to let the downstream service own the retry logic. Too many requests? Just error out as soon as possible.
Each team manages its budget and each other team adapts.
It's not AI that is going to replace you, folks. Btw is it a coincide or has Uber become a family business now?
There are only 2 people on the list with the same surname, and Srivastava is a very common surname.