The hard part is not retrying. It is knowing when not to.
Highlights
- A call that fails is normal rather than exceptional, and the useful question is which failures are worth trying again.
- Status codes do not tell you whether to retry, since a 500 caused by your own malformed request fails identically every time.
- Retrying that 500 five times with backoff produced five identical errors and fifteen seconds of waiting.
- Sixty clients backing off on the same schedule took 63 seconds to clear, because they all arrived together at every step.
- Adding one line of randomness to the same backoff cleared all sixty in one second.
- A 429 often tells you exactly how long to wait, and reading that is better than any formula you would pick.
- Streaming breaks the usual timeout, since a response can take minutes while never actually stalling.
Every LLM API call fails sometimes. Rate limits. Timeouts. A provider having a bad afternoon.
The usual answer is to retry. That is right, and it is also where two expensive mistakes live. Retry the wrong error and you burn time on something that will never succeed. Retry on a fixed schedule and every client comes back at the same instant.
In this article we measures both against a real API, then builds the handling that avoids them.
What you need before you start
Everything here runs with Python and nothing else. A KodeKloud playground or your own machine both work.
Get a key
KodeKey gives you one key reaching Claude, GPT and Gemini through a single OpenAI compatible endpoint.
export KODEKEY_API_KEY="your key here"Set up the project
mkdir retries && cd retries
python3 -m venv .venv
source .venv/bin/activateNothing to install, since urllib and random are both in the standard library. You will build four files.
Step 1: Find out what your provider returns
Create probe.py. Every provider words its errors differently, so knowing yours beats reading a general guide.
import json
import os
import urllib.error
import urllib.request
KEY = os.environ["KODEKEY_API_KEY"]
URL = "https://api.ai.kodekloud.com/v1/chat/completions"
GOOD = {"model": "claude-haiku-4-5-20251001",
"messages": [{"role": "user", "content": "hi"}], "max_tokens": 3}
def call(payload, key=KEY, timeout=30):
request = urllib.request.Request(
URL, data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {key}"},
)
try:
urllib.request.urlopen(request, timeout=timeout).read()
return 200, "ok", {}
except urllib.error.HTTPError as exc:
return exc.code, exc.read().decode()[:160], dict(exc.headers)
except Exception as exc:
return None, f"{type(exc).__name__}: {exc}", {}
if __name__ == "__main__":
cases = [
("valid request", dict(GOOD)),
("model that does not exist", {**GOOD, "model": "not-a-real-model"}),
("malformed body", {"model": "claude-haiku-4-5-20251001"}),
]
for name, payload in cases:
code, body, headers = call(payload)
retry_after = headers.get("retry-after", "none")
print(f" {name:28s} {str(code):5s} retry-after: {retry_after:6s} {body[:70]}")
code, body, _ = call(GOOD, key="sk-obviously-wrong")
print(f" {'bad key':28s} {str(code):5s} {body[:70]}")python3 probe.pyNotice the function returns the headers as well as the code. That is the whole reason to write this rather than read documentation. A Retry-After header is worth more than any backoff formula you would invent.
Reading what came back
Four different problems, four different codes, and only one of them is worth retrying.
The 401 is a bad key. Retrying cannot fix it, and the same request will fail identically in an hour.
A 403 says the model is not available on this plan. Also permanent, and also not something waiting will change.
That 500 is the interesting one, and the section below is about it.
A 429 is missing from that list because it needs load to trigger. It is also the one failure everybody plans for. It means you are sending faster than your allowance, and it is genuinely temporary.
Step 2: The 500 that retrying cannot fix
Standard advice says retry on 5xx. Server errors are the provider's problem, and usually transient.
That advice sent a malformed request five times. Create retry_500.py to see what that costs.
import time
from probe import call
BROKEN = {"model": "claude-haiku-4-5-20251001"} # no messages field
waited = 0
for attempt in range(5):
code, body, _ = call(BROKEN)
print(f" attempt {attempt + 1}: {code} after waiting {waited}s in total")
if code == 200:
break
delay = 2 ** attempt
time.sleep(delay)
waited += delaypython3 retry_500.pyFive attempts, five identical 500s, and fifteen seconds of waiting before the last one. The request was missing a required field, so the server failed while handling it. No amount of patience adds a missing field.
The lesson is not that 5xx errors are never retryable. Most are. It is that a status code tells you where the failure happened, not whether it will happen again.
Row two is where people lose time. A 500 from a provider outage clears on its own. A 500 from your bad payload does not, and both look identical from outside.
Watch out
A 500 that repeats identically is your bug, not theirs, and retrying it is the slowest possible way to find out. The practical test is cheap. If the same request fails the same way three times in a row, stop retrying and log the request body. Retry budgets should be spent on failures that might clear, and a malformed payload is not one of them. Capping attempts at three rather than five or ten is partly about this, since the difference between attempt four and attempt ten is almost never a success.
Step 3: Watch a retry storm happen
Now the mistake that only shows up under load, which is why almost nobody catches it in testing.
Exponential backoff means waiting one second, then two, then four. Every client follows that same schedule, and that is the problem. When a rate limit hits a fleet at once, they all wait one second. They all retry together. They all get rejected again. Then they all wait two seconds.
Create storm.py.
import random
CLIENTS = 60
CAPACITY = 10 # requests the provider accepts in any one moment
MAX_ATTEMPTS = 8
def simulate(jitter, seed=7):
random.seed(seed)
waiting = list(range(CLIENTS))
finished_at = {}
clock = 0.0
for attempt in range(MAX_ATTEMPTS):
base = 2 ** attempt
arrivals = {}
for client in waiting:
delay = base * random.random() if jitter else base
arrivals.setdefault(round(clock + delay, 2), []).append(client)
waiting = []
for moment in sorted(arrivals):
group = arrivals[moment]
for client in group[:CAPACITY]:
finished_at[client] = moment
waiting += group[CAPACITY:]
clock += base
if not waiting:
break
return finished_at
print(f" {CLIENTS} clients rate limited at once, provider accepts {CAPACITY} at a time\n")
print(f" {'strategy':26s}{'all served by':>15s}{'still waiting':>15s}")
for name, jitter in [("backoff, no jitter", False), ("backoff with jitter", True)]:
done = simulate(jitter)
latest = max(done.values())
print(f" {name:26s}{latest:>13.1f}s{CLIENTS - len(done):>14}")python3 storm.pySixty three seconds against one. Same backoff, same clients, same provider capacity. The only difference is one multiplication by a random number.
Without jitter the clients are synchronised, and synchronised clients waste every attempt after the first ten. Each round doubles the wait. By the sixth round the unlucky clients have waited a minute for a service that was never busy for more than a moment.
With jitter they spread out. The provider sees a trickle rather than a wall, and everyone is served on the first retry.
Quick tip
The randomness is one line, and where you put it matters. Multiplying the whole delay by a random number between 0 and 1, which is called full jitter, spreads clients across the entire window. Adding a small random amount to a fixed delay does not, since everyone still lands in roughly the same place. If you are going to write time.sleep(2 attempt), write time.sleep(2 attempt * random.random()) instead. It is the same line and it is the difference above.
Step 4: Timeouts, and what streaming does to them
A timeout answers a question the other two do not. Retries handle a call that failed. Timeouts handle a call that is neither failing nor finishing.
Without one, a hung connection ties up a thread until something else gives up. On a request path that means a user watching a spinner for as long as your framework allows.
Setting one is a single argument.
urllib.request.urlopen(request, timeout=30)The number is the interesting part. Too short and you kill requests that were about to succeed. That turns a slow response into a failed one, then into a retry that costs you again. Too long and a hung call blocks for minutes.
Two things should shape it. How long your longest legitimate response takes, which you can measure rather than guess. And how long the person waiting will tolerate, which is usually shorter.
Streaming changes the question entirely. A streamed response can run two minutes and be perfectly healthy, arriving a token at a time. A total timeout would kill it. What you want is a gap timeout, measuring time since the last chunk rather than since the start.
Those are starting points rather than settings. Measure your own slowest successful call and leave headroom above it.
In the wild
A timeout that fires and then retries can double your bill without you noticing. The provider may have generated the whole response and been billing for it while your client gave up waiting. You then retry, generate it again, and pay twice for one answer. This is worth knowing before you set an aggressive timeout on an expensive model. If you see cost rising faster than traffic, look at your timeout and retry counts before you look at anything else.
Step 5: Put it together
Create client.py. Everything above in one wrapper.
import json
import os
import random
import time
import urllib.error
import urllib.request
URL = "https://api.ai.kodekloud.com/v1/chat/completions"
KEY = os.environ["KODEKEY_API_KEY"]
RETRYABLE = {429, 500, 502, 503, 504}
MAX_ATTEMPTS = 3
TIMEOUT = 30
class Failed(Exception):
"""Raised when a call failed and retrying will not help."""
def call(messages, model="claude-haiku-4-5-20251001", max_tokens=200):
payload = json.dumps({"model": model, "messages": messages,
"max_tokens": max_tokens}).encode()
last_body = ""
for attempt in range(MAX_ATTEMPTS):
request = urllib.request.Request(
URL, data=payload,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {KEY}"},
)
try:
with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
return json.loads(response.read())
except urllib.error.HTTPError as exc:
last_body = exc.read().decode()[:200]
if exc.code not in RETRYABLE:
raise Failed(f"{exc.code}, not worth retrying: {last_body}")
wait = retry_delay(exc.headers, attempt)
except (TimeoutError, urllib.error.URLError) as exc:
last_body = f"{type(exc).__name__}: {exc}"
wait = retry_delay({}, attempt)
if attempt == MAX_ATTEMPTS - 1:
break
print(f" attempt {attempt + 1} failed, waiting {wait:.1f}s")
time.sleep(wait)
raise Failed(f"gave up after {MAX_ATTEMPTS} attempts: {last_body}")
def retry_delay(headers, attempt):
"""Do what the server asked, and otherwise back off with full jitter."""
told = headers.get("retry-after") or headers.get("Retry-After")
if told:
try:
return float(told)
except ValueError:
pass
return (2 ** attempt) * random.random()
if __name__ == "__main__":
reply = call([{"role": "user", "content": "Name three reasons a pod restarts."}])
print(reply["choices"][0]["message"]["content"])python3 client.pyFour decisions in there are worth naming, because each one came from something above.
RETRYABLE is an explicit set rather than a range check on 5xx. Everything outside it raises at once, so a bad key fails in milliseconds instead of after three rounds of waiting.
retry_delay reads Retry-After before falling back to a formula. When a server tells you how long to wait, it knows something you do not. Ignoring it is how a rate limit lasts longer than it needed to.
The jitter is random.random() applied to the whole delay rather than added to it, which is the difference measured in step 2.
And MAX_ATTEMPTS is 3. Attempt ten almost never succeeds when the first nine failed. By then you have spent a minute learning what attempt three already told you.
What this does not cover
Three things sit just beyond a retry wrapper, and knowing they exist is worth more than implementing them early.
A retry budget caps retries across your whole application rather than per call. Without one, an outage means every request retrying three times, tripling your load on a provider already struggling.
Circuit breakers stop calling entirely after repeated failures, then probe occasionally to see whether things recovered. That turns a slow cascade into a fast failure, which is usually what you want when something is genuinely down.
And idempotency matters more than it looks. A timeout leaves you unsure whether the request completed. Retrying a read is free. Retrying something that writes to your database is not, so wrap the retry around the call rather than the whole operation.
What you should be able to answer now
Which errors should you retry? A 429 and most gateway errors. Not 400, 401, 403 or 404. And a 500 only until it repeats identically, since that means the fault is in your request.
Why does backoff need jitter? Because identical schedules synchronise clients. Sixty clients took 63 seconds to clear without it and one second with it.
What should you do before falling back to a formula? Read Retry-After. The server knows how long the limit lasts and you are guessing.
Why is streaming different? Because a healthy streamed response can take minutes. Time the gap between chunks rather than the whole call.
The measurement worth keeping is the second one. The retry logic was identical in both runs. One multiplication separated a service that recovers instantly from one that looks down for a minute.
Ready to Build the Rest of the Pipeline?
Reliable calls are one layer of getting an AI feature into production. The AI Agents Fundamentals course covers tokens, embeddings, retrieval, orchestration and MCP through an end to end project, and the AI Learning Path sequences it alongside vector databases and agents. Start by finding out what your own provider returns when it fails.
FAQs
Q1: Which LLM API errors should I retry?
A 429 always. It means you are over your rate limit, and that is temporary by definition. Gateway errors like 502, 503 and 504 too, since they usually mean something upstream is briefly unhappy. Do not retry 400, 401, 403 or 404. A malformed request fails the same way however long you wait. So does a bad key, a permission problem, or a wrong model name. The awkward case is 500. Most are worth one or two attempts, since server errors are often transient. But measured on a real gateway, a 500 from a missing required field returned the same error five times, across fifteen seconds of backoff. So retry a 500 until it repeats identically, then stop and log the request body. At that point the fault is yours.
Q2: What is jitter and why does exponential backoff need it?
Jitter is randomness added to the wait between retries. Without it, backoff synchronises your clients. Every client waits exactly one second, then exactly two. They all arrive together, and only the first few get through each time. Simulated with sixty clients against a provider accepting ten at a time, identical backoff took 63 seconds to serve everyone. The same clients with jitter were all served in one second. The change is one line of code. Instead of time.sleep(2 ** attempt), write time.sleep(2 ** attempt * random.random()). That form is called full jitter, and it spreads clients across the whole window. Adding a small random amount to a fixed delay does much less, since everyone still lands in roughly the same place.
Q3: How long should my timeout be?
Long enough for your slowest legitimate response, and shorter than your users will tolerate. You find that by measuring rather than guessing. Around 30 seconds suits short completions and 120 seconds suits long generation. Treat those as starting points and check your own slowest successful call. Too short is worse than it looks. Killing a request that was about to succeed turns a slow answer into a failure, then into a retry that costs you again. The provider may already have generated the whole response and be billing for it while your client gave up. Streaming changes the question completely. A streamed response can run for minutes while perfectly healthy. So time the gap between chunks instead of the whole call, with around 15 seconds of silence as the limit.
Q4: What do I need to build this?
Python and an API key. Everything here uses the standard library, so urllib, random and time cover all of it. Nothing to install beyond a virtual environment. For the key, KodeKey gives you one that reaches Claude, GPT and Gemini through a single OpenAI compatible endpoint. The most useful first step is not writing code at all. Ask your own provider to fail in several ways and record what comes back. Every provider words its errors differently, and the headers matter as much as the status code. A Retry-After header is worth more than any backoff formula you would invent. You only find out whether yours sends one by looking.
Q5: Should I use a library rather than writing my own retry logic?
For anything beyond a simple wrapper, yes. Libraries like Tenacity handle the fiddly parts well. Most official SDKs retry on your behalf already, which is worth checking before you add a layer on top and end up with nine attempts where you meant three. The reason to write it once yourself is to understand what those defaults do. An SDK retrying three times with backoff is making choices for you. Which errors are worth retrying. How long to wait. Whether to read Retry-After. Those choices are what we measure. Once you know what you want, take the library. The wrapper here is about sixty lines, and it exists to make the decisions visible rather than to compete with anything.
Q6: What else do I need beyond retries?
Three things, in rough order of when they start to matter. A retry budget caps retries across your whole application rather than per call. That matters during an outage. Every request retrying three times triples your load on a provider already in trouble. A circuit breaker stops calling entirely after repeated failures, then probes occasionally to see whether things recovered. That turns a slow cascade into a fast failure. And idempotency deserves thought, since a timeout leaves you unsure whether the request completed. Retrying a read costs nothing. Retrying something that writes to your database can duplicate work, so wrap the retry around the API call rather than the whole operation. None of these are needed on day one. All three become obvious the first time a provider has a bad hour.
Discussion