Rate Limits & Backoff

You send a burst of API calls, one of them comes back with a 429 Too Many Requests, and your first instinct is to just retry immediately. That's the exact wrong move, if the provider is already telling you to slow down, hammering it again a millisecond later doesn't help you and makes the provider's overload problem worse. This lesson is about the retry pattern that actually works: back off, wait longer each time you fail, and add a little randomness so a fleet of retrying clients doesn't all retry in lockstep.

What you'll learn

• Why immediate retries make a rate-limit problem worse, not better
• Exponential backoff with jitter, the pattern every reliable API client uses
• A working retry decorator you can drop into any agent loop

The problem

Every commercial LLM API enforces limits, requests per minute, tokens per minute, to keep the service stable under load. An agent loop making several calls in quick succession (checking a tool result, calling the model again, checking again) hits these limits far more often than a person clicking through a chat UI ever would. When a 429 comes back, retrying instantly is the naive response, and it's actively harmful at scale: if a hundred clients all got rate-limited at the same moment and all retry after exactly the same delay, they all hit the API again at the same instant, a "thundering herd" that keeps the overload going instead of resolving it.

The pattern: exponential backoff with jitter

The fix has two parts. Exponential backoff means each retry waits longer than the last , 1 second, then 2, then 4, then 8 , so a client that keeps failing backs off instead of hammering. Jitter means adding a small random offset to each wait, so that a hundred clients that failed at the same moment don't all retry at the same moment too.

from tenacity import retry, stop_after_attempt, wait_random_exponential

@retry(
    wait=wait_random_exponential(min=1, max=60),  # exponential, randomized between 1s and 60s
    stop=stop_after_attempt(5),                     # give up after 5 tries, don't loop forever
    reraise=True,                                    # let the real error surface if all retries fail
)
def call_model_resiliently(prompt: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text

wait_random_exponential is doing both jobs at once: the wait time roughly doubles each attempt, and it's randomized within that growing window rather than a fixed number, so retries from many clients spread out instead of clustering. stop_after_attempt(5) matters just as much as the backoff itself, an agent that retries forever on a genuinely broken call just burns time and API budget without ever surfacing the problem to you.

What goes wrong

Mistake	How you notice it	The fix
Retrying immediately with no delay	Rate limits get worse under load instead of resolving	Add exponential backoff, never retry instantly on a 429
Fixed delay, no jitter	Many clients retry in sync, causing repeated bursts of failures at regular intervals	Use randomized (jittered) backoff, not a fixed wait time
No retry limit	A permanently broken call retries forever, silently burning time and cost	Always set stop_after_attempt , surface the failure once retries are exhausted, don't hide it
Retrying every kind of error the same way	A genuinely broken prompt (e.g. a malformed request) gets retried 5 times for no reason	Only retry on transient errors (429, 502, 503) , a 400 Bad Request won't succeed no matter how many times you ask

Confirm it worked

To confirm that your retry logic successfully caps attempts and eventually raises an exception rather than hanging or looping indefinitely under persistent failures, run this test harness:

# Simulate exhausting retries and confirm the real error surfaces instead of hanging forever
import time

start = time.time()
try:
    call_model_resiliently_but_always_fails()  # a version pointed at a bad endpoint
except Exception as e:
    elapsed = time.time() - start
    print(f"Failed after {elapsed:.1f}s, should be well under a minute, not indefinite")

If the failure surfaces within a bounded, predictable time instead of hanging or looping forever, the retry pattern is doing its job.

Next: Automating Workflows with Events.