Calling AI APIs Reliably

Every model provider's API looks trivial in a "hello world" example and gets a lot less trivial the first time your agent is making dozens of calls a minute across a real workload. Connections drop, providers throttle you, responses take longer than you budgeted for, none of that shows up in a tutorial, and all of it shows up in production. This lesson covers the patterns that keep an agent's API calls actually working under real, sustained use.

What you'll learn

• The real cost of a network round-trip, and why connection reuse matters more than people expect
• Time-to-first-token vs. throughput, two different numbers that both affect how "fast" an agent feels
• Picking a provider under real rate-limit and latency constraints, not just on paper specs

The problem

An agent loop calls a model API far more often than a person clicking through a chat interface ever would, checking a tool result, feeding it back, calling again. Every one of those calls pays a real cost before the model even starts generating: establishing a connection, sending the request, waiting for the provider's queue. Ignore that cost and your agent feels sluggish even when the model itself is fast. Ignore rate limits and your agent starts failing under exactly the load you built it to handle.

What actually costs you time and money

Connection overhead. Every new HTTPS connection costs roughly 200-300ms in handshake time before your request even reaches the model. An agent making many calls in a tight loop that opens a fresh connection each time is paying that tax over and over, reusing a connection (connection pooling, which most SDKs do by default if you reuse the client object) can meaningfully cut total execution time in a busy loop.

Time-to-first-token vs. throughput. These are two different numbers and both matter, for different reasons. Time-to-first-token (TTFT) is how long you wait before anything starts coming back, it's what determines whether an agent feels responsive. Throughput is how fast tokens keep flowing once generation starts. A larger, more capable model tends to have higher TTFT (more to compute before the first token) but doesn't necessarily have lower throughput once it gets going, the two aren't the same tradeoff.

Context size and cost. A longer context window costs more and, past a certain size, risks the "lost in the middle" problem covered in Prompting & Context Engineering , sending more context isn't free and isn't automatically better.

Picking a provider

Provider	Typical rate limits	Strengths	Watch out for
Google Gemini	Generous on Flash-tier models, tighter on Pro-tier	Large context windows, strong price-to-performance on Flash	Pro-tier models have noticeably higher TTFT on complex prompts
Anthropic Claude	Stricter by default, scales with usage tier	Strong reasoning and tool-use reliability	Rate limits can be tight for a new account under bursty load
OpenAI	Variable by model and tier	Broad tool-calling ecosystem support	Cost rises quickly with tool-calling-heavy workloads

None of these numbers stay fixed for long, check current published limits before committing to one provider's numbers as a design constraint, and build your retry logic (see Rate Limits & Backoff) so a provider's specific limits are absorbed rather than assumed.

Build it: a resilient call, wired to a real SDK

To make API calls robust against intermittent network drops and provider transient errors, wrap the Google Generative AI SDK calls with Tenacity's retry helper using exponential backoff:

import os
import google.generativeai as genai
from tenacity import retry, stop_after_attempt, wait_random_exponential

genai.configure(api_key=os.environ["GEMINI_API_KEY"])
model = genai.GenerativeModel("gemini-2.5-flash")

@retry(
    wait=wait_random_exponential(min=1, max=60),
    stop=stop_after_attempt(5),
    reraise=True,
)
def call_gemini_resiliently(prompt: str) -> str:
    response = model.generate_content(prompt)
    return response.text

The retry logic itself, why exponential backoff with jitter, not a fixed delay or an immediate retry, is covered in full in Rate Limits & Backoff; this is the same pattern applied directly to a real SDK call.

What goes wrong

Mistake	How you notice it	The fix
Opening a new connection per call in a tight loop	Latency is worse than expected even though the model itself responds quickly	Reuse the same client/session object across calls instead of instantiating a new one each time
No timeout set on API calls	An agent hangs indefinitely when a provider is slow or unresponsive	Set an explicit timeout and treat it as a retryable failure, same as a 429 or 502
Treating every error the same way	A malformed request gets retried 5 times for no reason, wasting time	Only retry transient errors (429, 502, 503); a 400 won't succeed no matter how many times you ask
Assuming published rate limits without checking current values	Code that worked last month starts failing after a provider changes its limits	Check current documented limits before treating a specific number as a hard design constraint

Confirm it worked

To verify that the model inference runs asynchronously on the remote provider instead of consuming local compute resources, monitor system activity while running a prompt call:

# Watch CPU while a call runs, since the work happens on the provider's servers,
# you should see near-zero local CPU usage during the wait, confirming this is
# a network-bound wait, not local computation (contrast with running a model
# locally, see the optional Local LLM Hosting course, which spikes every core).
htop

Then deliberately trigger a rate limit (fire calls faster than your tier allows) and confirm your retry logic handles it gracefully, the call should eventually succeed or fail cleanly after the retry budget is exhausted, never hang indefinitely.

Next Step: proceed to Rate Limits & Backoff for the retry pattern itself in full detail.