LLM APIs and token economics

Concept

Every production agent decision — which model to call, how much context to send, how many tool round-trips to allow — is, underneath, a cost and latency decision. Before you can reason about any of that, you need to understand how LLM providers actually bill and behave.

Providers charge per token, not per request. A token is roughly 4 characters of English text, or about ¾ of a word. When you call a chat completion endpoint, you pay for two separate pools: input tokens (your system prompt, conversation history, retrieved documents, tool definitions) and output tokens (what the model generates). Output tokens are typically priced 3–5x higher than input tokens, because generation is more computationally expensive than processing a prompt.

Context windows set the ceiling on how much you can send in a single call — current flagship models support context windows from 128K to over 1M tokens, but a larger window doesn't mean you should fill it. Stuffing a prompt with irrelevant context increases cost, increases latency, and — counterintuitively — often decreases answer quality, because the model has to search a larger haystack for the relevant signal (a problem sometimes called "lost in the middle").

There's also a quality/cost/latency triangle between model tiers. Flagship models (the largest, most capable) cost more per token and respond more slowly, but make fewer reasoning errors on complex tasks. Smaller "mini" or "flash"-tier models are 5–20x cheaper and often 2–3x faster, and are the right default for simple classification, extraction, or routing tasks where a flagship model would be overkill. A well-architected agent routes different sub-tasks to different model tiers rather than using one model for everything.

Every response from a chat completion API includes a usage object reporting exact input and output token counts for that call. This is your ground truth for cost tracking — never estimate tokens by counting characters or words when the API tells you exactly how many were used.

Two more billing mechanics matter once you're past a toy script. Prompt caching: several providers now discount (sometimes by 50–90%) the input-token cost of a prefix that's identical across repeated calls — a long system prompt or a static set of tool definitions sent on every turn of a conversation. Structuring your prompt so the stable parts come first and the variable parts (user message, latest retrieved documents) come last lets you capture this discount automatically on most providers. Rate limits: every API key has limits expressed in requests-per-minute and tokens-per-minute, and both scale with account tier, not just plan price — a new account can be throttled well before it's cost-constrained. Production agents need backoff-and-retry logic for 429 responses from day one, not as an afterthought once traffic grows.

Worked Example

Suppose your RAG-grounded support agent sends a 4,000-token prompt (system instructions + 3 retrieved documents + conversation history) and receives a 500-token answer back.

# Illustrative pricing per 1M tokens (check current provider pricing before using in production)
PRICING = {
    "flagship": {"input": 3.00, "output": 15.00},   # e.g. GPT-4-class
    "small":    {"input": 0.15, "output": 0.60},    # e.g. GPT-4o-mini-class
}

def cost_per_call(input_tokens, output_tokens, tier):
    rates = PRICING[tier]
    return (input_tokens / 1_000_000) * rates["input"] + 
           (output_tokens / 1_000_000) * rates["output"]

flagship_cost = cost_per_call(4000, 500, "flagship")
small_cost = cost_per_call(4000, 500, "small")
print(f"Flagship: ${flagship_cost:.4f}  Small: ${small_cost:.4f}")
# Flagship: $0.0195  Small: $0.0009

At 10,000 calls/day, that's $195/day on the flagship tier versus $9/day on the small tier — a $5,600/month difference for what might be the same quality of answer on a simple FAQ-style query. Now layer in caching: if 3,000 of the 4,000 input tokens are a static system prompt and tool schema repeated on every call, and your provider discounts cached input tokens by 75%, the flagship cost drops further — the arithmetic below extends the function above:

def cost_with_cache(cached_tokens, fresh_input_tokens, output_tokens, tier, cache_discount=0.75):
    rates = PRICING[tier]
    cached_cost = (cached_tokens / 1_000_000) * rates["input"] * (1 - cache_discount)
    fresh_cost = (fresh_input_tokens / 1_000_000) * rates["input"]
    output_cost = (output_tokens / 1_000_000) * rates["output"]
    return cached_cost + fresh_cost + output_cost

print(cost_with_cache(3000, 1000, 500, "flagship"))  # notably cheaper than $0.0195

This is why module 5 dedicates an entire lesson to cost-and-latency optimization: the gap compounds fast at real traffic, and caching plus tiering are the two levers with the highest leverage for the least engineering effort.

Try It

Write a small script that sends the identical prompt to two different model tiers (a flagship and a smaller model) through the OpenAI or Anthropic SDK. For each call:

1. Read response.usage.prompt_tokens and response.usage.completion_tokens (field names vary slightly by provider — check the SDK docs).
2. Record wall-clock latency with a simple timer around the call.
3. Compute and print a cost comparison table showing tokens, latency, and dollar cost per call for each tier.
4. Repeat the flagship call twice with an identical system prompt and check whether the provider's response reports any cached-token count — note what you find even if your account doesn't qualify for caching yet.

You'll reuse this exact cost-tracking pattern in every later module — from tool-calling loops in Module 2 to the production observability dashboard in Module 6.

Recap

• LLM APIs bill separately for input and output tokens, with output priced several times higher; the usage field in every response is your exact cost ground truth.
• Larger context windows cost more and can hurt accuracy via "lost in the middle" effects, and prompt caching plus model tiering are the two highest-leverage cost levers.
• Rate limits are independent of cost — production agents need retry-with-backoff on 429s from the first deployment, not after traffic grows.

Resources

• OpenAI API pricing and usage docs: https://platform.openai.com/docs/pricing
• Anthropic prompt caching guide: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
• "Lost in the Middle" paper (Liu et al., 2023): https://arxiv.org/abs/2307.03172
• Provider rate-limit documentation: https://platform.openai.com/docs/guides/rate-limits