How LLMs work: tokens, training, context windows

The concept

A large language model (LLM) like GPT-4 or Claude doesn't "understand" your business the way an employee does. It's a pattern-prediction engine trained on enormous amounts of text. Three ideas explain almost everything you need to know to use one well: tokens, training, and context windows.

Tokens are the chunks of text a model actually reads — not quite words, not quite letters. "Automation" might be one token; "unautomated" might split into three ("un", "automat", "ed"). English averages roughly 4 characters per token, so a 1,000-word business memo is usually 1,300–1,500 tokens. Every API call you pay for is priced per token, in and out, so token count is also your cost meter. Numbers, punctuation, and especially non-English text tokenize less efficiently — a sentence in French or Japanese can cost 1.5–3x the tokens of the English equivalent, which matters if you're building workflows for international customers and budgeting cost per call.

Training is how the model learned its patterns: it was shown a vast slice of the internet, books, and code, and repeatedly predicted the next token until it got good at it. That means its knowledge has a cutoff date, it has no built-in memory of your company, and it learned both excellent writing and plenty of nonsense — it doesn't "know" which is true, it just reproduces patterns that are statistically likely. A second training phase — fine-tuning and reinforcement learning from human feedback — is what teaches the model to follow instructions, refuse harmful requests, and format answers helpfully instead of just completing text like a raw autocomplete.

Context windows are the limit on how much text the model can hold in its "working memory" for one request — your prompt, any documents you paste in, any prior conversation turns, and its own reply, all counted together in tokens. A 128,000-token context window sounds huge until you paste in a 40-page contract, your company's style guide, and ask for three alternative drafts — now you're budgeting that window like RAM on an old laptop. Context is also not free of cost: most providers charge for every token you send as input, whether or not the model "needs" all of it, so a bloated prompt with irrelevant pasted text is both a quality risk (burying the real question) and a cost leak.

Worked example

Say you need to summarize a 3,000-word customer contract (roughly 4,000 tokens). Feed it to a model with an 8,000-token context window and you still have room for your instructions and its answer — it fits in one pass. Feed the same contract plus two prior contracts for comparison (12,000 tokens of source material) to that same model, and it won't fit; you'd need to either use a model with a larger context window or chunk the documents and summarize each piece separately, then summarize the summaries. GPT-4 and Claude both offer high-context variants (128K–200K tokens) specifically so you rarely have to chunk routine business documents, but it's not infinite, and every extra token you send costs money and adds latency, so "just paste in everything" is not free.

This is also why a vague prompt like "summarize this" on a huge document can quietly fail: the model either truncates silently (some APIs reject the call; others drop content) or produces a summary that skipped the middle third of your document because it never fit in the window the way you assumed. The failure is rarely loud — you get a plausible-sounding, confidently wrong summary rather than an error message, which is exactly why understanding the mechanics up front saves you from shipping a broken workflow that looks fine in a demo.

Exercise

Take a real document you use at work — a proposal, a contract, or a policy.

1. Estimate its token count. A fast rule of thumb: divide your word count by 0.75 (so 1,000 words ≈ 1,300 tokens), or paste it into any free tokenizer tool to get an exact count.
2. Compare that number to two different models' published context window limits (for example, an 8K-context model and a 128K-context model).
3. Write two sentences: does your document fit in one request to each model, or would it need to be split into chunks? If it needs chunking, note roughly how many chunks.
4. Note whether your document contains significant non-English text, tables, or code — all three tokenize less predictably than plain prose, so pad your estimate by 20–30% if so.

Keep this number — you'll use the same document again when you compare model costs in the next lesson.

Example estimate:
Document: 14-page vendor contract, ~5,600 words
Estimated tokens: 5,600 / 0.75 ≈ 7,470 tokens
Model A (8K context): fits in one pass, barely — no room for a long reply
Model B (128K context): fits easily, room for multi-part analysis in one call

Recap

• LLMs predict the next token based on patterns learned during training, not live understanding of your business.
• Tokens are the unit of both memory (context window) and cost — roughly 4 characters each, more for non-English text, tables, or code.
• Always size your documents against a model's context window before assuming "just paste it in" will work; this vocabulary is the foundation for every model-selection and cost decision in the rest of the course.

Resources

• OpenAI, "What are tokens and how to count them" — tokenizer reference and official token-counting tool.
• Anthropic, "Models overview" — published context window sizes per Claude model.
• Hugging Face, "Tokenizers" documentation — how sub-word tokenization works under the hood.