Tokens, Context Windows, and Why They Matter

The concept

LLMs don't read in words — they read in tokens. A token is a chunk of text, often part of a word, a whole short word, or a punctuation mark, that the model converts into a number before any processing happens. "Tokenization" splits "unbelievable" into pieces like "un," "believ," and "able," for example, because the model's vocabulary is built from common sub-word fragments rather than whole dictionary words — this lets the model handle rare words, typos, and even invented terms by assembling them from familiar pieces instead of needing every possible word pre-defined. As a rough rule of thumb, 1,000 tokens is about 750 words of English text — useful for estimating how much you can fit into a prompt, though the ratio shifts for code, non-English languages, and text with lots of numbers or symbols, which tend to tokenize less efficiently.

Every model has a context window: the maximum number of tokens it can "see" at once, counting your prompt, the system instructions, the entire conversation history, and any documents you've attached. This is a hard ceiling, not a soft guideline. Once a conversation (plus any files) exceeds that limit, the oldest content has to be dropped, truncated, or summarized to make room for new tokens — which is exactly why long chat sessions sometimes seem to "forget" something you said earlier, or lose track of details from the beginning of a long pasted document. The model isn't being careless; the earlier content is often simply no longer present in what it's processing.

Context windows vary a lot by model and have grown dramatically — from a few thousand tokens in early models to hundreds of thousands, even millions, in newer ones — but the underlying mechanic is the same: more tokens in means less room for the model to reason with, and once you're past the ceiling, something gets left out. There's also a subtler cost even within the limit: models generally pay more attention to information near the start and end of a long context than to content buried in the middle, an effect sometimes called "lost in the middle." A fact you need the model to use reliably is safer stated close to your actual question than six pages earlier in a long document dump.

Understanding this changes how you work with AI: it's why restating key facts partway through a long conversation, or breaking a huge document into smaller chunks, often produces more accurate results than dumping everything in at once and hoping the model tracks all of it. It also explains why starting a fresh conversation for a new task — rather than continuing an old, cluttered one — frequently gives sharper, more focused answers.

Worked example

Consider the sentence: "Tokenization is tricky." A human reads this as four words. A tokenizer might break it into something like: "Token", "ization", " is", " tricky", "." — five or six tokens for four words, because "Tokenization" isn't a single common unit in the model's vocabulary, so it gets split into familiar sub-word pieces ("Token" + "ization"). Short, common words like "is" and "a" are usually single tokens; rare, technical, or made-up words get chopped into multiple pieces, and so do long numbers and unusual punctuation.

Scaling this up: a 1,500-word blog post is roughly 2,000 tokens. A 300-page novel is roughly 150,000–200,000 tokens. If your AI assistant's context window is 128,000 tokens, you could fit most of that novel in — but if you also need room for the model's own answer, your conversation history, and any system instructions, you're already close to the ceiling before you start asking questions about it. A practical case: pasting an 80-page PDF contract (roughly 40,000 tokens) into a 32,000-token-window assistant simply won't fit; the tool will either reject it, silently truncate it, or summarize it before you ever see the full document reflected in its answers.

Exercise

Paste a long email thread, a long article, or (if you have one) a multi-page document into a chat assistant. Ask a specific question about something in the opening paragraph. Then, in the same conversation, paste in several more pages of unrelated text (another article, or just a long block of notes) and ask the same question again. Compare the two answers: did the model still reference the opening paragraph accurately the second time, or did its answer get vaguer, generic, or wrong?

As a second step, take that same long document and instead of pasting it all at once, break it into two or three shorter chunks, feeding one at a time and asking the model to summarize key points after each chunk before you ask your original question. Compare the accuracy of this chunked approach against the single-dump approach. Write down what happened in both cases — this is a direct, hands-on observation of context window pressure in action, not just a theoretical risk, and a first look at a technique (chunking) you'll reuse in the research-workflow lesson.

Recap

• Tokens, not words, are the model's real unit of processing — roughly 1,000 tokens ≈ 750 words, with code and non-English text often tokenizing less efficiently.
• The context window is a hard ceiling on prompt + history + attachments combined; once exceeded, older content gets dropped, truncated, or summarized, and content in the middle of a long context gets less attention even before that ceiling is hit.
• Restating key facts near your actual question and chunking long documents keeps long AI sessions accurate — a practical habit this course will return to in the research and workflow lessons.

Resources

• OpenAI Tokenizer (try tokenizing your own text) — https://platform.openai.com/tokenizer
• OpenAI: "Managing context for text generation" — https://platform.openai.com/docs/guides/text-generation
• Anthropic: "Context windows" — https://docs.anthropic.com/en/docs/build-with-claude/context-windows
• Hugging Face: "Summary of tokenizers" — https://huggingface.co/docs/transformers/tokenizer_summary