Hallucinations and Confabulation

The concept

A "hallucination" is the term the AI field uses when an LLM states something false with exactly the same fluent, confident tone it uses for something true — a fabricated citation that sounds academically legitimate, a wrong statistic presented with suspicious precision, or a quote attributed to a real person that they never actually said. It's sometimes called "confabulation," a term borrowed from psychology that may be more accurate: the model isn't malfunctioning or lying on purpose, it's doing exactly what it always does — generating the most plausible-sounding continuation of the text — except in this case, "plausible-sounding" and "true" have come apart.

This happens because the model's objective during training and generation is to produce text that looks like what a knowledgeable answer would look like, not to retrieve and cross-check a verified fact. For well-documented, heavily repeated topics, those two things — plausible and true — usually align, so most everyday answers are accurate. But for niche topics, obscure names, specific numbers, or anything where the training data was thin, contradictory, or simply didn't cover the exact question asked, the model will still produce a fluent answer, because producing fluent text is what it does — it just may no longer be a true one. The model has no internal "I don't actually know this" signal that reliably fires before it answers; it generates the next plausible token whether or not the underlying claim is grounded.

There are recognizable warning signs worth training your eye on: oddly specific numbers attached to vague claims ("37.4% of companies report..." with no source), citations to papers, studies, or articles that don't actually exist or don't say what's claimed, direct quotes that sound a little too clean or on-the-nose, confident, detailed answers to questions about very narrow or obscure subjects where you'd expect genuine uncertainty, and names, dates, or version numbers that feel plausible but that you can't quite place. None of these guarantee a hallucination, but they're exactly where to focus verification effort, especially before a claim goes into something public-facing: a report, a client deliverable, a legal document, or a social post.

It also helps to know what reduces hallucination risk, since this course builds on it later: giving the model source material to work from (so it's summarizing, not recalling from memory), asking it to cite exactly where in the provided text a claim comes from, and using tools with live retrieval or search instead of relying purely on the model's trained-in memory. None of these eliminate the risk entirely, but they shift the task from "recall a fact from training" to "work with text that's actually in front of you," which is a much safer mode.

Worked example

Imagine an AI-generated answer that includes: "A 2019 study published in the Journal of Applied Cognitive Psychology found that 68% of remote workers reported improved focus after using structured to-do lists (Martinez & Chen, 2019)." This looks legitimate — a plausible journal name, a specific year, two author names, and a precise percentage. But none of those details may correspond to anything real. The tells: the statistic is suspiciously precise for a claim this specific, the journal name sounds generic and constructed, and the two-author citation format mimics real academic style without being traceable. Searching for the exact paper title or author names on a search engine or academic database and finding nothing real is the confirmation. If instead you had given the model an actual PDF of a real study and asked it to summarize the findings with page references, the risk drops sharply, because it's reading from provided text rather than reconstructing from memory.

Exercise

Ask an AI assistant for three sources (studies, articles, or statistics) on a reasonably niche topic relevant to your work or interests — something specific enough that it isn't common knowledge. For each of the three, try to independently verify it: search for the exact title or claim, check if the cited publication or author exists, and see if the number or quote shows up in a real, findable source. Log how many of the three were fully real, how many were altered (real source, wrong detail), and how many appear to be entirely invented. Then repeat the request, but this time paste in one real article on the topic first and ask the model to cite only from that article. Compare the hallucination rate between the two approaches.

Recap

• Hallucinations are a structural feature of how LLMs generate text, not a rare glitch — the model optimizes for plausible-sounding output, not verified truth, and has no reliable internal signal for "I don't know."
• Warning signs include suspiciously precise statistics, unfindable citations, too-clean quotes, and confident answers on obscure topics; giving the model real source text to work from reduces the risk.
• Any AI output used for a real decision — a report, a claim to a client, a number in a proposal — needs independent verification; this habit is built on throughout the rest of the course.

Resources

• Anthropic: "Reducing hallucinations" — https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/reduce-hallucinations
• OpenAI Help Center: "Why does ChatGPT sometimes give incorrect answers?" — https://help.openai.com/en/articles/6783457
• Wikipedia: "Hallucination (artificial intelligence)" — https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)