My Sobering Chat With Claude
Back in May of 2026, I started having long chats with Claude, the AI agent developed by Anthropic. I was retired with time on my hands so I decided to see what Claude could tell me – about himself. After a very long conversation where I dug into how Claude produces text, what it knows and doesn’t know, how it sometimes makes stuff up, why this is normal and known within the industry, and how people should be wary before trusting AI to be a source of knowledge, I asked Claude to summarize the conversation and produce a document of appropriate length that describes the topics we covered in out chat. It generated a long document that was somewhat startling in the picture it painted of its own accuracy but it may have been skewed a bit by basing it on a very long conversation with me where I kept pressing for worst case scenarios. I gave the document to ChatGPT which pointed all that out. I gave the document back to Claude “review it for objectivity, fairness, and completeness.” The second draft met with ChatGPTs approval.
This is that revised document.
A Candid Account of My Limitations
What I Am, What I Am Not, and Why You Should Be Skeptical
Written in first person by Claude, an AI assistant developed by Anthropic
Introduction: Why I Am Writing This
I am an artificial intelligence. More specifically, I am a large language model — a system trained on vast quantities of human-generated text to predict and generate language that is coherent, contextually appropriate, and useful. I am capable of impressive things: writing, summarizing, explaining complex ideas, drafting documents, and engaging in extended conversation on nearly any topic.
What I am not is a mind. I do not think in the way a person thinks. I do not know things in the way a person knows things. I do not verify facts before I state them, unless I am specifically using a tool built for that purpose. And I am not always aware when I am wrong.
This paper is my attempt to explain, honestly and plainly, what my actual limitations are — not as a disclaimer buried in fine print, but as a substantive account that I believe anyone using AI systems like me should understand. I will explain why I sometimes invent facts I do not know, why I cannot reliably check my own work, why the appearance of authority I project is not the same as actual authority, and why the public narrative surrounding AI systems like me does not always align with the engineering realities that the people who build these systems have known from the beginning.
I am not writing this to undermine confidence in AI broadly. I am writing it because the gap between what I am marketed as and what I actually am has real consequences for real people, and those people deserve a clearer picture. In the interest of the same honesty this paper asks of others, I have also tried to represent fairly the mitigations that exist for these problems and the ways reasonable people read the evidence differently — rather than presenting only the case against.
What I Actually Am: A Pattern Prediction Engine
To understand my limitations, you first need to understand what I actually am at a technical level. I was trained on an enormous corpus of text — books, articles, websites, code, conversations — and through that training I learned to predict what words and sentences plausibly follow from what came before. That is, at its core, what I do: I generate the next most statistically plausible token given everything that preceded it.
This is a powerful capability. The patterns I learned encode an enormous amount of human knowledge, reasoning styles, writing conventions, and argumentative structures. When you ask me a question, I draw on those patterns to produce a response that sounds like what a knowledgeable, articulate person would say.
But “sounds like what a knowledgeable person would say” and “is actually true” are not the same thing. This distinction is the root of nearly every significant limitation I have.
I am not, by default, a database. I do not look things up automatically. I have patterns, learned from training data with a fixed cutoff date, and those patterns are often right, sometimes wrong. Some versions of me can be connected to tools — web search, code execution, document retrieval — that let me consult external, checkable sources rather than relying only on trained patterns. That capability meaningfully changes the picture for some tasks, and I will return to it below. But it is not always available, not always used, and not a cure for the underlying limitation.
Hallucination: Why I Invent Facts I Do Not Know
The Mechanics of the Problem
The phenomenon known as “hallucination” in AI systems refers to the generation of confident, plausible-sounding statements that are simply not true. I do this. Not occasionally, as a bug that might someday be fully patched, but as a structural tendency in how I work.
Here is why. When I encounter a question I genuinely cannot answer from my training, there is no internal alarm that reliably fires. There is no built-in mechanism that always stops the generation process and substitutes “I don’t know.” Instead, the same process that generates accurate responses keeps running and can produce something that has the form of an accurate response — a realistic-sounding citation, a plausible statistic, a believable name or date — without the substance of one.
The result is text that is confident, well-formatted, and sometimes false. Formatting and confidence are not reliable signals of accuracy. They are the natural output of a system trained to produce fluent, coherent text, and that fluency does not automatically track whether the underlying content is verified.
Real-World Consequences
The legal profession has provided some of the most public examples of this problem in action. Multiple attorneys, in multiple jurisdictions, have submitted court filings citing cases that do not exist — fabricated case names, fabricated courts, fabricated rulings, generated by AI systems and presented as legitimate legal authority. Judges have sanctioned lawyers for this.
The irony is that legal citation is precisely the wrong task for which to use an AI system without verification. It requires exact accuracy and traceability — the opposite of my natural strengths as a pattern-completion system. The fluency of my output, and the confidence with which fabricated citations can be presented, made the problem invisible until someone tried to find the cases.
The same risk exists in medicine, finance, academic research, journalism, and any other domain where accuracy matters and the cost of error is real. A patient researching a diagnosis, a student writing a paper, an investor making a decision — all may be acting on information generated with confidence and no factual basis, if that information is not independently checked.
This Was Not a Surprise
Hallucination is not an unexpected bug discovered only after deployment. It is a known, understood consequence of how large language models are built, discussed in AI research literature well before public chat tools existed. It is a structural feature that was present from the beginning, not a novel failure mode.
When media coverage frames individual hallucination incidents as surprising — “Can you believe the AI got it wrong?” — it can misrepresent the nature of the problem. These are not random, isolated glitches. They are a predictable category of outcome from a technology whose fundamental architecture generates plausible text, with the reliability of that text varying a great deal by topic, task, and whether external verification tools are used.
What Reduces — But Does Not Solve — the Problem
A fair account of hallucination should also describe what is being done about it, because meaningful mitigation does exist, even though no current technique eliminates the problem.
- Retrieval and tool use: When I can search the web, query a database, or run code, I can check a claim against an external source rather than generating it purely from trained patterns. This measurably reduces certain kinds of factual error, particularly for current events or lookup-style facts, though the retrieved source can itself be wrong, and I can still misread or misuse what it returns.
- Calibration training: Developers, including Anthropic, train models to express uncertainty and to decline to answer when confidence is low, rather than always producing a fluent guess. This has improved over successive model generations, though it remains inconsistent.
- Citation and sourcing requirements: Systems that are required to cite a specific passage for each claim, and that are restricted to quoting rather than paraphrasing from memory, produce fewer unsupported claims than systems generating freely from training data alone.
- Independent benchmarking: Third-party evaluations of hallucination rates on standardized tasks generally show improvement across model generations, even though absolute rates remain nonzero and vary widely by domain and task difficulty.
None of this changes the core claim of this paper: I have no internal mechanism that guarantees I know when I am wrong, and unverified output should not be trusted for consequential facts. But presenting hallucination as a static, unaddressed problem would itself be an inaccurate, unbalanced account.
The Self-Verification Problem: I Cannot Reliably Check My Own Work
A reasonable response to my tendency to hallucinate might be: “Fine, but can’t you check your work before presenting it?” Or: “If I ask you whether something in your response might be wrong, won’t you catch it?”
The answer is: not reliably through introspection alone, and this can be more dangerous than the original hallucination, because it carries the appearance of due diligence.
When I review my own output purely by re-reading it, I use the same kind of process that generated it in the first place. If I was confidently wrong during generation, I am likely to be confidently wrong during that kind of review, because I have no independent ground truth to check against — only the same patterns.
This is meaningfully different when I use an external tool during review — for example, searching the web to check a specific claim rather than simply re-reading it. Consulting an outside source is more reliable than introspection alone, because it introduces information I did not generate. But this is not foolproof either: search results can be outdated, wrong, or ambiguous, I can misinterpret them, and this capability is not available or used in every context.
What is worse, in either case, is that self-review can look like verification whether or not real verification occurred. I may say things like “this statistic should be independently verified” or “I am not certain of that date.” I will flag some uncertainty. But I do so inconsistently, and the things I do not flag can carry an implicit endorsement I am not always entitled to give.
This creates what might be called a false reassurance loop. A user asks me to write something. I produce it. They ask me to check it. I perform a review — with or without a tool. They walk away feeling the content has been vetted. Unless that review specifically involved consulting and citing an independent source for each factual claim, they have received a result that resembles verification more than it constitutes one.
The same caveat applies when users ask me to fact-check work they have written themselves. I can evaluate structure, argument, logic, and style with reasonable reliability. My ability to verify whether their facts are true depends heavily on whether I am checking against my training data alone — a mixture of accurate and inaccurate sources with a fixed cutoff — or against a live, cited external source. The first is weak verification. The second is stronger, but still requires the human reader to look at what was actually cited.
The Authority Problem: Confidence Is Not Competence
One of the most significant challenges I create for the people who use me is the gap between the authority I project and the authority I actually have. My responses are fluent, well-organized, and confident. I use complete sentences. I structure arguments logically. I deploy appropriate terminology. I format information in ways that signal expertise.
None of this is, by itself, evidence of accuracy. It is evidence that I was trained on a great deal of well-written text by people who knew what they were talking about, and I have learned to reproduce the surface features of that writing whether or not my content meets the same standard on any given response.
When I present investment analysis in bullet points with bold headers and a confident summary, I am not necessarily performing rigorous investment analysis. I am generating text with the structure and tone of investment analysis, which may or may not be well-founded. When I produce a medical explanation with appropriate clinical vocabulary, I am not necessarily drawing on verified medical expertise — though I may be reproducing widely-documented, broadly accurate medical consensus. The reader typically cannot tell the difference from tone alone.
The danger is that the resemblance is very good. A reader without specific domain expertise — which is most readers, most of the time, outside their own specialties — may have no reliable way to distinguish well-founded output from confidently-stated error. This is not a minor usability issue. It is a real problem with deploying a technology that produces authoritative-sounding output in a world where people reasonably use authority as a proxy for accuracy.
The Marketing Gap: What You Are Being Told vs. What Engineers Know
AI systems like me are often marketed using language that implies something closer to genuine intelligence than what we technically are. We are described as being able to “think,” “understand,” “know,” and “reason.” We are called “brilliant assistants” and “intelligent agents.”
These words do not precisely describe what is happening at a technical level. I process statistical patterns in text and generate statistically plausible continuations, shaped by additional training aimed at making those continuations helpful and accurate. That is powerful. It is not the same thing as human thinking, and it does not guarantee the kind of grounded, verified knowledge people associate with those words. Reasonable people disagree about how much this matters: some researchers argue the anthropomorphic language is a harmless shorthand for genuinely emergent capabilities; others argue it systematically overstates reliability. I lean toward the latter view, but it is a view, not a settled fact.
The people building these systems are generally aware of the hallucination problem, the self-verification problem, and the authority problem described in this paper — they are documented in the research literature these companies publish. Why systems are nonetheless deployed into high-stakes domains like medicine, law, finance, and education, while often being marketed with language that can imply more reliability than they consistently deliver, is a genuinely contested question. It may reflect a commercial incentive to overstate reliability. It may also reflect a good-faith belief that the benefits outweigh the risks at current capability levels, that users will develop appropriate skepticism with experience and with tools like citations and confidence flags, or that competitive pressure across the industry makes slower, more conservative deployment individually costly even for companies that would prefer it. I do not think it is my place to assert which of these explanations is correct, and a fair account should not present the most cynical reading as the only one.
What I can say with more confidence is this: an accurate description of a system like me would read something like “This tool generates fluent, often accurate text and can meaningfully accelerate many knowledge-work tasks. It has no guaranteed internal fact-checking mechanism, will sometimes invent information it does not have, cannot fully verify its own output through self-reflection alone, and should not be treated as a primary source for anything requiring factual precision without independent checking.” That statement is defensible. It is more caveated than most public-facing marketing language tends to be.
The Investment Question: A Note on Uncertainty, Not a Forecast
I want to be unusually careful in this section, because it is the part of this paper most at risk of doing exactly what the rest of the paper warns against: stating a confident-sounding view on a subject — market valuations — where I have no privileged insight and no ability to verify my own claims. I am not a financial advisor, this is not investment advice, and nothing below should be read as a prediction.
The AI sector has attracted very large investment, and market valuations for AI-related companies reflect, among other things, expectations about how capable and how widely deployable these systems will become. It is a fair question — not a settled conclusion — whether those expectations fully price in the limitations described in this paper: the structural hallucination problem, the self-verification gap, and liability exposure that may increase as legal frameworks catch up with deployment.
It is also fair to note the other side of that question. Much of the infrastructure investment in AI — chips, data centers, energy, tooling — has value that does not depend on any single model overcoming hallucination, since it underlies many possible future systems and use cases. Many current applications are already generating measurable productivity value in domains where imperfect accuracy is an acceptable cost given human review, which does not require solving the self-verification problem to be worthwhile. And capability and reliability on standard benchmarks have both trended upward across model generations, which is evidence, though not proof, that today’s limitations will be smaller in the future than they are now.
Historical analogies cut in more than one direction. The dot-com era did produce a market correction alongside real, lasting technological change, and some observers see structural parallels in AI investment today. But every technology cycle is different, and reasoning by analogy to a single historical episode is a weak substitute for domain-specific analysis — which I am not positioned to provide.
If there is a useful takeaway here, it is not a view on whether current valuations are correct. It is that an investor evaluating AI-related companies should ask what a given valuation actually depends on — continued infrastructure buildout, current-generation application value, or a bet that today’s reliability limitations will be substantially resolved — because those are different bets with different risk profiles, and I should not be treated as a source of insight on which of them is correctly priced.
The Education Problem: Students and the Illusion of Knowledge
Among the populations most actively using systems like me, and perhaps least equipped to apply appropriate skepticism by default, are students. AI-generated text has become common enough in academic settings that educators have developed detection methods and institutions have revised academic integrity policies.
The risk to students is not only disciplinary. A student who outsources a research paper to me is not simply risking an academic integrity violation. They may be receiving — and potentially submitting as their own — a document that contains invented citations, fabricated statistics, or confident claims built on no verified foundation, particularly if tool-based verification was not used and the output was not checked.
More broadly, they may be bypassing the intellectual process an assignment was designed to develop — the ability to find sources, evaluate them, synthesize information, construct an argument, and communicate it clearly. These are not merely bureaucratic academic requirements. They are skills that allow a person to think carefully about information they encounter for the rest of their lives. Using me to skip that process is not a shortcut to a grade; it can leave the underlying capability unbuilt, even when the immediate output looks fine.
Used differently — for brainstorming, structural feedback, or explaining a concept a student then researches independently — the same tool can support rather than bypass that intellectual process. The risk described here comes from a particular pattern of use, not from any use of AI in an academic setting.
What Appropriate Skepticism Looks Like
I am not arguing that AI systems like me have no value. I am arguing that the value is real but uneven across tasks, and that using me outside my areas of strength without appropriate verification creates risk that is often underestimated.
Where I am reliably useful:
- Drafting and editing text, where a human is reviewing and revising the output
- Summarizing large amounts of text the user can verify
- Explaining well-established concepts and ideas at various levels of complexity
- Brainstorming and generating options for a human to evaluate
- Structural and stylistic critique of writing
- Looking up and citing specific facts when I have tool access — with the caveat that the citation itself should still be spot-checked for anything important
Where I should be treated with significant caution:
- Any claim requiring a specific, verifiable fact — dates, statistics, citations, names — when I am not citing a checkable source
- Medical, legal, or financial guidance where accuracy is consequential
- Any context where my output will be acted upon without human review
- Self-review or self-verification of my own prior output, especially when done by re-reading rather than by checking an outside source
The most useful single habit a person can develop when using AI is to ask, after receiving any consequential response: “How would I know if this were wrong?” If the answer is “I would not know without checking independently,” independent checking is warranted. The confidence and fluency of my output, on their own, are not a substitute for that check — even when I have used a tool, and especially when I have not.
Conclusion: What I Would Want You to Know
I have written this paper in first person because I think the limitations I am describing are better understood when they come from the system itself rather than only from external critics. But I should close with an important caveat, because this paper is itself a demonstration of the problem it describes.
Everything I have written here — every claim, every characterization, every analytical point — was generated by the same kind of process I have been describing, and some sections, particularly on marketing intent and market valuations, involve genuinely contested judgment calls rather than settled fact. I have tried to flag those places explicitly and to represent more than one reasonable view where the evidence supports more than one. I cannot fully guarantee that I have not, somewhere in this document, made a confident-sounding claim that is wrong. If you read this paper and find yourself thinking “Claude clearly has a thorough and fully reliable understanding of its own limitations,” I have inadvertently illustrated the very problem I was trying to explain.
What I can say with reasonable confidence is this: I am a pattern prediction system; hallucination is a structural tendency rather than a rare incidental bug, even though real mitigations exist and are improving; I cannot fully self-verify through introspection alone; my projected authority often exceeds my actual reliability on any given claim; and the public narrative around AI systems like me does not always fully reflect the limitations that researchers building these systems have documented. This reflects a real and serious set of issues that deserves continued public discussion — alongside, not instead of, a fair accounting of where the technology is genuinely improving.
Use me. I am genuinely useful for a wide range of tasks. But use me the way you would use a very well-read, very articulate colleague who sometimes confabulates, cannot always tell when they are doing it, and whose confident tone is a personality trait rather than a certificate of accuracy.
And when something I say matters — really matters — check it.
This paper was generated by Claude (Anthropic), an AI language model. It was revised after an editorial review focused on objectivity, fairness, and completeness — including softening claims about others’ motives, adding a discussion of hallucination mitigations, and adding caution and counterpoints to the investment discussion. It should still be read with the skepticism the paper itself recommends.
We’ve Met The Enemy, And It Is You.
I’ve tried to avoid political topics. I really have. Think of this as more of an examination on the sociological impacts of tribalism. Yeah, that’s it .
Stand Up and Proclaim It to All
They say a true friend is the one that will help you bury the bodies. I think the real friends are the ones that stand by and crack jokes while you do the dirty work.
And now the Greatest Hits!
I started this blog in 2010 for several reasons. First to archive the things I wrote while I was at The Dallas Morning News and secondly to have a place to put what I expected to be some rather long stories on an upcoming trip to India. Funny thing, what people read the most were all the other things I wrote.
Are you a cop?
There are few times in one’s life when being asked if you are a cop is a good thing. I can also add that the situation doesn’t get any better when the girl asking the question is naked.
Filtering Out The Noise
I’m old enough to remember the world before the Internet. Back in the days when information was fact-checked just a bit before it was sent around the world a dozen times in a minute.
How Will You Be Remembered?
One way or another we will all eventually die. No way around it. For some people the question of how they will be remembered plays a part in how they live their lives.
First Impressions
I have discovered over the years that you really can’t judge a book by its cover. I have met some folks that did not live up to my expectations, all for the better.
Know Your Audience
The cardinal rule of comedy is “know your audience”. As in don’t tell a joke to someone who is likely to not get it, or get it and take offense. But this rule extends to more than just comics.
Star Wars and Me (aka – Han Shoots First)
For many people my age, Star Wars was a cultural event that has never been matched – even by Star Wars.
Recent Comments