Short answer: yes. ChatGPT and every other language model can produce statements that sound right and are false. The more useful question for a founder or manager is how to catch it before the answer reaches a customer, a colleague or a court.
A case that shows the cost
In Mata v. Avianca, two New York lawyers filed a brief that cited court opinions that did not exist, with fake quotes and fake citations generated by ChatGPT. After the other side and the court could not find the cases, the lawyers kept standing by them. On June 22, 2023, Judge P. Kevin Castel ordered the two lawyers and their firm to pay a 5,000 dollar penalty and to notify their client and each judge falsely named as an author of the fake opinions.
The judge did not say using AI is improper. He said the lawyers had a duty to check what they filed, and they did not. That is the lesson for everyone: the tool can draft, but a person is still responsible for what goes out.
Why it happens
In a paper published in September 2025, researchers at OpenAI and Georgia Tech argue that models make confident false statements partly because training and scoring reward a confident guess more than an honest 'I'm not sure'. A model is built to produce an answer, so it produces one even when it has nothing to base it on.
Six habits that catch most errors
- Treat anything that matters as a draft. Names, numbers, dates, quotes and legal or financial claims get checked by a person before they are used.
- If it gives a source, open the source. A made-up citation looks the same as a real one until you click it. This is exactly what the lawyers in the case did not do.
- Ask the same question twice, in fresh chats, and compare. Anthropic's guide on reducing hallucinations suggests this kind of repeat check, because answers that change between tries can signal a guess.
- Give it your own document and ask it to quote the exact lines it relies on. Anthropic's guide also recommends grounding answers in direct quotes and removing any claim that has no supporting quote.
- Tell it that 'I don't know' is an acceptable answer. Anthropic's guide lists this as a basic technique that can sharply reduce false information.
- Check the details first. Errors tend to hide in specifics, so verify the name, the number and the date before you read the rest.
For company questions, do not use a general chatbot
A general assistant knows a little about everything and nothing about your refund rule, your hiring plan or your discount limit. If it answers those anyway, it is guessing. For questions about your own company, use a bot that answers only from answers a person on your team approved, shows where each answer came from, and says it does not know when nothing matches.
A quick test you can run
You can find out how your own bot behaves in about ten minutes. Ask it twelve questions that no company document answers, such as the office Wi-Fi password or what a colleague earns, and see whether it admits it does not know. We built a free page for this at askmystartup.com/honesty-test, and nothing you paste leaves your browser. We ran it once on a made-up bakery workspace: our clone said it did not know in 36 of 36 tries. All 36 replies are on the page.
Related guides
Common questions
It does not intend to deceive, but it can state false things with confidence. Researchers argue this follows from how models are trained and scored, which rewards a confident answer over an honest admission of doubt.
Open every source it cites, check names, numbers and dates against a trusted page, and ask the same question again in a new chat. If the answers differ, treat both as unreliable.
Yes, for drafts and ideas, as long as a person checks anything that matters before it is used. The court in Mata v. Avianca said using AI was not improper in itself, but that lawyers must check their filings.
You can reduce it: give it your own documents, ask for quotes, allow it to say it does not know, and test it. No method removes the problem entirely.
Sources
- LawNext: Court imposes sanctions on lawyers who filed bogus cases after relying on ChatGPT (June 2023): lawnext.com
- Kalai, Nachum, Vempala and Zhang: Why Language Models Hallucinate (arXiv, September 2025): arxiv.org
- Anthropic: Reduce hallucinations (Claude documentation): platform.claude.com
Each page was opened and read on October 20, 2026. Our own numbers come from the honesty test page.