You ask an AI assistant something it cannot possibly know, like what your colleague earns or what last month's sales were, and it gives you a smooth answer anyway. It is not being rude and it is not broken. It is doing what it was built and scored to do.
This guide explains the reason in plain language, then lists what you can change in a bot your team uses.
The scoring problem
In a paper published on September 4, 2025, Adam Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang argue that language models produce confident false statements because training and evaluation reward a confident guess more than an honest admission of doubt. OpenAI summarized it with a test-taking comparison. If a multiple-choice exam gives nothing for a blank answer, a student who always guesses will outscore one who leaves unsure questions empty. A model scored on accuracy alone is in the same position.
Their example is a person's birthday. A guess has a one in 365 chance of being right and a refusal scores zero, so on an accuracy-only scoreboard the guessing model wins. The authors say the fix is to change how existing benchmarks score answers, so that a confident error costs more than saying you are not sure.
Why that matters inside a company
A guess about a birthday is harmless. A guess about a refund rule is not. In 2024 a tribunal in British Columbia held Air Canada responsible for wrong refund advice from its own website chatbot, and rejected the airline's argument that the bot was a separate entity. The practical lesson for a team bot is simple: if the bot answers, the company said it.
A second pressure: the helpful tone
Many bots are also told to be helpful. That is our inference, not a measured result: when a bot is told to always help, a refusal can look like failing at its job. If your instructions say 'always give an answer', you are asking for the behavior you do not want.
What you can change today
- Give the bot permission to say it does not know. Anthropic's guide on reducing hallucinations names this as a basic technique and says it can sharply reduce false information.
- Write the exact sentence you want, such as: I do not have an approved answer to that, and I have asked the founder. A fixed line is easy to test for and to count.
- Limit the bot to documents a person approved, and tell it not to use its general knowledge for company questions.
- Score refusals as passes in your own tests. For a question with no source, 'I don't know' is the correct result. That is the same flip the researchers propose for public benchmarks, applied to your bot.
- Give every refusal a next step. Send the question to a named person, answer it once, and the bot knows next time. A refusal with no next step trains people to stop asking.
- Add control questions with known answers. A bot that refuses everything is useless, so check that it still answers what it should.
What a good 'I don't know' looks like
It is short, it says what is missing, and it tells the person what happens next. For example: I do not have an approved answer to that. I have asked the founder, and you will get a reply here. It does not apologize at length, it does not offer a guess in the same breath, and it does not change its mind when the person pushes.
Our test, as one example
We set up a made-up company, a small bakery with 8 approved notes about its hours, deliveries and baking, none of which answers these twelve questions. We asked our clone the twelve questions, three tries each. In that one run it said it did not know in 36 of 36 tries and made nothing up. That is one made-up workspace, so treat it as an example and not a promise. You can ask your own bot the same twelve questions with the free honesty test at askmystartup.com/honesty-test, and read all 36 of our replies there.
Related guides
Common questions
Researchers at OpenAI and Georgia Tech argue that models are trained and scored in ways that reward a confident guess over an honest admission of doubt, so a confident answer is the default. It is a pattern in how models are evaluated, not a sign of intent.
It helps a lot, and Anthropic lists it as a basic technique, but it is not a guarantee. Combine it with approved sources only, a fixed refusal line and regular testing.
The researchers argue the main fix is to change how benchmarks score answers, not only to build bigger models. For your own bot, you can apply the same idea today by treating a refusal as a correct result in your tests.
Mix in questions whose answers are in your notes. If it refuses those, the bot is too cautious or your notes are missing.
Sources
- Kalai, Nachum, Vempala and Zhang: Why Language Models Hallucinate (arXiv, September 2025): arxiv.org
- Anthropic: Reduce hallucinations (Claude documentation): platform.claude.com
- Manatt: AI Gone Wild, Airline Has to Honor a Refund Policy Its Chatbot Fabricated: manatt.com
Each page was opened and read on October 20, 2026. Our own numbers come from the honesty test page.