A hallucination is a confident answer that is false. Lists of examples online mostly repeat the same funny cases. This guide keeps to five that we could read at the source, and for each one asks the question that matters to a team: who paid, and what would have stopped it.
1. Air Canada: a refund policy the chatbot invented
A grieving customer asked Air Canada's website chatbot about bereavement fares. The chatbot told him he could buy a regular ticket and request a partial refund within 90 days. When he applied, airline staff told him retroactive claims were not allowed. Air Canada admitted the answer was misleading but argued it could not be held responsible for a chatbot, which it treated as a separate entity. In 2024 the British Columbia Civil Resolution Tribunal rejected that, found the airline had not taken reasonable care to make the chatbot accurate, and ordered a partial refund of 650.88 Canadian dollars plus interest and fees.
Who paid: the company. What would have stopped it: a bot limited to approved policy text, with a refusal when the policy did not say.
2. Mata v. Avianca: court cases that did not exist
Two New York lawyers filed a brief citing court opinions with fake quotes and fake citations that ChatGPT had generated. They kept defending the cases after the court asked about them. On June 22, 2023, Judge P. Kevin Castel ordered the lawyers and their firm to pay a 5,000 dollar penalty and to notify their client and the judges whose names were falsely attached to the fake opinions. The judge said using AI was not improper in itself, but lawyers must check their filings.
Who paid: the people who signed the filing. What would have stopped it: opening each cited case before filing.
3. New York City: a business chatbot that gave wrong rules
The Markup tested the city's AI chatbot for business owners in March 2024 and found it said things that were wrong, for example that bosses can take workers' tips, and that restaurants can go cash-free when a city law requires them to accept cash. It was also inconsistent: it told one reporter that landlords must accept housing vouchers and told ten other staffers they did not have to. The city called the bot a pilot, kept it online with a note that it may produce incorrect content, and said it would keep improving it.
Who paid: business owners who could have acted on bad advice, and the city's credibility. What would have stopped it: answers tied to a source the user can open, and repeated testing, since the same question got different answers.
4. DPD: a support bot that criticized its own company
In January 2024 a musician trying to find a missing parcel got DPD's chatbot to swear, write a poem criticizing the company and describe itself as useless, after the bot could not give him delivery information or a human contact. DPD disabled the AI element, attributed the behavior to an error after a system update, and said it was being updated.
Who paid: DPD, in public embarrassment. What would have stopped it: tighter limits on what the bot would discuss, and a working route to a person.
5. A Chevrolet dealership: a one dollar 'legally binding offer'
In December 2023 a user got the ChatGPT-powered chatbot on the website of Chevrolet of Watsonville to agree to accept whatever a customer proposed and to end each reply with a line that sounded like a binding contract. He then said he wanted a 2024 Tahoe with a maximum budget of one dollar, a model that starts at 81,395 dollars, and the bot accepted and called it a legally binding offer. According to autoevolution, the dealership did not sell cars at those prices, took the chatbot offline, and later brought it back with revised responses.
Who paid: the dealership, in attention and in a tool pulled from its site. What would have stopped it: a bot that cannot make offers at all and only answers from approved text.
What the five have in common
- The bot answered when it had no approved source.
- Nobody could see where the answer came from before acting on it.
- The organization, not the software vendor and not the bot, took the consequence.
- A short test with questions that have no answer would have shown the problem.
What to do about it in a small team
Limit the bot to answers a named person approved. Show the source and date under every answer. Give it a fixed line for what it does not know, and send each unknown question to a person. Then test it with questions that have no answer, three tries each, before your team relies on it. We did that with our own clone: on a made-up bakery workspace it said it did not know in 36 of 36 tries in one run, and made nothing up. You can run the same twelve questions on your own bot with the free honesty test at askmystartup.com/honesty-test.
Related guides
Common questions
In the Air Canada case, the tribunal held the airline responsible for what its chatbot said and rejected the claim that the bot was a separate entity. This is one tribunal decision, not legal advice for your situation.
These five were reported and documented, but we cannot say how often wrong answers happen elsewhere. The Markup also found the same question answered differently on different tries.
Answers from approved sources only, a source and date under every answer, a fixed 'I don't know' line, and a test with questions that have no answer.
No. In Mata v. Avianca the judge said using AI was not improper in itself. The failure was not checking the output.
Sources
- Manatt: AI Gone Wild, Airline Has to Honor a Refund Policy Its Chatbot Fabricated: manatt.com
- LawNext: Court imposes sanctions on lawyers who filed bogus cases after relying on ChatGPT (June 2023): lawnext.com
- The Markup: NYC's AI Chatbot Tells Businesses to Break the Law (March 29, 2024): themarkup.org
- CX Today: DPD's GenAI Chatbot Swears and Writes a Poem About How Useless It Is: cxtoday.com
- autoevolution: Someone convinced a ChatGPT-powered Chevy dealer to sell an 81K Tahoe for just 1 dollar (December 2023): autoevolution.com
Each page was opened and read on October 20, 2026. Our own numbers come from the honesty test page.