The mistakes these tools make are not the mistakes you're braced for. They don't produce obvious nonsense you'd spot immediately. They produce something articulate, well-structured and entirely plausible, and entirely false.
The industry calls this "hallucination," which is a soft word for it. Fabrication is closer.
Why it happens
As covered in what AI actually is: these tools produce what a good answer would probably look like. If you ask for a section number, a court case or a statistic, a correct-looking one is what usually follows that question, so that's what you get. Whether it exists is a different matter entirely.
Crucially, it sounds exactly the same whether it knows or is guessing. There's no wobble in the voice.
The five things to never take on trust
- Numbers and calculations. Any figure that ends up in an invoice, a return or a report gets checked. Every time.
- Legislation, sections and case references. It will invent a section of the Companies Act that sounds perfect. Go to the source.
- Quotes and who said them. Frequently reassigned to a more famous person.
- Anything recent. These tools have a cut-off date. Ask about last month's SARS change and you may get an answer built on old information, stated with total confidence.
- Links and sources. Plausible-looking URLs that go nowhere are common. Click every one.
If being wrong about it would embarrass you in front of a client, cost money, or land on a document with your name at the bottom, check it against the actual source. Not a second AI tool. The source.
How to catch it
Ask how sure it is
Try: "Which parts of this are you confident about, and which should I verify?" It's surprisingly good at flagging its own soft spots when directly asked. Not perfect, but far better than volunteering it unprompted.
Ask for the source, then go and look
Requesting a source does two things: it makes the tool more careful, and it gives you something to check. The checking part is not optional. A confidently cited source that doesn't exist is the classic failure.
Notice when it agrees too easily
Push back on something correct and it will often fold and apologise. That's a design tendency toward agreeableness, not a signal that you were right. If it caves the moment you frown at it, that tells you nothing about the facts.
Watch for the too-neat answer
Real professional situations are messy and full of "it depends." An answer that's suspiciously tidy, with everything in threes and no caveats, is often a sign it's producing a shape rather than an analysis.
Where the risk is highest
The danger isn't uniform. It scales with how much you'd be able to tell.
- Low risk: rewriting your own email, summarising your own document, brainstorming, improving something you wrote. You'd notice if it went wrong.
- Medium risk: drafting something you'll review properly before it leaves your hands.
- High risk: anything factual you can't verify, in a field you don't know, that someone will rely on.
The question isn't "is it accurate?"It's "would I be able to tell?"
One practical habit
When AI has done meaningful work on something that carries your name, keep a note of what it did and what you checked. Not for anyone else, for you, six months later, when someone asks how a figure was arrived at.
Professional bodies are still working out their positions on disclosure, and requirements differ by industry. If you're in a regulated profession, check your own body's guidance rather than assuming. The habit of knowing what you leaned on costs nothing and may matter a great deal.
The balanced view
None of this is a reason not to use these tools. Cars are dangerous and we all drive.
It's a reason to use them where you're the expert, check what matters, and never let something leave your desk that you haven't actually read. That's not an AI rule. That's the rule your work experience already taught you.