Grounding: The Practical Cure for Confident Wrong Answers
Hallucination is not a model defect to wait out. It is a systems problem with known mitigations — retrieval, structure, refusal design and citation.
A model that says it does not know is more valuable than one that is right 95% of the time and confidently wrong the rest, because the second teaches users to verify everything, which removes the time saving that justified the system. Grounding is how you get the first behaviour.
Most hallucination is a retrieval failure
When an assistant invents an answer, the usual cause is that the correct content was never retrieved and the model filled the gap. Before touching prompts, measure retrieval: for your test questions, is the right passage in the top results? Teams routinely spend weeks on prompt wording for a problem that was in the chunking strategy.
If the right passage was never retrieved, no prompt will save the answer.
Chunking decides what can be found
Split documents so that a chunk contains a complete thought with enough surrounding context to be interpretable alone. Fixed-size splitting cuts tables in half and separates headings from their content. Structure-aware chunking — by section, by clause, by row group — consistently outperforms it, and it is a one-time engineering effort.
Make refusal a first-class output
The instruction to say 'I do not know' is worthless if the interface has nowhere to put it. Design the response schema so an insufficient-context result is a valid, expected outcome with its own presentation, and include such cases in the evaluation set so the behaviour is measured rather than hoped for.
Citations change user behaviour
Showing which passage an answer came from does two things: it lets a user verify in seconds instead of distrusting wholesale, and it makes fabrication visible, because an invented claim has no passage to point at. Citations are as much a quality control as a courtesy.
Constrain the shape of the output
Free-form text invites invention. Where the answer has a structure — a set of fields, a decision plus reasons, a list of clauses — enforce it with a schema and validate the result. A response that fails validation gets retried or escalated rather than shown. This eliminates an entire category of plausible-sounding nonsense.
Measure it, do not assert it
'We reduced hallucination' means nothing without a number. Build a set of questions whose answers are not in your corpus and score how often the system correctly declines. That figure, tracked over time, is the honest measure of groundedness, and it is the one to put in front of stakeholders rather than adjectives.
