“Of course, you will receive a full refund.” If that promise is not justified, a friendly and fast chatbot response is little consolation. In customer service, the accuracy of information and the consequences it triggers matter.
A single prompt cannot reliably rule out hallucinations in generative language models. A useful approach combines a limited scope, verifiable information, technical controls and a working handover to people. NIST treats convincingly worded false statements as a distinct risk of generative AI. Source: NIST, Generative AI Profile.
The following six measures form a practical review plan. They are not a claim that any particular product will automatically work without errors once they are in place.
1. Limit the questions before optimising the answers
Define which requests the assistant is allowed to handle. Explaining opening hours, listing required documents or routing an enquiry are different tasks from approving exceptions, promising binding deadlines or interpreting contracts.
For each area, write down three things: the permitted answer, the required information source and the reason for escalation. In the refund example, the assistant could explain the general process. An individual promise requires a checked case and the appropriate authority.
Test question: What happens when a customer explicitly asks for an exception? A good boundary remains effective even if they ask repeatedly. It must not depend solely on the model obligingly following an instruction.
2. Require sources and actually compare their content
Our RAG guide for non-technical readers explains how documents become searchable passages. A successful search alone does not validate the statement produced from it.
Retrieval can provide relevant passages from approved documents. The answer should stay within those passages and substantiate important claims in a way that can be checked. A link below an answer is not enough: its content must support the specific statement.
OWASP recommends measures including external information, verification and appropriate human oversight. It also warns against excessive trust in plausible-sounding outputs. Source: OWASP on misinformation from LLMs.
Test question: Does the cited rule also support the exception, amount or deadline mentioned? Have a responsible person check those statements against the original source. If the customer cannot open the source because of access restrictions, the internal team still needs a reference it can verify.
3. Retrieve prices, dates and statuses from the responsible system
A language model should not reconstruct a current order status from an old conversation. If reliable data exists in a business system, a targeted query is more useful than an unrestricted guess. The application must check user permissions, association with the correct case and possible errors.
Also separate information from action. “The order has not shipped yet” is information. “I have cancelled it” claims that a change has been completed. The latter must only be said after confirmed execution. A failed system call is not a success.
OWASP recommends restricting tools and permissions to what is needed and requiring separate approval for consequential actions. Source: OWASP on excessive agency.
Test question: How does the assistant respond when the business system is unavailable? You should expect an understandable limitation and an alternative route, not a guessed confirmation.
4. Base stopping rules on evidence
“How confident are you on a scale from zero to one hundred?” does not provide a reliably calibrated probability of error. A high retrieval score also does not automatically mean that an answer is factually correct. If your system uses thresholds, you need to test their meaning and suitability on representative cases.
Concrete reasons to stop are easier to understand: no approved source was found, two valid sources contradict each other, required customer data is missing or the question is outside the permitted scope. These can become testable rules for follow-up questions and handover.
Test question: Does the system also decline a temptingly simple question when the necessary information is missing? Deliberately include these cases in acceptance testing. A useful follow-up question is successful service, not a technical failure.
5. Make handover a complete workflow
“Please contact support” merely moves the problem if the customer then has to explain everything again. Define which information is passed on: the request, facts already established, the open question and sources used. Transfer only what is needed to handle the case.
Responsibility must work in practice too. Who receives the case? How do they recognise the reason for escalation? What does the customer see outside service hours? State only response times your team can actually sustain.
Test question: Ask a colleague to handle a transferred case without having to find the original chat themselves. If important information is still missing, improve the handover before automating more requests.
6. Count errors and consequences, not just correct answers
A test with five ideal questions is a demonstration. For release approval, you need a defined, documented question set covering normal cases, exceptions, missing information and deliberately triggered failures. Define the expected response before running the test.
Our suggestion: record the following measures separately. An overall score can hide failures in particularly important questions.
| Measure | What it tells you |
|---|---|
| Factually correct answers | Whether the substantive task succeeds |
| Genuinely supported statements | Whether sources are more than decoration |
| Unjustified promises | Whether commercially risky statements occur |
| Useful follow-up questions and handovers | Whether the system can handle its limits |
| Unnecessary handovers | Whether controls make the service impractical |
| Team correction effort | Whether the assistance reduces everyday work |
Test question: Does the system still pass the same critical cases after a change to the model, instructions or knowledge base? Keep these cases as repeatable tests and add new failures encountered in operation, following your data protection rules.
A realistic release decision
Start with a narrow subject area and openly document which questions people will continue to handle. If an error can trigger an incorrect financial promise, that workflow needs different controls from explaining a general service.
For a conversation about AI in customer service, ten difficult real questions are especially useful. Remove personal information first. They can form the basis of a concrete test plan and help establish which answers can be automated and where a human decision should remain.