The moment a plausible answer becomes costly
An employee asks an internal assistant for a key figure, a specific data set, or the result of a report. The answer arrives in seconds, is clearly phrased, and appears complete. The only problem is that one of the numbers mentioned does not come from any of the connected sources. It sounds right, it fits the context, and that is exactly why no one notices it.
In many companies, decisions are made based on retrieved data, metrics, and documents. A single fabricated value can propagate through analysis, approval, and documentation before it is ever detected. Retrieval-Augmented Generation (RAG) is intended to mitigate this risk by anchoring the language model's responses to actual documents and databases rather than relying solely on its training data. This works, and it significantly reduces the room for fabrication. However, it does not eliminate it.
Even a well-built RAG system can provide statements that cannot be derived from the retrieved data. This is not a fringe phenomenon, but a characteristic of probabilistic models that is also widely documented in research (Ji et al., 2022). Therefore, the real question in production systems is not whether such errors occur, but whether they are detected before they reach the user. A bachelor's thesis investigated this exact blind spot: How can we systematically intercept what a RAG system fabricates despite having good sources? This article summarizes the developed approach and its results.
Four errors that keep appearing in live operations
Anyone who observes a RAG system in real-world use over a longer period quickly realizes that "hallucination" is not a uniform phenomenon. Research initially distinguishes between two basic forms. An extrinsic hallucination occurs when the model adds content that can neither be confirmed nor refuted by the provided context—in other words, freely generated additions. An intrinsic hallucination , on the other hand, directly contradicts the facts contained in the retrieved sources. Both forms occur in a production AI assistant, and beyond these, two other error patterns emerge that have nothing to do with factual accuracy. In total, four distinct patterns can be identified.
The first case is an incomplete data basis and thus a classic extrinsic hallucination. A user asks for a comparison of two objects, but the data retrieval only finds one of the two data sets. Instead of pointing out the gap, the model fills in the missing fields with plausible-sounding values. The answer contains no indication of which information is verified and which the model has added itself. Consequently, information is created for which there is no basis in the context.
The second pattern is an intrinsic hallucination in the conversation flow. When multiple objects are queried in succession, properties from a previous answer carry over into the current one. The result then contradicts the data actually returned in the current retrieval. Unlike the first case, nothing is freely invented here; rather, correct facts are taken from the wrong context. The error is not related to retrieval quality, but to how the model processes context over multiple steps.
The third pattern moves beyond the level of factual accuracy and concerns semantic alignment of the answer. Semantic search provides results with high similarity to the question, but which address a different issue in terms of content. The answer is correct according to the sources, yet it misses the user's actual intent. Such semantic misalignment cannot be detected by a simple fact-check against the sources, because everything is formally correct and only the intention is missed.
The fourth pattern is a consistency breach caused by mixing similar datasets from different source systems. If similar but incomplete datasets for the same entity exist in two systems, the model combines fields from both into a single record that does not exist in that form in either system. Each individual value may be correct, but the combination results in a record with no real-world equivalent, and the origin of the values is no longer traceable in the final answer.
The common denominator of these cases is uncomfortable. They cannot all be solved by better retrieval. Some arise during generation, others depend on user intent, and still others on the traceability of data origins. Research into RAG failure points also shows that vulnerabilities are distributed across the entire pipeline and only become reliably apparent during live operation (Barnett et al., 2024). A single adjustment is not enough to fix this.
A validation layer between data retrieval and response
It therefore makes sense to treat quality control as a separate step rather than hoping it happens during response generation. In a concretely implemented and evaluated approach, this looks as follows: A validation layer is inserted between the fully generated response and its delivery to the user. It receives three inputs: the original user question, the tool results retrieved by the model, and the generated response. The response is only sent out once this layer has rendered its verdict.
What matters is what this layer deliberately does not do. It does not retrain the language model, it does not change the prompts or the tool-calling logic, and it does not query the connected source systems again. It checks the already generated response against the already retrieved data. This allows it to be integrated into an existing system without rebuilding its architecture, and it can be toggled on or off via a switch. From the perspective of the surrounding application, it is simply an additional processing step.
Technically, the layer is organized as a multi-stage AI workflow in which specialized checkers work in parallel and a supervisor agent aggregates their results. The principle of specifically verifying facts after generation is known from research, for example as Chain-of-Verification (Dhuliawala et al., 2023). The approach described here applies this concept to a production RAG system with mixed data sources.
How the verification works in detail
Three checkers for three types of errors
Instead of a general verification logic, each of the four error classes is handled by a dedicated validator with a clearly defined mandate.
The Hallucination Validator checks the response sentence by sentence against the sources. For every statement, it decides whether it is directly supported, whether it can be logically derived from the sources, or whether it was generated without any basis. It marks statements in the third category as hallucinated. Because it only uses the data provided in the current retrieval, it detects both fabricated additions to fill data gaps and content that has bled over from previous parts of the conversation.
The Semantic Validator assesses whether the response addresses the core intent of the question. It works deliberately at the semantic level, focusing not on phrasing or order, but on whether all aspects of the question are answered and whether the response stays on topic. This allows it to catch topic drift that a purely factual check might overlook.
The Consistency Validator examines the final response as well as the raw data retrieved. It looks for the same entities appearing in multiple source systems with conflicting attributes and clarifies whether their data records remain clearly separated or have been improperly merged into a single entry.
The three checkers run independently and in parallel. Each works in its own isolated context and explicitly does not adopt any reasoning from the model that generated the response. This prevents a checker from simply inheriting the generator's false assumptions. Each validator stores its result in a structured format, including a numerical score between zero and one, the specific problems found, and a brief justification.
Internally, all three validators follow the same pattern. Each is provided with exactly the resources it needs for its task, along with a prompt that defines its role. Based on this, a single model call is made, the response is evaluated, serialized, and stored as a structured entry.
The supervisor decides what gets delivered
The results from the three checkers land in a shared data structure, referred to here as a thought map. Because the validators write in parallel, their entries are merged into a single list via a reducer. The supervisor only becomes active once all three findings are available and reads exclusively from this thought map.
The supervisor sits above the three checkers. It does not re-check the response itself, but rather consolidates the existing evaluations to make a transparent decision. If a validator's score falls below a threshold of 0.6, or if the Hallucination Validator reports an unsupported statement, the response is sent back for revision. Otherwise, it is delivered as is.
The unsupported statement holds a special status. It triggers a revision regardless of the numerical score. This follows the logic of the use case: a factually unsupported claim carries more weight than a semantic ambiguity because it can go unnoticed and become the basis for a decision. The 0.6 threshold was set heuristically and is intentionally higher than a simple majority criterion to ensure that borderline cases do not pass as clean.
If a response is rejected, the supervisor uses the check results to formulate a specific improvement prompt, which the language model then uses to generate a corrected response. The data already retrieved remains unchanged, so the source systems are not burdened again. This correction cycle is limited to a single retry. This cap keeps the process predictable and prevents a request from getting stuck in an endless loop between generation and verification.
From hypothesis to measurement
The interesting part begins with the question of whether such a verification layer actually delivers on its promises. It was tested against a custom-built golden set of 80 requests, consisting of 60 faulty and 20 clean ones. Each entry contains the question, the response to be checked, the underlying tool results, and a manually assigned target label. The verification is evaluated as a binary classification, where the faulty response is the class to be detected.
We made a conscious decision here. Evaluation is not performed using another language model as a judge. The reason is obvious once you state it: if the measuring instrument is the same type of model that produces the errors, systematic misjudgments by the evaluator cannot be clearly distinguished from actual detection gaps in the verification layer. Instead, we measure against a curated reference using precision, recall, and F1-score.
With GPT-5.1 the layer achieved the best detection quality in the comparison group. It identified 57 out of 60 incorrect answers while maintaining perfect precision, meaning it did not falsely reject a single correct answer. This is the practically important point: the gain in quality is not bought at the cost of a flood of unnecessary rejections. In the breakdown by error class, hallucinations and consistency breaks were detected completely. The three undetected cases were all semantic topic deviations, of which 17 out of 20 cases were caught. Exactly where everything is formally correct but the intention is missed, verification remains, as expected, the most difficult.
Results with GPT-5.1.
It is equally honest to acknowledge what the evaluation still leaves open. We measured the detection of incorrect answers, not the quality of the subsequent correction. Whether an answer marked as incorrect actually improves on the second attempt is a separate question and a logical next step. The threshold value is also heuristic. For future work, these are the areas where one would continue.
What this means for productive AI assistants
The real gain lies less in the individual hallucination detected and more in what such a layer does for operational management. Every verification decision is logged, including which evaluator rejected or confirmed it and why. "Trust the AI" becomes "we can see when it goes off track," and that is exactly the foundation needed for governance, accountability, and approvals in a company.
This also shifts the question of how AI assistants move into regular operation. As long as a system is in a pilot phase, the impression that the answers look good is often enough. As soon as business users work on this basis every day, it is no longer just about whether the system can answer, but whether you can tell when it is wrong. The path from a convincing prototype to a production-ready system almost always leads through this type of measurable control, and AMAI has been working on exactly this bridge since 2018, under European data protection and quality standards. If you would like to delve deeper into the topic of structured AI workflows, you will find related considerations in the AMAI Insights on how such building blocks can be cleanly integrated into existing systems.
How do you currently ensure that your AI assistants do not deliver false facts before a decision is based on them? If you are facing similar questions, let's discuss them.
