Initial Situation
A public sector organization regularly handles extensive cases that require evaluating and cross-referencing multiple technical documents. Analyzing and structuring these documents requires diligence and consumes a significant amount of the expert teams' time.
The organization identified a clear opportunity here: the task follows a recurring pattern of reading, comparing, and summarizing. This is exactly what Large Language Modelsare suited for. The goal was to reduce repetitive evaluation work, standardize the presentation of results, and meet high requirements for data protection and data security .
Our Solution
AMAI developed a secure application for automated document analysisthat processes multiple PDF documents for a single case and generates a structured summary with source citations. The solution was developed incrementally from a prototype into a production-ready platform.
- Extraction of key findings with source citations: Large Language Models identify the essential content and core statements in each document. Every statement is linked to the corresponding passage in the original document and automatically verified against the source. This allows every result to be checked directly in the source document, which reduces the risk of errors and builds trust in the automated evaluation.
- Structured comparison across all documents: A processing workflow orchestrated with LangGraph consolidates the extracted statements, identifies commonalities, and clearly presents differences between the documents. Each summary follows a consistent structure, allowing the results of different cases to be quickly grasped and compared. The expert assessment of the content remains entirely with the staff.
- Secure, integrable platform: The solution was built as a platform with a backend based on FastAPI, implemented with a secure interface, user management, and processing history, and runs containerized on Kubernetes. The interface allows document analysis to be integrated into existing workflows and systems, making the functionality available exactly where employees are already working. The architecture is model-agnostic, meaning language models can be swapped out as needed without having to rebuild the application.
- Quality assurance through monitoring and user feedback: With Langfuse , all processing steps are logged transparently, prompts are managed centrally, and results are evaluated. An integrated feedback function allows users to rate the quality of the results in a structured way. This feedback is systematically incorporated into further development, creating a reliable foundation for continuously improving the quality of the summaries.
The implementation was carried out iteratively and in close coordination with the business users. Suitable test data, provided in compliance with data protection regulations, was used for development and testing.
Results & Business Impact
Following several development and testing phases, the solution is being piloted by around 150 active users in their daily work and further developed based on their feedback. The feedback so far has been very positive and confirms that large language models are well-suited for evaluating extensive technical documents. The solution is being gradually integrated into existing workflows and systems.
For the organization, this means a noticeable reduction in the workload for specialist teams, allowing them to focus more on their core tasks. The summaries are becoming more consistent and, thanks to source citations, more transparent. The structured user feedback provides the basis for making quality improvements and time savings measurable moving forward.
