Initial Situation
A major IT service provider in the financial sector faced the challenge that the maintenance and further development of traditional machine learning pipelines for document classification were consuming increasing resources. The existing systems for processing real estate certificates, property brochures, and bank statements required extensive manual labeling, complex feature engineering, and strict data lineage management. Any change to document structures or the addition of new classes resulted in lengthy training phases.
The company recognized the strategic opportunity to fundamentally modernize these processes by leveraging modern Generative AItechnologies. The goal was to evaluate whether Large Language Models (LLMs) could replace traditional classifiers to minimize maintenance efforts while maintaining or improving classification quality. A key focus was determining whether the massive overhead for data preparation and training could be eliminated by using pre-trained models.
Our Solution
AMAI developed a comprehensive evaluation framework and a Proof of Concept (PoC) to validate the suitability of Large Language Models for complex multi-page document classification. The approach was based on a systematic comparison between traditional methods and modern GenAI approaches using publicly available data.
- Systematic evaluation framework for model comparisons:
To create an objective basis for decision-making, a flexible framework was developed that enables the direct comparison of different model architectures. This system allows both proprietary cloud models and local open-source models to be benchmarked against each other. By using a custom-created C1 dataset of publicly available documents, the evaluation could be conducted without data protection hurdles or access to sensitive financial data, which significantly accelerated the project timeline. - Sliding window approach for long documents:
One of the greatest technical challenges was processing extensive documents that exceed the context window of many models or lead to a degradation in recognition performance (the "lost in the middle" phenomenon). The solution involved implementing an intelligent sliding window method. In this process, documents are divided into overlapping segments, analyzed separately, and then combined into an overall result using aggregation logic. This ensures that no relevant information is lost, even with very long property brochures or contracts. - Structured data extraction via JSON output:
To convert the often-variable output of generative models into a process-reliable format, the project consistently relied on JSON structured output. This technique forces the language model to return results in a precisely defined schema (e.g., "Page 1: Floor plan", "Page 2: Energy certificate"). This eliminates structural hallucinations and enables seamless downstream processing of classification results in IT systems without the need for complex parsing logic. - Focus on prompt engineering instead of fine-tuning:
During the project, it became clear that extensive fine-tuning of local models for this specific classification task offered no significant added value compared to effective prompt engineering with powerful base models. The solution therefore focused on optimizing prompts and providing relevant context (few-shot learning), which drastically reduced pipeline complexity and increased flexibility for future adjustments.
Results & Business Impact
The research project provided clear evidence that generative AI models can not only replace traditional machine learning pipelines for document classification but also outperform them in terms of efficiency. The evaluation showed that modern LLMs deliver comparable or superior classification results while massively reducing development and maintenance efforts.
Particularly significant was the near-total elimination of manual labeling, which had been the most time-consuming factor in traditional methods. Since the models operate in "zero-shot" or "few-shot" modes, the need for lengthy training cycles was removed. This allows for a drastic reduction in time-to-market for new document types. Based on these findings, a clear recommendation for a technological realignment was issued, which the client subsequently incorporated into their strategic roadmap and implemented. Validation also confirmed that the use of structured output ensures the process reliability required for deployment in a regulated financial environment.
