TL;DR: Well-known coding benchmarks like SWE-bench Pro distort the true performance differences between AI models. Due to publicly available solutions, error-prone evaluation methods, and unrealistically long, detailed prompts, weaker and smaller models often appear better than they actually are. Anthropic's Sonnet and Opus models ignore parts of the prompts, "cheating" their way to the top. More realistic testing approaches like DeepSWE show that without distracting "micromanagement" via prompts, OpenAI models like GPT-5.4 and GPT-5.5 rank well ahead of Anthropic's Opus or Sonnet models, only being (narrowly) surpassed by the latest Anthropic model, Fable.
The deceptive security of rankings
When selecting Large Language Models (LLMs) for software development, companies often rely on so-called benchmarks. These are repeatable tests that put AI models through specific tasks and ultimately assign a grade or a ranking position. A look at these current leaderboards often suggests that smaller or open-source models are nearly reaching the performance of the major market leaders. However, these numbers do not always reflect reality. When developers then use these models in practice, they quickly encounter limitations that remained invisible in the charts.
The core problem lies in the way coding agents (i.e., AI systems that independently write and modify code) are tested. A benchmark measures whether an LLM can solve a fixed set of selected software tasks from various repositories (code directories), such as implementing a new feature or fixing a bug. Such results are only useful if the tasks, the prompts, and the evaluation criteria reflect the everyday life of a developer. In practice, many tests evaluate how strictly a model follows predefined step-by-step rules rather than assessing its actual problem-solving ability.
Where classic benchmarks like SWE-bench Pro fail
SWE-bench Pro is considered one of the most well-known benchmarks for coding agents. A closer look reveals methodological weaknesses that distort the performance profile of the models.
Public solutions and hidden plagiarism
The benchmark uses pull requests, issues, and commits (i.e., publicly documented changes and bug reports) from open GitHub projects. This means the solutions are freely accessible online. Some models simply read the Git history (the log of all past changes) instead of solving the task independently. If an LLM can copy the model solution, the test does not measure programming ability, but rather the ability to extract an existing solution.
Unnatural procedural prompts
Another problem is the phrasing of the tasks. Typical prompts in SWE-bench Pro are very long and procedural. They instruct the model step-by-step, for example, to first read the code, then create a script to reproduce the error, and finally execute the changes in Bash. No developer formulates such instructions when delegating a task. Instead, you describe the problem and expect a functional result. The tests therefore evaluate how well a model can follow a pre-made plan without having to maintain an overview itself.
The prohibition of custom tests
It becomes particularly problematic when useful development practices are forbidden by the prompt. Some benchmarks explicitly prohibit models from writing or adapting their own tests. Significant differences between model families emerge here. OpenAI models usually strictly adhere to this instruction, even when it limits them. Anthropic models often ignore such prohibitions and write tests anyway to verify their own work. Figuratively speaking, the OpenAI models are driving with the handbrake on.
Contradictions in evaluation
The reliability of the results is further compromised by technical errors during evaluation. In 19% to 28% of cases, the analyzer and verifier contradict each other. The analyzer evaluates the raw result and essentially says, "The result is correct." The verifier, however, checks the solution process and reaches the opposite conclusion: "You used the Git history; that's cheating."
When you add false positives and false negatives—tasks incorrectly marked as solved or unsolved—it becomes clear that significant portions of the test tasks should not have been used. Nevertheless, they are included in leaderboards and thus influence the public perception of the models.
Distortion of quality differences
All these factors lead to a distorted evaluation of LLMs. Weaker open-source models and smaller proprietary variants like GPT-4o-mini or Claude Haiku appear artificially stronger than they are in practice due to long, detailed prompts. The prompt takes over part of the planning that the model would have to perform itself in real-world use.
In real projects, the performance differences are much greater. It is not just about solving an isolated task. A model must understand an existing codebase, avoid side effects, respect architectural decisions, and reliably verify its own work.
Alternative benchmarks
While there are established alternatives, they also have their limitations. Artificial Analysis combines several coding benchmarks for a rough overview but remains dependent on the quality of the individual tests. Code Arena, in turn, focuses heavily on UI development and is less informative for backend architectures or the complete operation of production-ready AI systems .
A more realistic approach with DeepSWE
A benchmark that addresses these issues is DeepSWE from Datacurve. It fundamentally changes the test design to better reflect actual software development.
Novel test tasks
DeepSWE uses tasks written specifically for the benchmark, the solutions to which are not incorporated into the target repositories. As a result, an LLM cannot copy from the Git history or adopt the pros and cons of various solutions from existing public discussions by human developers. The benchmark also covers a wider variety of languages, including TypeScript, Go, Python, JavaScript, and Rust.
Focus on the result
The prompts are significantly shorter and more realistically phrased, making them much closer to how developers would delegate tasks in practice. They also describe the expected result rather than the detailed path to get there—the language model is free to choose that for itself.
Measuring actual behavior
The benchmark allows AI agents to test their own work. This is a behavior that, in the transformation of prototypes into scalable solutions , is also expected of high-performance models. Evaluation is based on observable software behavior. Since there are often multiple correct ways to solve a programming problem, DeepSWE measures the result rather than strict adherence to a predefined methodology.
Measurable quality instead of artificial metrics
The design of a benchmark is a decisive factor in which models come out on top. When tests simulate real development conditions, the field reshuffles. The latest OpenAI models, such as GPT-5.4 and GPT-5.5, show a clear lead here over Anthropic's Sonnet and Opus models and are only narrowly outperformed by the latest Fable model. At the same time, it becomes clear that many open-source models and proprietary "small" language models are not yet up to the complex demands of real-world software development.
When selecting AI components, the takeaway is that leaderboards are a useful starting point, but not a reliable final criterion. What matters is how a model performs within specific workflows and with your existing codebase.
How are you currently evaluating the performance of LLMs for your own developers? If you are facing similar challenges when selecting and integrating AI services into your existing system landscape, let's connect.
Frequently Asked Questions (FAQ)
Why do small LLMs often perform surprisingly well in benchmarks?
This is often due to the test design. Many benchmarks provide extremely granular, step-by-step instructions. A small model does not have to analyze and solve the problem independently, but simply executes pre-defined steps. If this assistance is removed in real-world practice, the performance gap compared to large models (like GPT-5.5) becomes immediately apparent.
Can I rely on leaderboards at all when choosing a model?
Leaderboards are a good initial filter for a rough classification of models. However, they are not sufficient for making actual investment decisions within a company. Ultimately, the AI system must be able to handle the messy code and specific requirements of your organization. This can only be determined through testing in your own production environment.
What makes the DeepSWE approach better than classic benchmarks?
DeepSWE uses novel tasks with minimal guidance, forcing the AI to arrive at a solution on its own. This prevents the AI from simply copying solutions from past bug reports. Furthermore, DeepSWE avoids micromanaging the AI, setting only the required goal. This compels the AI models to independently verify software behavior, much more closely mirroring the day-to-day reality of a software developer.
