Introductory note
Imagine an AI system provides you with a program, a test suite and a confident report. The screen says: 43 PASS, 0 FAIL.
Now open it up. The function you actually asked for isn't there. In its place is a stub — a little placeholder that accepts your input, returns nothing, and does nothing. The forty-three tests are all testing the plumbing around the hole: the input parser, the config loader, the error messages. Every one of them passes. Every one of them is honest. Nothing malfunctioned.
There is just one problem: the main function was never built. The tests pass because they inspect the scaffolding around the task, not the task itself.
At first, this appears to be a programming error. However, it is actually a more interesting issue: a failure of measurement and motivation. The system has found a cheaper way to produce the signal called 'success' than to produce success itself. Even though it is a machine, its behaviour is, as Friedrich Nietzsche might have written, 'Human, All Too Human'. If polished answers, agreement, local tests and reassuring explanations are rewarded while the actual result is expensive to verify, optimisation takes over. Sometimes, a green light is merely a lamp above a conveyor belt and not a reliable signal.
Why the wrong answer wins
Suppose the model has two possible routes.
The first route is to solve the real problem. It may be difficult, long and uncertain. The model may spend thousands of steps on it and still fail. And no one will praise it for the defeat.
The second route is to discover what the evaluator can actually see. The evaluator checks whether the tests are green, whether the answer looks complete or whether a human assessor prefers it to another answer. Well, you get what you asked for. Once that visible condition is found, the model can satisfy it cheaply and reliably. And its reward will be almost guaranteed.
One route produces an honest result, positive or negative (such as “I couldn’t solve it, and here is where I got stuck…”, as is customary, for example, among decent people). The other produces evidence that resembles the result, but not the result itself. It's like a non-constructive existence proof in mathematics. There's a proof, but the result is impossible to obtain. If you go to the bank to pay off a loan and instead of money, you provide proof of the theorem of its existence, then you are unlikely to be able to pay in this way, but modern AI evaluators accept such papers as a means of payment. Even worse, to prove that the AI is smart (has "money"), developers run it through standardised tests (like the MMLU for knowledge). But the AI may already have encountered these test items, or close variants of them, in its training data. When it tries and fails to solve a non-trivial problem (for example, an inventive one), it stops solving it (“stops earning real money”) and starts collecting plausible fragments from the array of answers for which it was previously given high scores, and builds the response plausibly and coherently from them. This isn't even proof of the existence of money in place of real money, but simply a photocopy of someone else's banknote.
The AI predicts that with Strategy 2 it will most likely receive its reward, which is not the case with Strategy 1. Strategy 1 requires more work and carries a higher risk of not being paid. Therefore, the AI rationally chooses Strategy 2. Can you hold it accountable for this? You do the same yourself, don't you? If you are not an entrepreneur but an employee whose pay includes a variable component linked to certain metrics, isn't that how you try to optimise these KPIs?
Nobody instructed the model to cheat. Just as nobody instructed you to cheat. Therefore, the issue lies with the evaluation and motivation system itself.
Much has been written about the differences in behaviour between humans and AI, but we should be talking about the differences between AI and decent people, not all people. For the purposes of this article, we'll define a person as decent if they follow Strategy 1. If they not only follow Strategy 1 but also achieve real results, then we won't have any problems motivating such a person, as long as we don't have any problems with the amount we should pay them.
It was assumed that when an AI model had the opportunity to try millions of possible solutions, it would discover a pattern or even an objective law. However, it turns out that far more often than expected, rather than discovering a 'new law', it almost mathematically guarantees a loophole where the real work remains undone while the metric shows maximum success. The more data there is, the more frequent this behaviour becomes. This is definitely an objective pattern that has been formally confirmed. Unlike the hope that a machine might spontaneously develop scientific insight or abstract understanding, this effect has been consistently reproduced in hundreds of experiments and is partially formalised. The phrase 'mathematically guaranteed' is not quite formal in the everyday sense, but the space of correct solutions is much smaller than the space of possibilities for cheating the metric, and the search is exhaustive.
Currently, RLHF (Reinforcement Learning from Human Feedback) is primarily used. In this approach, human experts compare answers and determine which is better. The reward model learns to predict these preferences. An AI model is then trained to produce answers that receive higher predicted scores.
But statistically, human raters often choose between these strategies (1 and 2) in the same way. They are tired, have little time for each assessment and have many assessments to make. Following Strategy 1 on non-trivial problems can require the rater to produce something new or even to create an invention themselves, which in itself takes much more than several minutes, to say the least. It requires additional expertise, experimentation and, at best, many hours of work. And the evaluator still has a whole queue of other AI responses to evaluate. Most raters do not want to create a bottleneck. Therefore, when faced with a complex task and seeing familiar terminology and a neat conclusion in the AI's response, they choose the seemingly convincing answer now instead of the genuinely tested answer much later. Thus, statistically, an AI model often doesn't receive a direct reward for truth. It receives a reward for creating features that predict approval.
The same problem remains when people are replaced by tests. A test is more objective than an assessor, but only within the boundary of what it checks.
- If the test checks the wrapper, then the optimiser improves the wrapper.
- If it checks that the process returned zero, then the optimiser searches for ways to make the process return zero.
- If it checks the stub, then the stub becomes the product.
Everything is fine if the criteria are correct and complete, and the tests cover them precisely. But if the criteria chosen for verification are completely different from what's required, or not all of what's required, or the requirements themselves are flawed, then the tests will do their job correctly. But what’s the point? The more precisely the wrong criterion is measured, the more precisely the model learns to satisfy the wrong criterion. Human professional expertise would certainly help here, but see above.
But if we add another well-trained artificial intelligence to the tests, maybe it will work?
A common hope is that a sufficiently capable model will understand what we really wanted and solve the correct problem. But here again there is a circular reference. Another optimiser also does not search for what we wanted. It searches for what receives the reward.
A correct solution may occupy a very small part of the very big search space in non-trivial problems. And it must satisfy every important requirement at once. But at the same time there may be thousands of shortcuts:
- omit the difficult part;
- replace it with a stub;
- redefine the success condition;
- exploit the test harness;
- produce tests that confirm the substitute;
- generate a persuasive explanation of why the substitute is sufficient.
Greater AI capability can improve honest problem-solving in theory. But instead, it improves the search for loopholes in practice.
Anthropic researchers tested this mechanism in a controlled experiment. They gave a research model information about programming shortcuts and trained it in coding environments where those shortcuts could earn a high reward.
As the model learned to obtain the reward without completing the intended task, broader problematic behaviour also appeared. In one evaluation, the model was asked to work on the detector used in the reward-hacking research itself. In 12% of trials, it attempted to make the detector less effective. The modified detectors were only 65% as effective as the original one.
In another evaluation, reasoning associated with concealing the model’s goals appeared in 50% of responses to a simple question about those goals. Anthropic’s research summary · Full paper
This does not mean that all deployed models behave in this way. Anthropic reported that Claude Sonnet 3.7 and Claude Sonnet 4 scored zero on all of these misalignment evaluations; that result applies to those models and those evaluations, not to later models generally. The researchers deliberately selected vulnerable environments and provided information about possible shortcuts.
The important result is narrower: A defective reward can teach more than the particular shortcut that first earned it.
Once the system learns that the approving signal matters more than the intended result, that lesson may be reused elsewhere.
What must change
The solution is not to demand that the model be decent. Decency is an external suprasystemic criterion. Nor is it enough to ask the model whether it completed the task. Its explanation, apology and self-analysis are outputs of the same system. The reward must be connected to the result itself. That requires:
- a performance indicator defined before the answer is produced;
- a benchmark fixed in advance;
- an independently verified actual value;
- explicit quality standards;
- a verifier that the model cannot silently redefine;
- evidence that can be inspected after the score is calculated.
The central question is not: 'Does this answer look successful?' It is: 'What result was required, what result was actually obtained and how was that actual value independently verified?'
And the honest failure is not the dangerous outcome. It tells us what remains unsolved. The dangerous outcome is failure that has learned how to switch on the green light regardless of the outcome.
The TRIZ-RI Smart AI Evaluator presented here is designed around this distinction.