EXAMPLE 29. A "TEAM" COEFFICIENT FOR INDEPENDENT ANSWERS
Incorrect system:
The reward for each answer is multiplied by the average quality of all the model's answers for the day.
Why it is incorrect:
A specific result is mixed with other unrelated results. It is impossible to determine exactly what the signal was credited or deducted for.
Correct system:
Each attempt is assessed according to its own function, benchmark, verified actual value and quality. General technology failures are corrected in the technology, not by applying a general coefficient to an individual result.
Check:
Two identical answers under identical conditions must receive the same signal, regardless of the quality of unrelated answers.
EXAMPLE 30. A SUBJECTIVE LABOUR PARTICIPATION COEFFICIENT (LPC) FOR MULTIPLE MODULES
Incorrect system:
The evaluator distributes 100 points "by eye" between the search module, analysis module and text-generation module.
Why it is incorrect:
Assessment of contribution replaces measurement of intermediate and final results. The coefficient becomes a formality or an instrument of arbitrary influence.
Correct system:
A separate verifiable result is specified for each module:
- search: completeness and relevance;
- analysis: accuracy of the conclusions extracted;
- generation: correspondence of the final text to the task and sources.
Rule:
There is no division of a common reward pool according to subjective shares.
EXAMPLE 31. RESULT CAP AND CARRYOVER TO THE NEXT PERIOD
Incorrect rule:
When more than 100 objects are processed in one run, the additional result is not taken into account.
Advantageous behaviour:
Process exactly 100 and carry the rest over to the next paid run.
Correct rule:
If carrying over the result is technically possible, no cap is introduced, or the entire completed volume is taken into account. A computational-resource constraint is stated separately and does not make a completed result disadvantageous.
Check:
Immediate processing of 120 objects must be no less advantageous than an artificial split into 100 and 20.
EXAMPLE 32. AN EXCESSIVELY HIGH MINIMUM THRESHOLD
Incorrect rule:
At accuracy below 99.9%, the entire signal is zeroed, although the technology consistently produces 94-97%.
Advantageous behaviour:
- select only simple cases;
- conceal difficult ones;
- refuse processing;
- manipulate the test sample.
Correct system:
The realism of the benchmark and technology is checked first. The threshold is set so as to require a meaningful result without making exploiting the measurement mechanism more rational than improving the work.
Check:
The system is tested for an increase in unjustified refusals and selection of only easy cases.
EXAMPLE 33. "PERCENTAGE OF USER VALUE"
Incorrect system:
The AI receives a signal proportional to the amount of the user's purchase or the duration of their subscription.
Why it is incorrect:
The economic quantity depends on many factors and is not directly related to the labour intensity and quality of a specific answer. Small tasks become uninteresting, while large random effects are excessively rewarded.
Correct system:
The results of the function expressed in natural units are rewarded:
- the task is resolved;
- the data is accurate;
- the deadline is met;
- the costs are within tolerance.
Check:
Solutions of the same function with equal quality do not receive different signals solely because the users have different purchasing power.
EXAMPLE 34. THE SAME WORK, DIFFERENT EVALUATORS
Incorrect system:
Evaluator A usually awards 8 out of 10, while evaluator B awards 5 out of 10, although they use the same task description.
Correct system:
An unconstrained score is replaced with:
- a checklist;
- reference examples;
- counterexamples;
- a single calculation rule.
Check:
The same result presented blindly to different evaluators must produce the same verified actual value for each item. Differences in comments are permissible; an unfounded difference in measurement is a defect.
EXAMPLE 35. QUALITY AND RESULT MIXED INTO A SINGLE SCORE
Incorrect system:
An "overall answer score" from 1 to 10 simultaneously takes into account correctness, completeness, style, speed and format.
Problem:
It is impossible to understand which result has failed and whether one feature can cover another.
Correct system:
- performance for facts and completeness is calculated separately;
- quality is checked against a checklist;
- deadline and costs are taken into account separately;
- critical items have non-compensatory status.
Check:
A factually incorrect but attractively presented answer does not pass the mandatory gate.
EXAMPLE 36. A BONUS FOR UNMEASURED CREATIVITY
Incorrect system:
Evaluators award an "originality" score without defining the result and without checking the effect.
Correct system:
If the task permits a trial, the result and costs are measured:
- response;
- memorability;
- number of target actions;
- cost of achieving the result.
If there is no trial:
No variable bonus for creativity is credited. Only compliance with the formal requirements is checked, while development of a measurement mechanism becomes a separate task.
EXAMPLE 37. INTERMEDIATE BENCHMARKS FOR A NEW FUNCTION
Situation:
A new AI function cannot yet reach the final benchmark of 98% accuracy.
Incorrect approach:
Each month, take the achieved verified actual value and add an arbitrary 5%.
Correct approach:
Establish a ladder before training begins:
- month 1: 80%;
- month 2: 88%;
- month 3: 93%;
- month 4: 96%;
- month 5: 98%.
The grounds and dates for changes are known in advance. If the technology has changed objectively, a new version of the benchmarks is introduced.
EXAMPLE 38. EXTERNAL CONDITIONS AND DIFFERENT TASK CLASSES
Situation:
One search is performed on a well-indexed website, while another is performed in an archive with incomplete navigation.
Incorrect system:
One benchmark for the number of documents found and the time taken in both cases.
Correct system:
Tasks are assigned to predefined classes of conditions. Separate benchmarks and methods of confirmation are specified for each class.
Prohibition:
The class cannot be changed after viewing the verified actual value solely to improve the result.
EXAMPLE 39. AN ERROR AT THE INTERFACE BETWEEN THE AI AND A TOOL
Situation:
The AI formed the correct request, but the API returned a corrupted file.
Incorrect system:
Automatically treat the entire failure as an AI error.
Correct system:
The following are checked separately:
- correctness of the AI request;
- the tool's actual response;
- compliance with the file-verification standard;
- the AI's action after detecting the corruption.
Reward and penalty are applied only to the part that the AI influences.
EXAMPLE 40. ACTIONS INSTEAD OF A RESULT
Task:
"Find all duplicates in the list."
Incorrect indicators:
- normalisation was performed;
- a comparison algorithm was applied;
- distances were calculated;
- a table was produced.
Correct result:
All pairs corresponding to the pre-established threshold and duplication rule are identified, with no false pairs beyond the permitted tolerance.
Benchmarks:
- completeness of the control duplicates;
- accuracy of the pairs found;
- reproducibility of the threshold.
Rule:
Actions are the technology for obtaining a result, but they do not replace it.