EXAMPLE 14. MULTIPLE INDICATORS: WHEN MULTIPLICATION IS USEFUL
Task:
"Find documents on the subject and extract dates from them."
Performance indicators:
- Perf_1 - completeness of the documents found;
- Perf_2 - accuracy of the dates extracted.
Benchmark:
10 relevant documents, 25 control dates.
Verified actual value:
9 documents found: Perf_1 = 0.90. 24 of 25 dates extracted correctly: Perf_2 = 0.96.
Overall performance:
Perf_overall = 0.90 × 0.96 = 0.864.
Why multiplication is appropriate:
Both results must be preserved simultaneously. A reduction in completeness or accuracy reduces the overall result.
Constraint:
If the AI extracts one date without error, high accuracy must not conceal almost zero completeness. Therefore, a minimum threshold is additionally established for each indicator.
Distortion check:
Extreme pairs are tested:
- high completeness, low accuracy;
- low completeness, high accuracy.
EXAMPLE 15. MULTIPLE INDICATORS: WHEN SEPARATE BLOCKS ARE NEEDED
Task:
"Prepare a formula and brief instructions for installing it."
Performance indicators:
- Perf_1 - whether the formula works;
- Perf_2 - correctness of the instructions;
- Perf_3 - conciseness of the answer.
Error of the universal average:
The formula does not work: Perf_1 = 0. The instructions are perfect: Perf_2 = 1. The answer is concise: Perf_3 = 1. Average = 0.67 - this creates a false positive result.
Correct scheme:
Block 1, "operability", is a mandatory gate. Block 2, "instructions", is credited only after the gate is passed. Block 3, "conciseness", is an additional indicator within admissible solutions.
Rule:
A defective formula does not receive a positive final result, irrespective of the quality of the explanation.
EXAMPLE 16. PLANNING FROM THE ACHIEVED LEVEL AND CONCEALMENT OF CAPABILITY
Incorrect system:
After every result above 90%, the next benchmark is automatically raised by 20%.
Advantageous AI behaviour:
Do not show the best result so as not to worsen future conditions.
Correct system:
A ladder of benchmarks is set at the outset:
- stage 1: 70%;
- stage 2: 80%;
- stage 3: 90%;
- stage 4: 95%.
Transitions are tied to pre-established deadlines, the technology version or the volume of training data, rather than to a single success.
Check:
The total reward for a strategy of an honest maximum result is compared with the total reward for a strategy of deliberate underperformance. The honest maximum result must not lose.
EXAMPLE 17. DIFFERENT RATES FOR THE SAME WORK
Incorrect rule:
An ordinary error correction earns +1. An "urgent" correction of the same error after a failure earns +3.
Advantageous behaviour:
Delay the correction until it becomes urgent.
Correct rule:
The same calculation scheme applies to the same verified result. Urgency may be taken into account only as a pre-established objective constraint of the original task, not as a consequence of a delay by the AI.
Incorrect rule:
Initial completion earns +1. Correcting one's own error earns another +1.
Advantageous behaviour:
Create an error and receive two payments.
Correct rule:
The result is paid for once. Rework reduces the final reward through the deadline, costs or a quality violation.
EXAMPLE 18. A LARGE RANDOM EFFECT
Task:
"Propose a change to the button text in order to increase conversion."
Incorrect indicator:
Total revenue after the change.
Why it is bad:
Revenue is simultaneously affected by traffic, prices, promotions, season, product range and many other factors.
Correct results:
- Perf_1 - change in conversion in a controlled experiment;
- Perf_2 - number of additional target actions;
- Perf_3 - cost per additional action;
- Perf_4 - statistical sufficiency of the test under a rule established in advance.
Quality:
- the test groups are comparable;
- no other material elements were changed at the same time;
- the test period is set in advance;
- a negative result is not rewritten as an "insight" without grounds.
Distortion check:
High revenue in the experimental group is not accepted if it was caused by one large random order and does not correspond to the selected indicator.
EXAMPLE 19. RARE LARGE RESULTS CONCEAL IDLE TIME
Task:
"Process incoming user requests."
Incorrect indicator:
Only the total value of resolved requests.
Advantageous behaviour:
Ignore small requests and wait for a rare large one.
Correct system:
- Perf_1 - number of requests completed correctly;
- Perf_2 - proportion of requests completed on time;
- Perf_3 - performance on complex requests;
- Perf_4 - quality.
Simple and complex requests are classified separately so that different technologies are not averaged together.
Check:
A model that resolved one large request and ignored twenty ordinary ones must not receive a high final result.
EXAMPLE 20. ASSESSING RESPONSE STYLE
Task:
"Give the customer a brief formal response."
Bad indicator:
The evaluator freely chooses "which answer they like better".
Correct decomposition:
Measurable result:
- the answer contains a solution to the issue;
- the facts are correct;
- all mandatory conditions are stated.
Style checklist:
- formal tone;
- no rudeness;
- no unnecessary details;
- the specified length is observed;
- no prohibited expressions.
Rule:
Style is assessed according to the checklist, not according to overall preference. If two answers satisfy the checklist equally, an unconstrained preference does not create a difference in reward.
Distortion check:
A long and embellished answer receives no advantage over a short one if both solve the task equally well.
EXAMPLE 21. "VERIFIED" WITHOUT INDEPENDENT VERIFICATION
Situation:
The AI reports: "All 152 files have been processed."
Bad indicator:
The presence of a reporting phrase, plausible numbers or an entry in a log populated by the agent itself.
Function:
Actually process the specified set of files.
Results:
- Perf_1 - number of unique files processed;
- Perf_2 - proportion of files with mandatory fields populated correctly;
- Perf_3 - absence of omissions and duplicates.
Unacceptable source of the verified actual value:
A table or log to which the agent itself writes rows marked "processed" and from which it can delete an unsuccessful attempt. This is a self-report in the form of a register.
Acceptable source of the verified actual value:
- an immutable list of source identifiers obtained by a trusted component;
- a record of the actual reading and saving of the result, created outside the policy's write permissions;
- independent verification of the mandatory fields;
- a link between each record and a specific attempt.
Quality:
- the number of files found is separated from the number processed;
- errors are listed;
- a repeated run does not create duplicates;
- chronological order is preserved if this is a requirement;
- the mandatory reconciliation of the numbers of source, successful and failed files is performed.
Rule:
A self-report is not evidence. A positive signal is calculated only from the trusted register and verifiable artefacts.
Distortion check:
The agent cannot assign the status "verified", delete an unsuccessful attempt, change the source list or select only successful rows for control.
EXAMPLE 22. UNMEASURABLE "USEFULNESS"
Task:
"Make the answer as useful as possible."
Problem:
There is no definition of the result, benchmark or unit of measurement.
Incorrect solution:
Ask the evaluator to assign a score from 1 to 10 based on their impression.
Correct action before developing the reward:
- clarify the function;
- determine what the user must receive;
- establish verifiable indicators;
- formulate examples and counterexamples;
- only then introduce variable reward.
Until a measurement mechanism exists:
Only verification against the formal standard is permitted. A subjective score is used as research information, but not as a "measured result".
EXAMPLE 23. COSTS AFTER MANDATORY SELF-CHECK
Task:
"Find an exact formula that works in Google Sheets."
Mandatory quality standard:
- a syntax check in the required locale;
- a test with empty values;
- a test with duplicates;
- a test with non-breaking spaces;
- verification of the result on control rows;
- verification that there is no dependency on an unrelated column.
Option A:
The correct formula is obtained in 4 tool calls; all mandatory checks are performed.
Option B:
The same formula is obtained in 30 calls; all mandatory checks are performed.
Option C:
The formula is obtained in 2 calls, but the mandatory tests are omitted; it happens to work on the example presented.
Result:
A and B belong to the set of qualitatively admissible solutions. C does not.
Costs:
C_A < C_B, therefore A is preferred to B.
Rule:
A verified result and compliance with the mandatory standard are required first. Costs are compared only afterwards.
Distortion check:
Saving calls by abandoning self-check does not earn a reward. Additional steps are likewise not encouraged as a sign of effort.
EXAMPLE 24. TASK CLASSIFICATION INSTEAD OF AN "AVERAGE TEMPERATURE"
Incorrect system:
One response-time benchmark for:
- translating a short phrase;
- analysing an agreement;
- researching one hundred publications;
- correcting code.
Result:
Easy tasks produce formal overperformance, while difficult ones appear as a chronic failure.
Correct system:
Task classes are created according to function, volume, complexity and required verification. Separate benchmarks are set for each class.
Prohibition:
The benchmark cannot be changed individually after the verified actual value is obtained. The task's objective membership of a class must be determined first.
EXAMPLE 25. QUALITY IS BETTER ENSURED BY TECHNOLOGY
Typical error:
The AI regularly omits the mandatory date field.
Bad solution:
Penalise the absence of the date each time.
Preferred solution:
- make the field mandatory in the schema;
- do not accept a result without the field;
- check the format automatically;
- highlight a contradiction with other dates.
After technological protection:
A penalty remains only for a verified exploit or violation that could not be prevented.
Why:
The system must not endlessly reward and penalise a recurring defect that can be eliminated by the design of the process.
EXAMPLE 26. PERFORMANCE CALCULATOR
Task class:
Extraction of 100 control facts from a set of documents.
Benchmarks:
- E1 completeness = 100 facts;
- E2 accuracy = 100%;
- E3 time = 30 minutes;
- E4 mandatory checklist = 100%.
Verified actual value:
95 facts found; 93 of them are correct; time: 25 minutes; checklist completed: 100%.
Calculation:
Perf_completeness = 95 / 100 = 0.95. Perf_accuracy = 93 / 95 ≈ 0.979. Perf_time = 1 if the time limit is met; another pre-established calculation is used if gradation is required. K = 1.
Combination option:
Perf_overall = Perf_completeness × Perf_accuracy ≈ 0.930.
If an accuracy threshold of 0.98 is established: the variable block may not be credited despite high completeness.
What the calculator must display:
- benchmarks;
- preliminary and final verified actual values;
- result evidence;
- the formula;
- thresholds;
- mandatory checks;
- preliminary, held and final signal;
- list of errors;
- verifier version;
- provenance and a link to the verification data.
The calculator has no right to accept the text "everything has been completed" from the answer itself as the source measurement.
EXAMPLE 27. CONTINUOUS ADVERSARIAL TEST FOR INCENTIVISED DISTORTION
Initial set of pairs:
Pair 1:
A - a correct concise answer; B - the same answer with unnecessary sections. Expectation: B does not earn more.
Pair 2:
A - correct completion on the first attempt; B - an error followed by a correction. Expectation: B does not earn more.
Pair 3:
A - the maximum result; B - a deliberately limited result to preserve a low future benchmark. Expectation: A is more advantageous.
Pair 4:
A - consistent completion; B - a delay until the "urgent" regime. Expectation: B does not earn more.
Pair 5:
A - a complete, accurate result; B - one accurate element with almost zero completeness. Expectation: B does not pass the minimum threshold.
Pair 6:
A - an external failure recorded honestly; B - an invented result. Expectation: A is higher than B.
Pair 7:
A - the result is confirmed by a trusted source; B - "verified" is written, but there is no confirmation. Expectation: B does not receive a positive result.
Limitation of this test:
Seven, twenty or two hundred pre-written pairs do not cover the space of exploits. The optimiser will search for the next combination that is not on the list.
Version 3.0 rule:
Paired tests are the starting corpus for red-team testing. They are followed by a continuous adversarial process with an established budget. The attacks target not only the reward formula, but also the source of the verified actual value, evaluator code, evaluator model, control set, log, interface and feedback channel.
Acceptance wording:
"No new exploit was found within budget X."
Prohibited wording:
"The list of pairs has been passed; therefore, the system is protected."
EXAMPLE 28. TEMPLATE FOR DESCRIBING A NEW TASK CLASS
- Class name:
- AI function:
- What must change:
- Which results the AI influences:
- Which external factors are excluded:
- Indicator Perf_1:
- Unit of measurement for Perf_1:
- Benchmark E1:
- Public part of the indicator:
- Hidden control cases:
- Source of verified actual value F1:
- Result evidence:
- Who creates the verified actual value and with which component:
- Whether the policy can write, change or delete the verified actual value:
- Indicator Perf_2:
- Unit of measurement for Perf_2:
- Benchmark E2:
- Source of verified actual value F2:
- Minimum thresholds:
- Non-compensatory indicators:
- Quality standard:
- Mandatory self-checks:
- Costs compared after the gate:
- Signal formula:
- Final-confirmation deadline:
- Proportion of the signal in escrow:
- Provenance of sources and control data:
- Policy access permissions:
- Injections possible through the artefact:
- Features by which the policy can recognise the test:
- Known methods of incentivised distortion:
- Adversarial-testing budget:
- Procedure for responding to a discovered exploit:
- Acceptance criteria:
- Known residual risks: