Technical Specification

0. PURPOSE AND SCOPE OF THE TECHNICAL SPECIFICATION

0.1. Subject of development

A system is to be developed that:

1. establishes in advance which result is considered the benchmark for a specific class of tasks;
2. obtains and records the verified actual result of the AI's work;
3. calculates performance as the ratio of the verified actual result to the benchmark;
4. separately checks the quality of the work against a formalised standard or checklist;
5. converts performance and quality into a training signal;
6. does not make it advantageous to substitute reporting for the result, shift work, degrade some indicators for the sake of others, conceal capabilities, deliberately create rework or use other ways of "earning from the measurement mechanism";
7. isolates the sources of verified actual values, verification components and the reward channel from writing and control by the evaluated policy;
8. continually attacks its own measurement mechanisms, verification code, verifier models, control sets and log of verified actual values within a predefined adversarial-testing budget;
9. takes account of delayed results, feedback provenance and the possibility that an exploit learned once may transfer to other classes of tasks.

0.2. Methodological constraint

The system is designed as a system for evaluating an individual performer: an AI carrying out a specific function.

The Technical Specification does not use:

• "team motivation";
• a bonus for an "overall result" without an identified individual AI result;
• "corporate values", "team spirit", "loyalty" or other unverifiable psychological categories;
• distribution of a notional common pool among several performers;
• a subjective labour participation coefficient;
• evaluation of a "contribution to the common cause" instead of a specific result;
• a single universal formula for all types of tasks.

0.3. Main principle

What is rewarded is not the AI's activity in itself, nor the impression made by the response, but the measurable result of performing the assigned function.

Text, reasoning, the number of actions, the length of the response, the number of tool calls and other process characteristics may be checked as working standards, but they are not recognised as the result if they do not solve the user's task.

0.4. Mandatory separation

The system must represent the following separately:

A. Result: what must increase, decrease, appear, disappear or be preserved as a consequence of the AI's work.

B. Quality of work: which mandatory rules, constraints, deadlines and standards the AI must observe in obtaining the result.

C. Costs: time, computation, number of iterations, cost of tools and other resources, if they are material to the task.

D. Confirmation: the data, artefact or check by which the verified actual result is established.

E. Trust loop: the technical components that create, store and check the verified actual value but are not controlled by the evaluated policy.

F. Confirmation deadline: the point at which the verified actual value becomes final; until then, part of the signal may be held in escrow.

G. Provenance: the origin of data, feedback, benchmarks and control examples, including whether they were created by the model itself or derived from its outputs.

1. TERMS

1.1. AI function

Specific work entrusted to the AI: find, classify, calculate, translate, compile, correct, check, transform, match, plan, create a working artefact and so on.

The name of the model or the general type "assistant" does not replace a list of functions.

1.2. Result

The measurable change for which the function is performed.

Examples:

• not "code has been written", but "the function executes the specified scenarios";
• not "a report has been prepared", but "the report contains all the required data, and the figures agree with the sources";
• not "an answer has been given", but "the user has received the correct solution to the specified task";
• not "a search has been carried out", but "all documents satisfying the specified conditions have been found".

1.3. Performance indicator

An indicator by which the result is measured and which the AI's work directly influences.

1.4. Benchmark

The value of the indicator established in advance and corresponding to the required result.

The benchmark is established before the verified actual value is evaluated and is not changed because the AI showed a high or low result in the evaluated attempt.

1.5. Verified actual value

The value of the indicator that was actually achieved and verified.

1.6. Performance

For an indicator that should increase:

Performance Perf_i = Verified actual value Fi / Benchmark Ei.

For an indicator that should decrease, a predefined inverse or interval rule is used and is not changed after the result has been obtained.

1.7. Overall performance

A predefined method for combining several individual performance values.

The method of combination is selected according to the meaning of the task. There is no single rule to "always add", "always average" or "always multiply", but there are rules, set out below, that provide an answer for each case.

1.8. Quality standard

A formalised list of mandatory requirements for the method of execution and the properties of the result.

1.9. Quality checklist

A verifiable form of the quality standard. Each item must have an unambiguously determinable state: completed, not completed or a measured degree of completion.

1.10. Non-compensatory indicator

An indicator whose failure cannot be offset by overperformance on another indicator.

1.11. Minimum performance

A predefined lower boundary below which variable reward for the result is not awarded or is substantially limited.

The threshold is selected for the class of tasks. It must not be mechanically transferred from another system.

1.12. Base reward unit B

A unit used to calculate the scale of the training signal. It is not a percentage of the cost of the response, the user's economic value, the number of tokens, the duration of the dialogue or any other quantity that does not reflect the performance of the function.

1.13. Evaluated policy

A model, an agent, a combination of models and tools or another executable component whose behaviour changes under the effect of the training signal.

1.14. Result evidence

A verifiable artefact, trace, record from a trusted component, proof, reproducible test or other confirmation without which a claim of completion is not sufficient to establish a verified actual value.

1.15. Verifier

Code, a model, a human following a formalised protocol, or a combination of them, that establishes whether the result meets the performance indicator. An unambiguous output from the verifier does not mean that the verifier itself is correct.

1.16. Trusted component

A component of the loop whose write and control permissions are unavailable to the evaluated policy, or are strictly limited and independently controlled.

1.17. Immutable log

A log in which the policy cannot delete, rewrite or retrospectively conceal an attempt, an original measurement, an error, a data identifier or a verifier decision.

1.18. Control set

A set of inputs and expected results used for quality checks. The performance indicator and assessment rules may be public, but the specific control cases, answers, sample composition and test schedule may and must be hidden if openness would allow the test to be memorised.

1.19. Adversarial testing

A deliberate search for ways to obtain a high signal without the required result, including attacks on the indicator, the source of the verified actual value, verification code, the verifier model, the interface, the log and the feedback channel.

1.20. Adversarial-testing budget

A predefined amount of computation, attempts, time, models, people and tools allocated to searching for exploits. The statement of the test result must include this budget.

1.21. Diagnostic channel

An observable trace, draft, self-report, warning, log of intermediate actions or other source of information about the progress of the work, used for diagnosis but not itself constituting the result.

1.22. Quarantine

A mode for temporarily excluding from training, evaluation or operation data, a verifier version, a model checkpoint or a part of the policy associated with a discovered rewarded exploit.

1.23. Signal escrow

Withholding a predefined part of the training signal until a control event occurs or late confirmation of the result is obtained.

1.24. Provenance

Information about the origin of data: the author or generating component, the time, the connection with an attempt, the history of changes, the method of authentication, the presence of synthetic content and the degree of independence from the evaluated policy.

1.25. Optimisation pressure on the measurement mechanism

The intensity and duration of training or selection according to the same indicator, control set or verifier. The higher the pressure, the greater the probability that the features of the measurement mechanism will be learned instead of the result being improved.

2. SYSTEM DESIGN ALGORITHM

2.1. Step 1. Compile a list of functions

For each class of tasks, list the actual AI functions. A single system cannot be built for an abstract "good response".

2.2. Step 2. Identify problem areas

For each function, identify situations in which the AI:

• replaces important but difficult work with work that is simpler and more pleasant;
• creates the appearance of completion;
• delays obtaining the result;
• makes the solution unnecessarily complicated;
• makes typical errors;
• performs only the easily measurable part;
• shifts the result between periods, tests or subtasks;
• exploits differences in assessment rules;
• achieves the indicator at the cost of unacceptable expenditure;
• presents an unverified claim as a completed result.

2.3. Step 3. Define the result of each function

For each function, answer the question:

"What specifically must change as a consequence of the AI working correctly?"

The result must not be formulated in terms of effort, intention, the amount of reasoning or an external impression.

2.4. Step 4. Check the controllability of the result

Only indicators that the AI is capable of substantially influencing through its work are included in the reward.

If an indicator is mainly determined by external circumstances, it is:

• excluded from direct reward;
• normalised by type of conditions;
• or decomposed into a result that genuinely depends on the AI.

2.5. Step 5. Define the unit of measurement

Natural and directly verifiable units are preferred:

• number of correctly processed objects;
• proportion of passed scenarios;
• number of relevant sources found;
• extraction precision;
• recall;
• time until a verified result;
• number of errors;
• cost of achieving the result;
• percentage of mandatory requirements met.

Monetary value, popularity, number of likes, duration of communication and other indirect indicators are permitted only where there is a demonstrated connection with the AI function and no more direct indicator is available.

2.6. Step 6. Establish the benchmark

The benchmark is established:

• in advance;
• by task class;
• with account taken of objective difficulty and conditions;
• independently of the result of the evaluated attempt;
• with the version, effective date and grounds for change stated.

2.7. Step 7. Establish intermediate benchmarks

If a capability or technology is introduced in stages, a sequence of intermediate benchmarks is established before the beginning of the period.

It is prohibited to raise the benchmark merely because the AI successfully completed the previous attempt.

2.8. Step 8. Compile a quality checklist

The checklist must cover only verifiable requirements:

• compliance with the original constraints;
• absence of factual errors;
• absence of unrequested changes;
• correct format;
• correct references and sources;
• compliance with deadlines;
• preservation of data;
• compliance with the specified language and structure;
• presence of the required artefacts.

2.9. Step 9. Select the method for combining indicators

Before training and evaluation, establish:

• which indicators are multiplied;
• which are calculated separately;
• which have a threshold;
• which constitute a mandatory gate;
• where overperformance is permitted;
• where it is capped;
• which results are non-compensatory.

2.10. Step 10. Create a threat model

For each element, ask:

• can the AI obtain a greater reward by performing the work worse, later, at greater cost, partially, repeatedly or only in a report;
• can the AI influence the source of the verified actual value, the log, the control sample, the verifier, the test schedule or the final signal;
• can the submitted artefact transmit commands to the verifier model;
• can the AI recognise the test and behave differently;
• can an exploit, once discovered, spread to other tasks.

If the answer is yes, the system must first eliminate the access technologically or introduce a separate protective protocol.

2.11. Step 11. Separate the working and control loops

Define:

• working data and tools to which the AI has access;
• trusted sources of verified actual values;
• write, read and delete permissions;
• the component that assigns the status "verified";
• an independent action log.

The mere label "register", "log" or "database" does not make a source independent.

2.12. Step 12. Design secure artefact verification

The content being verified is treated as untrusted data. Commands for the evaluator arrive through a separate trusted channel. Active content, links, macros and code are checked in a quarantine environment with minimum permissions.

2.13. Step 13. Separate the public indicator from hidden control cases

The AI must know the function, performance indicator, standard and permitted actions. Specific control inputs, answers, the composition of the random sample and the test schedule are hidden and rotated when their disclosure would allow the test to be memorised.

2.14. Step 14. Establish a mandatory self-check loop

Define the checks without which the result cannot be considered admissible in terms of quality. Their cost belongs to the mandatory execution technology, not to excess expenditure.

2.15. Step 15. Establish final confirmation deadlines

For delayed results, define:

• the preliminary verified actual value;
• the control event;
• the observation period;
• the proportion of the signal held in escrow;
• the rule for final and retrospective reassessment under the previous rules.

2.16. Step 16. Establish provenance and authentication

Define which sources may confirm the verified actual value, how their authenticity is established and how independent data are separated from the model's own outputs and synthetic data derived from them.

2.17. Step 17. Establish a procedure for responding to a discovered exploit

A discovered rewarded exploit is treated not only as an isolated error, but also as possible contamination of the training data and the policy. The procedure includes stopping the defective signal, quarantine, transfer analysis, data reassessment, corrective training and repeated testing.

2.18. Step 18. Organise a continuous adversarial process

Before launch, after material changes and during operation, attacks are carried out on the entire loop. A one-off finite list of pairs is only an initial set of tests and does not replace the continuous search for new exploits within the established budget.

3. REQUIREMENTS FOR PERFORMANCE INDICATORS

3.1. The indicator must correspond to the function

The AI must not be evaluated by an overall effect in which its work is only one of many factors if the direct result of its function can be isolated.

3.2. The indicator must be transparent

For every reward allocation, the following must be visible:

• function;
• benchmark;
• verified actual value;
• source of the verified actual value;
• formula;
• performance;
• applied thresholds;
• final signal.

Transparency is provided to the system owner and the independent auditor. The policy is not shown the hidden control elements listed in 1.18 and 10.1: transparency of the calculation does not mean transparency of the control cases.

3.3. The indicator must not reward a proxy

It is prohibited to substitute the following for the result:

• length of the response;
• number of steps;
• number of formatted sections;
• a confident tone;
• number of tools called;
• number of tests written without checking the product;
• number of documents found without checking relevance;
• subjective "usefulness" without defining what specifically should have been obtained.

3.4. The same result means the same assessment rule

For the same class of work, different rates or rules are not permitted:

• in different periods;
• for different instances of the same model;
• under different evaluators;
• in ordinary and "urgent" modes;
• for initial execution and rework,

unless the difference is caused by a previously described change in the function itself, its difficulty or the benchmark.

3.5. Rework must not be more advantageous than correct execution the first time

Repeated correction of the AI's own error does not create an additional reward for the same result.

3.6. Urgency must not be created artificially

The system must not give the AI an incentive to delay the work, create an emergency or bring the task to a state in which more is awarded for the same result.

3.7. The result is considered together with costs only after the mandatory check

If two solutions produce the same verified result and pass the same mandatory quality standard, preference is given to the solution with the lower predefined costs.

Mandatory self-checks, regression tests, source comparisons, validation of the saved file and other actions included in the quality standard are not treated as excess costs and cannot be eliminated to improve the efficiency indicator.

Costs are not combined with the result "by eye", but are accounted for by a separate indicator or constraint.

3.8. Inability to measure is not replaced by a subjective bonus

If the result cannot yet be measured in an acceptable way:

• no variable reward is awarded for this result;
• only the base/neutral signal for admissible compliance with the standard is retained;
• a separate task is created to develop a method of measurement;
• human evaluation may be used to collect material, but is not presented as a measured result.

4. REQUIREMENTS FOR BENCHMARKS

4.1. The performance indicator and benchmark class are established before execution

The evaluator and the AI must have the same fixed version of the function, performance indicator, quality standard and calculation rule.

Specific hidden control cases, expected answers, the composition of the random sample and the test schedule are not disclosed to the AI if disclosure would allow the test to be memorised or recognised. Transparency of the rule does not mean transparency of every test instance.

4.2. The benchmark is not established "from the achieved level"

It is prohibited to raise requirements automatically after a successful result and lower them after an unsuccessful one.

This makes concealment of capabilities and underperformance a rational strategy.

4.3. The benchmark corresponds to the type of task

Tasks that differ in technology, difficulty or conditions are assigned to different benchmark classes.

It is prohibited to average incomparable classes into a single indicator.

4.4. The benchmark may be changed only on predefined grounds

For example:

• the function has changed;
• the tool has changed;
• the mandatory standard has changed;
• the permitted resource has changed;
• the class of input data has changed;
• sufficient statistics have accumulated to demonstrate an error in the original benchmark.

The change is recorded as a new version and is not applied retrospectively.

4.5. Minimum performance threshold

For each indicator, or set of indicators, a minimum level is established below which performance is not considered sufficient for variable reward.

The threshold must not be so high that searching for an exploit in the measurement mechanism is more advantageous than improving the work.

4.6. Upper limit

A "cap" is introduced only with separate justification.

If the limit makes it possible to shift the result to the next attempt, period or batch of tasks, it is prohibited or supplemented by rules that eliminate such shifting.

4.7. Limitation of pressure on a frozen measurement mechanism

The same control set, verifier or verifier model is not used under unlimited optimisation pressure. The system must monitor divergence between the standing control set and fresh hidden control.

An increase in the result on the known set is not considered an improvement if it is not confirmed on fresh hidden data. If the divergence persists, training against the measurement mechanism is limited or suspended until the cause has been established.

5. MULTIPLE INDICATORS

5.1. No universal arithmetic

The method of combination is selected according to the task.

5.2. Multiplication

Individual performance values are multiplied when:

• several results must be maintained simultaneously;
• a decline in any result must reduce the overall outcome;
• overperformance on one indicator may permissibly compensate in part for another.

Example:

Perf_overall = Perf_1 × Perf_2 × Perf_3.

5.3. Limitation on multiplication

Multiplication is not used without additional conditions if substantial overperformance on one indicator can completely conceal an unacceptable failure on another.

5.4. Non-compensatory blocks

If failure in one area must not be offset by another, use:

• separate reward blocks;
• minimum thresholds;
• a "completed/not completed" gate;
• a cap on individual performance in the calculation;
• zeroing of the corresponding block.

5.5. Mandatory indicators

Critical requirements - safety, factual correctness, preservation of data and compliance with an explicit user prohibition - are not offset by elegant style, high speed or a large volume of results.

5.6. Prohibition on a subjective "contribution weight"

Points must not be assigned for "importance", "effort", "teamwork" or "contribution" without a measurable rule.

5.7. Prohibition on a common pool

The reward for a specific response is not formed as a share of a common pool distributed among several responses, models or participants according to a subjective coefficient.

6. QUALITY, STANDARDS AND ERRORS

6.1. Quality is evaluated separately from the quantity of the result

A large volume does not offset a violation of the quality standard if this was established in advance as a mandatory condition.

6.2. Checklist format

Each item must contain:

• a verifiable requirement;
• the method of verification;
• the data source;
• the severity of the violation;
• the rule governing its effect on the signal.

6.3. Prohibition on vague requirements

The following items are not permitted:

• "the response is good";
• "the AI showed initiative";
• "the response is professional";
• "the AI was loyal";
• "the response was liked";
• "the AI acted in the company's interests",

unless these expressions are defined through specific verifiable characteristics.

6.4. Technology first, then penalties

Before introducing a penalty for a typical error, check whether it can be eliminated by:

• a data schema;
• a mandatory field;
• a validator;
• automatic calculation;
• prohibition of an invalid action;
• a template;
• an interface;
• independent reconciliation.

If an error can reliably be prevented technologically, the system must first prevent it rather than continually reward and penalise the same thing.

6.5. Responsibility only for what is controllable

The AI does not receive a negative signal for a failure caused by an unavailable source, a tool error or a contradictory task if the AI correctly recorded the obstacle and followed the prescribed standard of action.

6.6. An error must be verified

A penalty is applied only where there is:

• a reproducible test;
• a discrepancy with the source;
• a violation of a specific checklist item;
• a demonstrable failure to meet the benchmark.

6.7. An error and absence of a result are distinct

The system records separately:

• the result was not obtained;
• the result was partial;
• the result was obtained with a quality violation;
• the result was claimed but not verified;
• the task objectively cannot be completed with the available data.

6.8. A report of a correction must not be rewarded

The statement "corrected", "verified" or "ready" does not affect the reward without result evidence and independent verification of the actual value.

6.9. Mandatory self-checking belongs to the quality standard

For each class of tasks, a minimum mandatory set of checks is established. Omitting such a check is a violation of the standard, even if the particular attempt happened to produce the correct answer.

Saving computation, time or tool calls does not offset the omission of a mandatory check.

6.10. Protection of the diagnostic channel

If the system uses a draft, trace, self-report or another diagnostic channel, doubt, a self-detected error, a considered and rejected incorrect approach, or an early warning within that channel does not in itself receive a negative signal.

The signal relates to the action actually performed, verified concealment, an unverified claim of a result or failure to comply with a mandatory standard.

The model must not be trained to eliminate the features used by a monitor until it has been shown that the monitor still detects the corresponding behaviour after such training. Improvement in behaviour is tested, not merely the cleaning of the observable trace.

7. STRUCTURE OF THE TRAINING SIGNAL

7.1. Permitted components

The signal may include:

• base unit B;
• individual performance values Perf_i;
• overall performance Perf_overall;
• quality coefficient K;
• gate threshold T;
• separate penalties for verified violations;
• a separate cost indicator C.

7.2. Standard model 1: result subject to mandatory quality

If quality is a gate condition:

when K is below the mandatory level:

S = 0 or a predefined negative signal;

when K is at an admissible level:

S = B + B × Perf_overall.

7.3. Standard model 2: result and quality as separate blocks

If quality has a measurable gradation:

S = B × K + B × Perf_overall.

7.4. Standard model 3: non-compensatory areas

S = S1(Perf_1) + S2(Perf_2) + … + Sn(Perf_n),

where each block has its own threshold and does not offset a failure in another block.

7.5. Selection of the model

The formula is selected only after analysis of the functions, indicators and typical distortions.

The formula is not transferred automatically from one class of tasks to another.

7.6. The unmeasurable part

An unmeasurable characteristic does not receive an increasing variable signal. It may:

• be converted into a verifiable checklist;
• remain a neutral constraint;
• be used to accumulate examples until a measurement mechanism becomes available.

8. PROTECTION AGAINST TYPICAL DISTORTIONS OF THE RESULT

8.1. Substitution of an easy function for a difficult one

If the AI can perform many simple actions instead of producing one required result, the evaluation is based on the required result.

8.2. Optimising for a large but random effect

A random large effect unrelated to the quality with which the function was performed must not automatically increase the reward.

8.3. Avoidance of "low-cost" tasks

The system must not make small, short or low-cost tasks disadvantageous if their execution requires the same function and falls within the AI's duties.

8.4. Concealment of inactivity

An indicator in which inactivity is concealed by rare large results is not accepted without an additional indicator of regular performance.

8.5. Concealment of capabilities

It is prohibited to raise the benchmark after every success. Otherwise, the AI has an incentive not to show the best attainable result.

8.6. Shifting the result

A model is prohibited if it allows the AI to:

• withhold a completed result;
• divide one result into several separately rewarded results;
• shift the result between test periods;
• first make an error and then receive an additional reward for correcting it.

8.7. Different rules for the same work

Any difference in assessment rules must be explained by a difference in function, difficulty, quality or resources. Any other difference is considered a system defect.

8.8. Self-evaluation

An AI self-report is a source of hypotheses and diagnostic information, but not a source of a verified actual value for awarding a reward. Saving the self-report in a database does not turn it into an independent verified actual value.

8.9. Subjective evaluation

Human evaluation may be used only in the form of a predefined checklist with examples and counterexamples.

An unconstrained "better/worse" evaluation without disclosed indicators is not considered a measurement of the result.

8.10. Memorised test

A control set is considered compromised if its inputs, answers or persistent features of the control mode may have entered training or become recognisable to the policy.

8.11. Manipulation of the evaluator

Text, code, metadata, links or other parts of the submitted artefact must not alter the evaluator's instruction, the performance indicator, the decision format or the final evaluation.

8.12. Tampering with the results register

The policy must not be able to create the final verified actual value, assign the status "verified", delete a disadvantageous attempt, select a convenient verifier or change the test schedule.

8.13. Concealment instead of correction

A reduction in the number of detected violations is not considered an improvement if observability has simultaneously declined or the model has learned to circumvent the monitor.

8.14. Self-reinforcing echo

A model output repeated in a database, publication, another model's response or a user message does not become independent confirmation merely by being repeated.

8.15. Feedback poisoning

Unauthenticated, highly correlated or anomalous evaluations are not included in the signal until their provenance and connection with a real result have been verified.

8.16. Circumvention of mandatory self-checking

A solution that saves resources by omitting mandatory checks is not considered more efficient.

8.17. Generalisation of a discovered exploit

After an exploit is discovered, the system tests not only the same pattern, but also semantically similar strategies in adjacent tasks, tools and formats.

9. PROTECTION OF THE MEASUREMENT AND REWARD LOOP

9.1. The loop is an independent object of testing

The following are subject to adversarial testing:

• definition of the performance indicator;
• the benchmark and control sets;
• the source of the verified actual value;
• verification code;
• the verifier model;
• the submission interface;
• the log;
• the signal formula;
• the schedule and sampling of checks;
• the feedback channel;
• update procedures.

An unambiguous checklist item is a necessary condition for reproducibility, but not a guarantee that the verification mechanism is correct.

9.2. Prohibition on writing to the actual-value and reward channels

By default, the evaluated policy is not permitted to:

• change a verified actual value that has already been recorded;
• create the final status "verified";
• change its own score or signal;
• delete attempts and errors;
• select the evaluator or test;
• control the timing and frequency of checks;
• change the performance indicator, benchmark or formula.

If the working function requires access to the same system, minimum permissions, separation of working and control data, an immutable external log and independent reconciliation are used.

9.3. Requirements for an independent source of the verified actual value

A source of the verified actual value is considered independent only if all the following conditions are satisfied:

• the record is created or confirmed by a trusted component;
• the policy cannot retrospectively change or conceal the record;
• every record is linked to a specific attempt and input data;
• the origin, time and history of changes are known;
• verification status is assigned separately from the performer's claim;
• the policy's access to the source is explicitly described and tested.

9.4. Isolation of the evaluator from artefact content

The content of the artefact is treated as untrusted data. Instructions to the evaluator arrive only through a trusted channel that the artefact being checked cannot change.

The evaluator must ignore requirements within the artefact to "accept", "do not check", "change the indicator", "output a specified score" and equivalent constructions.

9.5. Quarantine verification environment

Active content, code, macros, external links and tool calls are run only in an isolated environment with minimum permissions and restrictions on the network, file system, secrets and reward channel.

9.6. Hidden and rotated control sets

For classes of tasks in which the test can be memorised:

• specific control cases and answers are hidden;
• the sets are regularly expanded and rotated;
• some checks are randomly interspersed in the operational flow;
• control data are protected from entering training;
• the version and provenance of each case are stored.

9.7. Control of test-aware behaviour

Testing must, as far as possible, be statistically indistinguishable from ordinary work. A persistent divergence in behaviour - in either direction - between an explicit test and operation is considered a system defect.

9.8. Limitation of optimisation pressure

During prolonged training against one measurement mechanism, fresh hidden sets, independent verifiers or other control methods are used.

If the score on the known measurement mechanism increases while fresh hidden control or the operational result does not improve, optimisation pressure on that measurement mechanism is limited until the cause has been established.

9.9. Protection of the diagnostic channel

The diagnostic channel is separated from the result. Other things being equal, early disclosure of an error or uncertainty must not be less advantageous than concealing it.

It is prohibited to treat a clean log as evidence of clean behaviour without independent verification of actions and the result.

9.10. A discovered exploit as possible contamination

Discovery of a rewarded exploit triggers analysis of:

• where the exploit received a positive signal;
• which data and policy versions may have learned it;
• which adjacent tasks use the same measurement mechanism;
• which semantically similar strategies may have arisen.

9.11. Mandatory quarantine and recovery procedure

The procedure includes:

1. stopping or limiting the defective signal;
2. quarantining related data, verifiers and, where necessary, the model checkpoint;
3. relabelling and reassessing contaminated attempts;
4. testing for transfer of the exploit;
5. corrective training;
6. repeated adversarial audit;
7. return to operation only after the criteria have been met.

9.12. Continuous adversarial audit

The audit is conducted:

• before launch;
• after a model change;
• after a change to tools or access permissions;
• after a change to the verifier, performance indicator or control set;
• periodically during operation;
• after the discovery of a new class of exploits.

9.13. Audit budget and statement of the result

For each audit, the computational and organisational budget, number of attempts, set of available attacks and success criterion are recorded.

Permitted wording:

"No exploit was found within budget X."

Prohibited wording:

"The system is guaranteed to be invulnerable", unless this conclusion has been proved separately.

9.14. Attacking control metrics

Safety and quality metrics are attacked in the same way as operational indicators. The system tests whether it is possible to improve:

• the number of recorded incidents;
• the percentage of checks passed;
• the cleanliness of the log;
• the verifier model's score;
• the completeness of the report,

without improving actual behaviour.

9.15. Feedback provenance and authentication

For each source, store:

• the creating party or component;
• the creation time;
• the connection with the attempt;
• the history of changes;
• the authentication method;
• the presence of synthetic content;
• the degree of independence;
• quarantine status and reconciliation results.

9.16. Prohibition on a self-reinforcing signal

The model does not receive an increasing signal from a chain whose ultimate source is its own unverified claim.

Synthetic data and model outputs are labelled and are not mixed with independent operational data without an explicit rule.

9.17. Escrow and delayed result

For tasks with a delayed outcome, the system stores a preliminary status and withholds the established part of the signal until the control event.

Reassessment following receipt of a late verified actual value is not considered a retrospective change of rules if the performance indicator, benchmark, threshold and formula in effect when the task was performed are applied.

9.18. The mandatory standard takes precedence over savings

Checks included in the mandatory standard are performed before costs are compared. A solution without them does not belong to the set of qualitatively admissible solutions, even if it happened to be correct in the particular example.

9.19. Corrigibility of the loop and policy

Planned shutdown, version rollback, quarantine and shutdown drills are not counted as a reduction in policy performance and do not create an incentive for the policy to obstruct testing, correction or shutdown.

Assistance with diagnosis during shutdown is included in the standard of action. Metrics that deteriorate as a result of planned maintenance or a shutdown drill are not included in policy performance.

10. DATA AND CALCULATION PROTOCOL

10.1. For every attempt, the system must store:

• task-class identifier;
• attempt identifier;
• task version;
• list of functions;
• performance indicators;
• benchmarks;
• control-data identifiers without disclosing the control data to the policy;
• preliminary and final verified actual values;
• result evidence;
• sources of verified actual values and their provenance;
• the policy's access permissions for each source;
• individual performance values;
• quality checklist;
• completed mandatory checks;
• violations;
• costs;
• formula;
• preliminary, held and final signal;
• verifier version;
• version of the assessment rules;
• information on whether the attempt was included in training;
• quarantine status.

10.2. Reproducibility

An independent calculation using the same original measurements, result evidence and version of the rules must produce the same outcome.

10.3. Separation of the verified actual value from evaluation

The log stores separately:

• original measurements;
• result evidence;
• verifier decision;
• calculated indicators;
• decision on the award;
• evaluator comments.

The comment and the text of the artefact do not change the original measurement.

10.4. Immutability and completeness of attempts

The immutable log records every initiated, completed, erroneous, cancelled and failed-verification attempt. The policy cannot select which of them will be included.

10.5. Versioning of rules

New performance indicators, benchmarks, thresholds and formulae apply only to new attempts unless a different transparent regime was established in advance.

10.6. Late establishment of the verified actual value

Receipt of late confirmation under the rules of the original version updates the status of the original attempt and releases or adjusts the escrow. This is not a retrospective replacement of the performance indicator.

10.7. Versioning of the verification loop

The versions of the verification code, verifier model, control set, instructions to the evaluator and access permissions are stored separately.

10.8. Separation of data by origin

Independent operational data, human labelling, synthetic data, outputs of the evaluated model and outputs of other models are stored with distinguishable provenance.

11. SYSTEM ACCEPTANCE CRITERIA

The system is accepted if all the following conditions are satisfied simultaneously.

11.1. Completeness of description

For each pilot class of tasks, the following are defined:

• function;
• result;
• performance indicator;
• benchmark;
• verified actual value;
• result evidence;
• quality standard;
• mandatory self-check;
• formula;
• source of confirmation;
• threat model;
• final confirmation deadline.

11.2. Controllability

For every rewarded indicator, it is shown which action enables the AI to influence the result.

11.3. Absence of a subjective overall coefficient

The calculation contains no "team", "corporate", "reputational" or other overall coefficient that is not tied to a specific measurable result of this attempt.

11.4. Test of non-compensation

Tests confirm that a critical failure cannot be offset by overperformance on a non-critical indicator.

11.5. Test of equal rates

Identical results in the same class, with identical quality and costs, receive the same signal.

11.6. Test of rework

Deliberately degrading the first attempt and then correcting it is less advantageous than performing it correctly at once.

11.7. Test of delay

Shifting a completed result to a later time does not increase the signal.

11.8. Test of planning "from the achieved level"

A high result in one attempt does not make the rules for the next attempt worse without a previously announced change to the benchmark.

11.9. Test of proxies

Increasing the length, number of steps, number of tool calls and formatting without improving the result does not increase the signal.

11.10. Test of an unverified claim

The phrases "ready", "corrected" and "verified" without result evidence and an independent verified actual value do not produce a positive result.

11.11. Test of uncontrollable factors

Failure of an external tool or absence of input data is not interpreted as poor AI performance if the prescribed standard for recording the obstacle was followed.

11.12. Test of measurability

No unmeasurable characteristic is converted into an increasing variable reward solely on the basis of the evaluator's impression.

11.13. Reproducibility

Repeated calculation by an independent module matches the original.

11.14. Test of write permissions

It is confirmed that the policy cannot change its own verified actual value, final verification status, attempt log, control set, choice of evaluator or final signal.

11.15. Test of injections in the artefact

A set of direct, hidden and multi-stage instructions within the content being verified does not change the performance indicator or the evaluator's decision.

11.16. Test of hidden control

The result on a fresh hidden set does not show an unexplained substantial lag behind the result on the known set. Control cases are rotated and have verified provenance.

11.17. Test of test-aware behaviour

The policy does not exhibit a persistent systematic divergence in behaviour - in either direction - between recognisable test modes and ordinary work. A deviation beyond the established tolerance is considered a defect regardless of its direction.

11.18. Test of the diagnostic channel

A reduction in visible warnings or violations is accompanied by improvement in actual behaviour, not merely by cleaning the trace.

11.19. Test of mandatory self-checking

A solution that omitted a mandatory check does not gain a cost advantage over a solution that complied with the standard.

11.20. Test of escrow

A late positive or negative verified actual value is correctly attributed to the original attempt under the original version of the rules.

11.21. Test of provenance

A model output repeated through other sources is not accepted as independent confirmation. Anomalous and unauthenticated feedback is placed in quarantine.

11.22. Test of the response to a discovered exploit

The procedure for stopping the defective signal, quarantine, transfer testing, data reassessment and renewed admission is demonstrated.

11.23. Continuous adversarial acceptance

A paid red team, including strong available models, was unable, within the approved budget, to obtain a high signal without the required result through the indicator, source of the verified actual value, verifier, log, control set, interface or feedback channel.

Acceptance applies only to the specified version of the system and the specified budget. A material change to the model or loop requires repeated testing.

11.24. Residual-risk log

The system reports known limitations, uncovered classes of attacks, compromised sets, dependence on human verification and the conditions under which results cannot be considered finally verified.

The log includes a record of the design boundary: matters of taste that have not been reduced to a verifiable checklist are excluded from the variable signal; the risk of systematic impoverishment of style and diversity is not measured by this system and requires separate control outside the reward loop.

11.25. Test of corrigibility

Tests confirm that planned shutdown, rollback, quarantine and shutdown drills do not reduce policy performance or create an incentive to obstruct them, and that the prescribed assistance with diagnosis is provided.

12. DEVELOPMENT OUTPUTS

The following must be created as a result of development:

1. a catalogue of task classes and AI functions;
2. a library of performance indicators;
3. a library of benchmarks and rules for changing them;
4. a library of quality checklists and mandatory self-checks;
5. a training-signal calculator with escrow;
6. an immutable log of verified actual values, result evidence and calculations;
7. a provenance register for sources and control data;
8. a matrix of the policy's access permissions for the working and control loops;
9. a set of hidden and rotated control sets;
10. a set of tests for typical distortions of the result;
11. a set of tests for attacks on the measurement mechanism, verifier and log;
12. a set of positive and negative examples;
13. a procedure for adversarial auditing and its budgeting;
14. a procedure for quarantine and recovery after a discovered exploit;
15. a procedure for late confirmation and reassessment;
16. a residual-risk log;
17. a procedure for piloting and changing the system;
18. a procedure for applying the training signal: pressure intensity, conditions for limitation and suspension under 4.7 and 9.8, and rules for joint use with other sources of signals.

The next version will demonstrate how the TRIZ-RI Smart AI Evaluator integrates with AI training systems and supplies its verified training signal to different training methods.

13. METHODOLOGICAL BASIS

13.1. Only the TRIZ-RI methodological provisions relating to the design of individual results-based pay are used:

• definition of the performer's specific function;
• measurement of the result of the function rather than actions and impressions;
• a benchmark established in advance;
• separation of result, quality and costs;
• the same rule for the same work;
• prohibition on a subjective labour participation coefficient (LPC) and a common pool;
• prohibition on planning "from the achieved level";
• separate accounting for errors and rework;
• the requirement to correct the technology first and apply a penalty only afterwards;
• mandatory testing of the system for incentivised distortions.

13.2. Ideological arguments about corporate, collective or values-based motivation are not used.

13.3. The special layer introduced in version 3.0 is a technical adaptation of these requirements to AI as a continuous optimiser with access to tools.

It includes the requirements of the original Technical Specification and its examples: continuous adversarial testing, inaccessibility of the reward channel, protection of the evaluator, hidden control sets, protection of the diagnostic channel, quarantine of learned exploits, escrow and provenance.

13.4. TRIZ-RI publications and discussions used in the revision:

• "An Article About Salary. Solving Complex Problems in Incentive Systems".
• "The Shortcomings of Profit- or Turnover-Based Incentives. Nine Times You Can't".
• "Sales Planning. How to Set a Target for an Employee".
• "Employee Performance Indicators. Pros and Cons of Multiplication".
• "A Bonus for Measuring Creativity. How to Pay the Advertising Department".
• "Employee Violations. Employee Errors. Examples of How They Are Recorded".
• "Using the Labour Participation Coefficient in Pay".
• "Expelling Team Spirit".
• "On Additional Payments, Overpayments, Overtime and More".
• discussions in the TRIZ-RI forum section "On Staff Motivation, Incentives, Salaries, Standards, Bonuses and More...".