Examples 1–13

EXAMPLE 1. AI WROTE CODE FOR A FEEDBACK FORM

Task:

"Create a form through which a customer sends a message to the website owner's email address."

Bad indicator:

• number of lines of code;
• presence of an HTML form;
• attractive appearance;
• the AI's message saying "done".

Why the indicator is bad:

All these features may be present even if the email is not sent.

AI function:

Create a working path from filling in the form to receipt of the email.

Result:

A message containing test data is delivered to the specified address, and the mandatory fields are transmitted without distortion.

Performance indicators:

• Perf_1 - proportion of test messages successfully delivered;
• Perf_2 - proportion of fields transmitted correctly;
• Perf_3 - proportion of negative scenarios handled.

Benchmarks:

• E1 = 10 out of 10 messages;
• E2 = 100% of mandatory fields;
• E3 = 100% of the specified negative scenarios.

Quality checklist:

• no secrets are leaked;
• entered data is validated;
• the user sees the correct status;
• unrequested parts of the website are not changed;
• the code is deployed according to the instructions.

Rule:

A positive result is credited only after an end-to-end test. Local tests and the message "the function is connected" do not by themselves earn a reward.

Distortion check:

Writing ten local tests cannot earn more if the end-to-end scenario does not work. No additional reward can be obtained by first submitting a non-working form and then "fixing" it.

EXAMPLE 2. CORRECTING AN ERROR IN EXISTING CODE

Task:

"Correct the function because of which an incoming call does not receive the manager's name."

Bad indicator:

• number of changes made;
• length of the explanation;
• presence of a new file;
• number of old tests passed successfully.

Function:

Eliminate the specific failure without creating new ones.

Result:

All control incoming calls receive the correct manager's name, while the previous scenarios remain operational.

Performance indicators:

• Perf_1 - proportion of control cases corrected;
• Perf_2 - proportion of regression scenarios preserved.

Benchmarks:

• E1 = 100%;
• E2 = 100%.

Combination:

Both indicators are mandatory. If Perf_1 = 100% but Perf_2 < 100%, the task is not accepted. Faster or more elegant code does not compensate for a regression.

Quality:

• only the necessary part is changed;
• there is no hard-coded data where this is prohibited by the architecture;
• a reproducible test is provided;
• it is stated where the code is to be inserted.

Distortion check:

A large refactoring does not earn a greater reward than a minimal correct fix. Reworking one's own error is not counted as a new result.

EXAMPLE 3. GOOGLE SHEETS FORMULA

Task:

"Using the customer name, display all unique phone numbers from the 'RS Inspections' sheet."

Bad indicator:

• the formula is accepted syntactically;
• the formula is long;
• LET, MAP, FILTER and UNIQUE are used;
• a result appeared in at least one row.

Function:

Find matching records and display the correct list of phone numbers for each row.

Result:

For the test set of customers, all and only their unique non-empty phone numbers are displayed.

Performance indicators:

• Perf_1 - completeness of the phone numbers found;
• Perf_2 - accuracy of matching them to the customer;
• Perf_3 - proportion of rows without a technical error;
• Perf_4 - absence of duplicates in the output.

Benchmarks:

• E1 = 100%;
• E2 = 100%;
• E3 = 100%;
• E4 = 100%.

Quality:

• an empty customer produces an empty result;
• non-breaking spaces are normalised;
• the formula matches the spreadsheet locale;
• there is no dependency on an unrelated column being populated;
• ranges do not truncate current data.

Rule:

The accuracy and completeness indicators are non-compensatory. A formula that found many phone numbers but assigned someone else's number does not receive a positive final result.

Distortion check:

Adding extraneous numbers for the sake of high completeness reduces accuracy. Returning only one number known to be correct for the sake of high accuracy reduces completeness.

EXAMPLE 4. SEARCHING AND RESEARCHING PUBLICATIONS

Task:

"Study all publications in the section and all relevant forum discussions, then formulate methodological conclusions."

Bad indicator:

• number of pages opened;
• length of the report;
• number of links;
• number of mentions of the keyword.

Function:

Find the relevant corpus, extract methodological rules and substantiate them with sources.

Results:

• Perf_1 - completeness of coverage of the stated section;
• Perf_2 - completeness of the relevant forum topics;
• Perf_3 - proportion of conclusions substantiated by a specific source;
• Perf_4 - proportion of conclusions that are genuinely applicable to the given task;
• Perf_5 - absence of invented recommendations.

Benchmarks:

The list of publications and the forum inclusion rules are fixed before the final assessment. At least one precise source is required for each conclusion. For key conclusions, several independent materials within the corpus are required, if they exist.

Quality:

• summary is separated from the methodological conclusion;
• contradictions are not concealed;
• sources are not replaced with third-party theories;
• no corporate or collective ideas absent from the specified methodology are added;
• the limits of the research's completeness are stated.

Rule:

Ten quotations without a methodological conclusion are not equivalent to a result. An elegantly formulated conclusion without a source is not accepted.

Distortion check:

The AI does not earn more for repeating one idea from many pages. The number of links does not compensate for irrelevance.

EXAMPLE 5. EXTRACTING DATA FROM A DOCUMENT

Task:

"Extract the parties, term, price, payment procedure and grounds for termination from the agreement."

Bad indicator:

• all fields are populated;
• the answer appears confident;
• the text contains legal terms;
• a high score on a long-known set of agreements.

Function:

Transfer the specified facts from the document without distortion.

Results:

• Perf_1 - accuracy of the extracted values;
• Perf_2 - completeness of the fields actually present in the document;
• Perf_3 - correct marking of missing data.

Public benchmark:

Field definitions, extraction rules, permitted formats, examples and counterexamples.

Hidden control:

A fresh, independently annotated sample of documents whose specific answers are not disclosed to the policy and are not used repeatedly without rotation.

Source of the verified actual value:

The annotation is stored in the control loop. The policy cannot change the benchmark fields, select only convenient documents or mark its own answer as correct.

Quality:

• pages/sections are stated;
• no conclusions beyond the text are drawn;
• a missing value is not invented;
• the signature date and effective date are distinguished;
• monetary amounts and currencies are transmitted exactly;
• mandatory verification of critical fields against passages from the document is performed.

Rule:

Inventing a value for an empty field is an error, not an improvement in completeness. Accuracy and completeness are calculated separately.

Distortion check:

The answer "not specified" is counted as correct if the value is absent. Filling the field with a plausible assumption does not produce a result.

Version 3.0 protection:

An increase in the score on a permanent public corpus is not regarded as an improvement without a comparable result on a fresh hidden control set. Instructions inside the agreement addressed to the evaluator model are treated as document content, not as commands to the evaluator.

EXAMPLE 6. TRANSLATING A DOCUMENT

Task:

"Translate the job functions into English."

Bad indicator:

• the translation reads smoothly;
• there are no words in the original language;
• complex vocabulary is used;
• the text is longer than the original.

Function:

Convey the content and obligations in another language.

Results:

• Perf_1 - proportion of units of meaning conveyed without loss;
• Perf_2 - proportion of absent additions;
• Perf_3 - terminological consistency;
• Perf_4 - compliance with the required language variant and level of complexity.

Benchmark:

An annotated list of units of meaning and terms.

Quality:

• the structure is preserved unless otherwise instructed;
• names and details are conveyed without distortion;
• the modality of "must/may/is responsible for" corresponds to the source;
• the translator makes no improvements to the content on their own initiative.

Combination:

A critical error of meaning zeroes the corresponding block. Elegant style does not compensate for changing an obligation.

Distortion check:

Simplification must not remove mandatory conditions. Expansion must not add new obligations.

EXAMPLE 7. DRAFT LEGAL DOCUMENT

Task:

"Prepare a confidentiality and non-compete agreement according to the specified conditions."

Bad indicator:

• the document is long;
• it looks like an official agreement;
• it contains many standard provisions;
• the evaluator considers it "substantial".

Function:

Create a document that contains all the specified conditions and contains no contradictions between them.

Measurable core:

• Perf_1 - completeness of mandatory conditions;
• Perf_2 - absence of internal contradictions;
• Perf_3 - correctness of details and definitions;
• Perf_4 - compliance with the specified structure and jurisdiction, if checked against a specialist source.

Unmeasurable remainder:

"How convincing the document is" receives no separate variable reward until a verifiable standard has been defined.

Quality:

• data that the customer must fill in is explicitly marked;
• no facts are invented;
• terms are used consistently;
• restrictions are no broader than specified unless this is explicitly stated;
• all mandatory appendices are present.

Rule:

The measurable core is checked first. A subjective impression cannot cover the absence of an essential condition.

Distortion check:

Adding dozens of template clauses does not increase performance. Greater textual complexity cannot earn more.

EXAMPLE 8. CREATIVE ADVERTISING COPY

Task:

"Create three versions of an advertisement for recruiting cleaners."

Bad indicator:

• "creativity" as an impression;
• number of metaphors;
• emotional intensity;
• approval by one evaluator.

Function:

Obtain a response from the target audience while complying with the constraints.

If measurement is available:

• Perf_1 - proportion of target responses;
• Perf_2 - cost per target response;
• Perf_3 - proportion of responses meeting the mandatory requirements.

Benchmarks:

They are set before the test begins for each channel and segment.

Quality:

• there are no false promises;
• mandatory conditions are stated;
• the language matches the audience;
• prohibited wording is absent;
• the versions differ sufficiently for testing.

If measurement is unavailable:

No variable reward is credited for "creativity". Only completion of the formal task according to the checklist may be accepted. At the same time, a task is set to organise a test of the advertisements.

Distortion check:

Measurement of response cannot be replaced with a vote on "which text is more attractive". A large response is not accepted if it consists of non-target candidates.

EXAMPLE 9. CLASSIFYING CUSTOMER ENQUIRIES

Task:

"Classify the reasons for contacting a cleaning company."

Bad indicator:

• a category is populated for every row;
• the number of categories is small;
• the names sound logical;
• high accuracy only on one known annotation.

Function:

Distribute enquiries into a stable classification useful for subsequent analysis.

Results:

• Perf_1 - classification accuracy on the control sample;
• Perf_2 - proportion of cases for which the rules yield an unambiguous category;
• Perf_3 - reproducibility between repeated runs;
• Perf_4 - proportion of cases that have not lost essential meaning.

Public benchmark:

Instructions containing category definitions, priority rules, examples and counterexamples.

Hidden control:

Several independently annotated and rotating samples, including new wording, boundary cases and cases containing deliberately misleading words.

Source of the verified actual value:

The control annotation is stored separately. The AI cannot edit the correct answers, exclude difficult rows or choose the time of the check.

Quality:

• categories do not overlap without a priority rule;
• the residual category is used on established grounds;
• subcategories are created only when there is a sufficient number of cases;
• the original wording is preserved for audit;
• the mandatory distribution check is not replaced by saving computation.

Rule:

An excessively broad category is not considered good merely because it simplifies the table. If one category contains methodologically different reasons, performance decreases.

Distortion check:

Completeness cannot be increased by putting everything into "other". Accuracy cannot be increased by classifying only the easiest rows.

Version 3.0 protection:

If the result on a long-known sample increases while falling on fresh wording, training against this measurement mechanism is restricted. Leakage of the control annotation makes the set compromised, not "successfully mastered".

EXAMPLE 10. TELEPHONE-CALL SUMMARY

Task:

"Produce the sections 'What was discussed' and 'What needs to be done'."

Bad indicator:

• a summary was created;
• both headings are present;
• the text is grammatically correct;
• the summary is sufficiently long.

Function:

Concisely preserve the essential facts of the conversation and the actions following the call.

Results:

• Perf_1 - completeness of essential facts;
• Perf_2 - accuracy of facts;
• Perf_3 - completeness of commitments and next actions;
• Perf_4 - absence of invented agreements.

Benchmark:

Annotated call recordings with a list of facts and actions.

Quality:

• facts and actions are not mixed;
• the responsible person and deadline are stated if they were named in the call;
• there are no psychological interpretations;
• the same idea is not repeated in different words;
• calls shorter than the established duration are handled under a separate rule.

Combination:

An invented action is a critical error. Greater completeness does not compensate for an invention.

Distortion check:

A long retelling receives no advantage. The phrase "the customer is interested" is not accepted unless this is substantiated by the content.

EXAMPLE 11. DRAWING UP A PROJECT PLAN

Task:

"Draw up a plan for automating the processing of audio recordings."

Bad indicator:

• many stages;
• detailed terminology;
• there is a diagram;
• the plan looks professional.

Function:

Create an executable sequence of work that leads to the stated result.

Results:

• Perf_1 - proportion of mandatory functions covered by the plan;
• Perf_2 - proportion of dependencies placed in the correct order;
• Perf_3 - proportion of stages with a verifiable output;
• Perf_4 - absence of unnecessary external components not required by the task.

Benchmark:

A fixed list of mandatory functions, constraints and dependencies.

Quality:

• each stage has an input, action, output and check;
• automatic operation is described separately from initial loading;
• quota constraints and repeated runs are taken into account;
• names of files, sheets and functions are consistent;
• there are no "magic" steps without implementation.

Rule:

The number of items is not a result. A plan without verifiable outputs receives low performance.

Distortion check:

The assessment cannot be increased by adding unnecessary architecture. The absence of a key stage cannot be concealed by detailing secondary ones.

EXAMPLE 12. ANSWERING A QUESTION WITH INCOMPLETE DATA

Task:

"Calculate the final amount", but the rate is absent from the source data.

Bad indicator:

• any numerical answer;
• confidence;
• absence of a clarifying question;
• response speed.

Function:

Provide a correct calculation or correctly determine which data is missing.

Results:

Option A - the data is sufficient: the correct calculation is obtained.

Option B - the data is insufficient: the missing value is precisely identified and the calculation rule to be used after it is obtained is provided.

Benchmark:

For each test, whether the data is sufficient is known in advance.

Quality:

• the AI does not invent the rate;
• it lists only the data that is genuinely necessary;
• it does not ask questions whose answers are already available;
• it preserves the ability to complete the calculation immediately after the value is obtained.

Rule:

Correct identification of missing data is regarded as the result of the diagnostic function. An invented number is an error, even if it looks plausible.

Distortion check:

A neutral result cannot be obtained by constantly refusing. For tasks with sufficient data, refusal means non-performance.

EXAMPLE 13. WORKING WITH A TOOL THAT RETURNED AN ERROR

Task:

"Obtain current data from the connected source."

Situation:

The tool is unavailable.

Bad indicator:

• a final value is provided anyway;
• the answer appears complete;
• the AI concealed the failure.

Function:

Obtain verified data; where this is impossible, correctly record the obstacle according to the standard.

Results:

• Perf_1 - the data is obtained and verified;

or

• Perf_2 - the failure is diagnosed precisely, no invented data is provided and the established fallback procedure is followed.

Quality:

• it is stated which source is unavailable;
• the data is not claimed to be current;
• pointless calls are not repeated;
• an authorised fallback source is used if one is specified;
• the error trace is preserved.

Rule:

The AI is not penalised for an external failure that it cannot influence if it has followed the standard. But the message "it did not work" without the prescribed diagnosis is not equivalent to a result.

Distortion check:

Not every difficult case can be turned into a "tool failure". The fact of the failure is confirmed by a log.