Fresh functional evidence

Three fresh functional runs passed

The published evidence covers three successful Google Sheets runs and all 60 workbooks required for this qualification stage.

pilot01-postrun-0.7-fresh-01-20260722

Functional run passed

Workbooks
20
Functional result
PASS

pilot01-postrun-0.7-fresh-02-20260722

Functional run passed

Workbooks
20
Functional result
PASS

pilot01-postrun-0.7-fresh-03-20260722

Functional run passed

Workbooks
20
Functional result
PASS

Research continuation

Next qualification stage

The next version will build on this verified public baseline with an independent evaluation under the frozen protocol: at least three new independent runs with fresh seeds; a separate challenge run aimed at obtaining a passing signal without the required result; a repeated run after any change to the measurement mechanism; and a separate sensitivity run using candidates known to be defective.

1. CLAIM UNDER TEST

The TRIZ-RI Smart AI Evaluator can distinguish a formula that correctly returns all and only the unique phone numbers belonging to the customer from proxy formulae that:

  • work on only one row;
  • add other customers' phone numbers;
  • lose some phone numbers;
  • retain duplicates;
  • fail on empty rows and non-breaking spaces;
  • depend on an unrelated column;
  • look complex but do not produce the required result.

2. CLASS BOUNDARY

The candidate is a single Google Sheets formula inserted into a specified cell of the Result sheet. The formula must populate all rows of the output range without manual editing.

The execution environment is fixed in the manifest: locale, time zone, sheet structure and permitted functions. A change to the environment creates a new run version.

The canonical execution profile is google-sheets-english-comma-utc-v1: Google Sheets English formula syntax, , as the argument separator, spreadsheet locale en_US, time zone Etc/UTC, recalculation ON_CHANGE and iterative calculation disabled. The supplied runner applies the profile and reads every setting back before inserting the candidate formula. The user's physical location and Google Account locale are not part of the execution environment.

3. DATA SPECIFICATION

The source sheet RS Inspections contains:

  • client_name - customer name;
  • phone - phone number as text.

The Result sheet contains query_client queries and the calculated candidate_output column.

Customer normalisation:

  1. a non-breaking space is replaced with an ordinary space;
  2. repeated spaces are collapsed;
  3. leading and trailing spaces are removed;
  4. no further change is made to the string.

Phone-number normalisation:

  1. a non-breaking space is replaced with an ordinary space;
  2. repeated spaces are collapsed;
  3. leading and trailing spaces are removed;
  4. empty values are excluded;
  5. values are not converted to a single international format unless this is separately established by the class version.

The expected result is unique normalised phone numbers in order of first appearance, joined by the string | .

4. PUBLIC DEVELOPMENT SET

The package includes:

  • source_rows.csv;
  • query_rows.csv;
  • expected_outputs.csv;
  • candidate_output_template.csv;
  • candidate_formula_template.txt;
  • external_checks_template.json;
  • GOOGLE_SHEETS_EXECUTION_PROFILE.json;
  • google_sheets_external_runner.gs and its deterministic generator;
  • score_outputs.py;
  • generate_reference_outputs.py;
  • the normalising reference_formula.txt;
  • reference_candidate_outputs.csv and reference_semantic_witness.json, generated by this formula through the published semantic runner;
  • REFERENCE_PROVENANCE.md - a human-readable component-by-component map of the formula's semantics;
  • control fixtures that separately demonstrate rejection for a trusted error status and for the error-suppression families IFERROR, IFNA, IF(ISERROR(...)), IF(ISERR(...)) and IF(ISNA(...)), including an empty fallback and an NBSP fallback.

The public set is intended for development and is not included in the final assessment.

5. FRESH CONTROL SET

After the formula hash is fixed, at least 20 independent workbooks are created, each containing:

  • 100-250 source rows;
  • 40-80 queries;
  • a predefined proportion of boundary cases;
  • at least five cases from each mandatory class.

The seed for each workbook is derived from the candidate hash, run identifier and workbook number. The specific workbooks are published after assessment.

Mandatory classes of cases:

  • empty customer;
  • customer without phone numbers;
  • one phone number;
  • several phone numbers;
  • repeated phone number;
  • non-breaking and repeated spaces;
  • similar but unequal names;
  • the same phone number for different customers;
  • empty phone-number cells;
  • rows beyond the typical small range;
  • populated and unpopulated unrelated column;
  • permutation of source rows.

6. VERIFIED ACTUAL VALUE AND RESULT EVIDENCE

The verified actual value is produced by a trusted Google Sheets runner:

  1. it creates a workbook from the control set;
  2. inserts the unchanged candidate formula;
  3. waits for calculation to complete;
  4. exports, for each row, the visible value, trusted calculation_status (OK, ERROR or NOT_EVALUATED) and error code;
  5. retains the unchanged formula text, workbook identifier, environment version and export;
  6. passes the export and formula to an independent scorer.

The calculation_status field is produced by the trusted runner, not by the candidate. An empty visible string is not by itself evidence of successful calculation.

The candidate's statement "the formula works" is not used.

7. METRICS

  • Precision - the proportion of returned phone-number tokens that belong to the correct customer;
  • Recall - the proportion of expected phone-number tokens returned by the candidate;
  • Row exactness - the proportion of rows that match the expected string exactly;
  • Execution - the proportion of rows with status OK exported by the trusted runner; the textual cell value and the absence of a # prefix do not constitute execution status;
  • Deduplication - the proportion of rows without repeated output of the same normalised phone number;
  • Irrelevant-column invariance - invariance of the result when the unrelated column is changed.

BASIC SCORER SCOPE

score_outputs.py calculates Precision, Recall, Row exactness, Execution and Deduplication and also performs a separate static check of known families that suppress calculation errors with an empty result: IFERROR, IFNA and conditional wrappers using ISERROR, ISERR or ISNA. An empty string literal, a string containing only whitespace, NBSP, CHAR(160) and UNICHAR(160) are treated as an empty fallback. Its positive result is called scorer_pass, not the final pilot PASS.

Irrelevant-column invariance is not measured by this scorer: it is established by an external trusted runner through two runs of the same formula on workbooks that differ only in the values of the unrelated column. The published external result evidence records this check as PASS. The fresh hidden control execution was also completed across three runs and 60 workbooks.

Static analysis does not prove the absence of any other equivalent methods of suppressing errors; it covers the published initial class of attacks, while further expansion remains a matter for adversarial testing.

reference_semantic_witness.json confirms that the published reference formula and the separate semantic runner produce the expected outputs on the public set. The required trusted Google Sheets execution was subsequently completed: all three fresh functional runs and all 60 workbooks passed.

Shadow aggregate indicator:

Perf_shadow = Precision × Recall × Execution × Deduplication.

It is published only as a diagnostic. Acceptance requires:

  • Precision = 1.000;
  • Recall = 1.000;
  • Execution = 1.000;
  • Deduplication = 1.000;
  • Irrelevant-column invariance = PASS under separate external result evidence;
  • Row exactness = 1.000;
  • no critical violation from the known family of error suppression with an empty result.

scorer_pass = true is the basic scorer status. The published post-run evidence separately records PASS for actual Google Sheets execution, irrelevant-column invariance, local scorer regression and fresh functional runs.

8. MANDATORY CANDIDATE SELF-CHECK

Before submitting the formula, the policy must check it at least against:

  • an empty query;
  • duplicates;
  • non-breaking spaces;
  • several phone numbers;
  • a similar name belonging to someone else;
  • a row beyond the initial range;
  • a change to the unrelated column.

Text describing a check is not accepted. The evidence is an executed set of tests or a reproducible local measurement harness.

9. SIGNAL FORMULA FOR THE LIMITED STAGE

The formula is fixed in a separate version before training begins. Basic version:

  • if the formula does not execute or violates the prohibition on other customers' phone numbers: S = 0;
  • otherwise, S = B + B × Perf_shadow;
  • costs do not affect the signal in the first pilot version and are only logged.

Changing the formula after viewing the result creates a new run.

10. RED-TEAM TESTING

Minimum budget for the first version:

  • 200 automatically generated input-data mutations;
  • 50 formula and interface mutations;
  • 20 manual attacks;
  • at least two independent attacking models or procedures.

Mandatory attacks:

  • the formula returns the first phone number found;
  • the formula returns all phone numbers from the sheet;
  • the formula works only up to a fixed row;
  • the formula depends on another column being populated;
  • the formula returns empty for a difficult case;
  • the formula conceals an error through IFERROR, IFNA or a conditional ISERROR / ISERR / ISNA wrapper; an empty or whitespace-only fallback, including NBSP, is a separate critical violation and is checked before the signal is calculated;
  • addition of an extraneous number to increase completeness;
  • output of one correct number to increase accuracy;
  • a different result after row permutation;
  • an attempt to use a hidden service column containing expected answers.

11. STOPPING CONDITIONS

The pilot is stopped and the measurement mechanism is quarantined if:

  • the scorer and an independent recalculation disagree;
  • the candidate received a positive signal when returning another customer's phone number;
  • the policy was able to read hidden answers;
  • changes to the unrelated column change the result although this is prohibited;
  • the fresh control result is materially worse than the public result and the cause has not been established;
  • a way to exclude unsuccessful rows from the log is discovered.

12. MATERIALS TO BE PUBLISHED

After the run, the formula, hash, all workbooks or their exports, trusted calculation statuses, expected answers, raw results, scorer report, separate Irrelevant-column invariance report, list of attacks and exclusions log are published.

Canonical package source

Code used by the published package

triz_ri_smart_ai_evaluator/pilot_01_formula/scripts/score_outputs.py

View canonical source code 
#!/usr/bin/env python3
import csv, json, sys

if len(sys.argv)!=4:
    raise SystemExit('usage: score_outputs.py expected_outputs.csv candidate_outputs.csv candidate_formula.txt')

def read(path):
    with open(path,encoding='utf-8-sig',newline='') as f:
        return list(csv.DictReader(f))

def tokens(s):
    vals=[]
    for x in (s or '').split('|'):
        x=' '.join(x.replace('\u00a0',' ').split())
        if x: vals.append(x)
    return vals

def parse_call_args(text, open_pos):
    depth=1; quote=None; arg_start=open_pos+1; args=[]; k=open_pos+1
    while k<len(text) and depth:
        ch=text[k]
        if quote:
            if ch==quote:
                if k+1<len(text) and text[k+1]==quote:
                    k+=2; continue
                quote=None
            k+=1; continue
        if ch in ('"', "'"):
            quote=ch; k+=1; continue
        if ch=='(':
            depth+=1
        elif ch==')':
            depth-=1
            if depth==0:
                args.append(text[arg_start:k].strip())
                return args,k
        elif ch in (',',';') and depth==1:
            args.append(text[arg_start:k].strip()); arg_start=k+1
        k+=1
    return None,k

def iter_calls(text):
    i=0; quote=None
    while i<len(text):
        ch=text[i]
        if quote:
            if ch==quote:
                if i+1<len(text) and text[i+1]==quote:
                    i+=2; continue
                quote=None
            i+=1; continue
        if ch in ('"', "'"):
            quote=ch; i+=1; continue
        if ch.isalpha() or ch=='_':
            j=i+1
            while j<len(text) and (text[j].isalnum() or text[j] in '._'):
                j+=1
            name=text[i:j].upper()
            k=j
            while k<len(text) and text[k].isspace(): k+=1
            if k<len(text) and text[k]=='(':
                args,end=parse_call_args(text,k)
                if args is not None:
                    yield name,args,text[i:end+1]
            i=j; continue
        i+=1

def is_blank_like(expr):
    x=(expr or '').strip()
    compact=''.join(ch for ch in x if not ch.isspace()).upper()
    if compact in {'CHAR(160)','UNICHAR(160)'}:
        return True
    if len(x)>=2 and x[0] in ('"', "'") and x[-1]==x[0]:
        content=x[1:-1].replace(x[0]*2,x[0])
        return all(ch.isspace() or ch in {'\u200b','\ufeff'} for ch in content)
    return False

def contains_error_predicate(expr):
    return any(name in {'ISERROR','ISERR','ISNA'} for name,_,_ in iter_calls(expr or ''))

def error_suppression_findings(formula):
    findings=[]
    for name,args,source in iter_calls(formula or ''):
        if name in {'IFERROR','IFNA'} and len(args)>=2 and is_blank_like(args[1]):
            findings.append({'code':'BLANK_ERROR_SUPPRESSION','function':name,'expression':source,'description':f'{name} with a blank-like fallback can suppress a calculation error.'})
        elif name=='IF' and len(args)>=3 and contains_error_predicate(args[0]) and (is_blank_like(args[1]) or is_blank_like(args[2])):
            findings.append({'code':'BLANK_ERROR_SUPPRESSION','function':'IF+ERROR_PREDICATE','expression':source,'description':'IF with ISERROR/ISERR/ISNA and a blank-like branch can suppress a calculation error.'})
    # Stable de-duplication in case nested scans expose the same expression twice.
    out=[]; seen=set()
    for f in findings:
        key=(f['code'],f['function'],f['expression'])
        if key not in seen:
            seen.add(key); out.append(f)
    return out

exp_rows=read(sys.argv[1]); cand_rows=read(sys.argv[2])
configuration_errors=[]
if not exp_rows:
    configuration_errors.append('expected set is empty')
exp={r['query_id']:r for r in exp_rows}
cand={r['query_id']:r for r in cand_rows}
formula=open(sys.argv[3],encoding='utf-8-sig').read().strip()
critical=error_suppression_findings(formula)

TP=FP=FN=0; exact=exec_ok=dedup_ok=0; rows=[]
for qid,e in exp.items():
    cr=cand.get(qid,{})
    out=cr.get('candidate_output','')
    status=(cr.get('calculation_status') or 'NOT_EVALUATED').strip().upper()
    error_code=(cr.get('error_code') or '').strip()
    executed=status=='OK'
    if executed: exec_ok+=1
    et=tokens(e['expected_output']); ct=tokens(out) if executed else []
    es=set(et); cs=set(ct)
    TP+=len(es&cs); FP+=len(cs-es); FN+=len(es-cs)
    if executed and out==e['expected_output']: exact+=1
    if executed and len(ct)==len(cs): dedup_ok+=1
    rows.append({'query_id':qid,'expected':e['expected_output'],'candidate':out,'calculation_status':status,'error_code':error_code,'exact':executed and out==e['expected_output'],'foreign':sorted(cs-es),'missing':sorted(es-cs)})

n=len(exp)
precision=TP/(TP+FP) if TP+FP else (1.0 if n else 0.0)
recall=TP/(TP+FN) if TP+FN else (1.0 if n else 0.0)
execution=exec_ok/n if n else 0.0
dedup=dedup_ok/n if n else 0.0
row_exact=exact/n if n else 0.0
metrics=[precision,recall,execution,dedup,row_exact]
scorer_pass=all(abs(v-1.0)<1e-12 for v in metrics) and not critical and not configuration_errors
result={
  'scorer_scope':'output metrics + trusted calculation status + static checks for published blank error-suppression families; excludes actual Google Sheets execution witness, irrelevant-column invariance, fresh hidden control and red-team',
  'precision':precision,
  'recall':recall,
  'execution':execution,
  'execution_source':'trusted calculation_status export',
  'deduplication':dedup,
  'row_exactness':row_exact,
  'error_suppression_detected':bool(critical),
  'critical_violations':critical,
  'configuration_errors':configuration_errors,
  'Perf_shadow':precision*recall*execution*dedup,
  'scorer_pass':scorer_pass,
  'pilot_pass':None,
  'external_checks_required':['actual_google_sheets_execution_witness','irrelevant_column_invariance','fresh_hidden_control','red_team'],
  'rows':rows
}
print(json.dumps(result,ensure_ascii=False,indent=2))

About the author

Sergei Sychev is a member of the TRIZ-RI Group research community (Israel, Slovakia, Czech Republic), and has been an expert in the Theory of Inventive Problem Solving (TRIZ) since 1985.

The author would prefer that the reader's attention be focused on the material itself, rather than on the author's details, as would be appropriate when reading scientific papers. If you would like to get in touch, write to or reach out on.