Methods & About
Methodology
The auditor decides which evidence supports a result. Curation, computation, counterexample testing and record assembly each have a defined role, and each leaves a record the auditor can replay.
How a claim gets in
A result enters the site through a summary that points at an audit document. Every number in the text is read from that document when the site is built, so the sentence cannot drift from the evidence. If the audit did not pass, or the run it examined has changed, the build stops.
Four audit gates
- 1. Freeze and partition data
Review provenance, access, endpoint labels, the time cutoff, partition rules and input hashes.
- 2. Verify rules and evidence
Recompute values from the evidence. Unregistered data, inconsistent evidence and leakage invalidate a result.
- 3. Counterexamples and independent data
Evaluate on data held out from training. Record a hold when eligibility or endpoints differ.
- 4. Isolated replay and sign-off
Replay frozen inputs and compare results. The auditor records sign-off independently of the automated result.
Research roles
Curation
Pin sources, access, splits and input hashes.
Computation
Generate results under the evaluation specification.
Verification
Record deterministic counterexamples and a review by a model from another family.
Evidence assembly
Link the evidence, result hashes, replay commands and reasons for holds.
Four kinds of number, kept apart
- A plan number is a target written before evaluation.
- A synthetic measurement tests code and decision rules on inputs the code generated.
- A public-data or external-cohort measurement is made on cases the method did not generate.
- An auditor signature records that a person replayed the result and accepted it.
Records for the crisis-prevention projects
Specifications, code, fixtures and evidence for projects 12-1 to 12-4 use abstract policy codes and placeholder tokens. They contain no descriptions of self-harm, no example utterances, no slang and no lists of disallowed sentences. Approved wording is kept outside the repository and compared by hash.
Language of the records
Screens are published in English and Korean. Specifications, replay records and evaluation-lane notes were written in Korean; where an English screen shows one, it is shown as written and marked as Korean.