Rare Disease Diagnostic Research

Research question and target

Rank candidate genes using variants, frequencies and symptom codes. Report conflicting evidence, uncertainty and reference versions.

Internal target

At least 90% of causal genes in the top five

Programme

Discovery and data

Evidence on this record

Published-case evaluationPublic-record audit (acmg19)Synthetic replay (rare19)Common synthetic checks

Evaluation data

172 of 1,874 cases had the reported gene in the top five (9.18%; 95% interval by paper cluster 1.21% to 20.96%). Below the 90% research target.

At least one candidate received a score in 1,873 of 1,874 cases. The reported gene was in the December 2022 knowledge base in 702 of 1,874 cases.

Open the published-case evaluation · Aggregate results (JSON)

• Monarch Phenopacket Store with the December 2022 Human Phenotype Ontology

  Phenopacket Store 0.1.27 · HPO v2022-12-15 · HGNC 2022-10-01

  Manifest locked
• NCBI ClinVar GRCh38 VCF archive

  2022-12-17

  Manifest locked
• Paired patient-level VCF and HPO with adjudicated labels

  Eligible open dataset pending

  Blocked

Evaluation registry · Labels and split definitions (KR)

Public-record audit · acmg19

ACMG criteria claims audited against public records

Every ACMG criterion an agent claims is checked against an evidence ledger built from public records before it can reach a report. On 20 held-out genes the pipeline kept 1,323 of 1,324 expert-benign variants out of the pathogenic classes (99.92%) and called 984 of 1,890 expert-pathogenic variants pathogenic or likely pathogenic (52.06%). The audit gate stopped 9,493 of 9,493 injected unsupported claims and blocked 0 of the 164,867 criteria derived from the ledger.

Records measured, 40 genes

8,667 expert-classified variants · 480 simulated patients

Variant specificity, held-out · development genes

1,323 / 1,324 = 99.92% (95% interval 99.57–99.99) · 681 / 685 = 99.42%

Variant sensitivity, held-out · development genes

984 / 1,890 = 52.06% (95% interval 49.81–54.31) · 912 / 2,112 = 43.18%

Simulated-patient specificity, held-out · development

120 / 120 = 100.00% (95% interval 96.90–100.00) · 120 / 120 = 100.00%

Simulated-patient sensitivity, held-out · development

58 / 120 = 48.33% (95% interval 39.58–57.19) · 66 / 120 = 55.00%

Combining rules reproduce the expert class

8,257 / 8,667 = 95.27%, from the criteria codes the experts recorded

Ledger-derived criteria blocked by the gate

0 of 164,867

Injected unsupported claims stopped

9,493 / 9,493

Ledger tampering rejected

360 / 360

Language-model curator, ledger only

7 criteria blocked, 39 accepted; after one revision 1 blocked, 44 accepted (24 variants)

Language-model curator, recall allowed

23 criteria blocked, 30 accepted; 8 stated classes differed from the audited class

Literature step, held-out sample

19 of 27 findings admitted; pathogenic missense called pathogenic 16 → 18 of 36; second reader azure/gpt-5.4-mini objected to 2 of 28

Limits

• This module measures the audit of ACMG criteria claims. It is a different measurement from the project’s target, the share of causal genes ranked in the top five, and does not show that target has been met.
• Rules and thresholds were refined on the 20 development genes only. The 20 held-out genes were used only to produce the final figures.
• The reference classes are ClinGen expert-panel classifications. The panels drew functional, segregation and case evidence from the literature that this ledger does not yet hold, so the sensitivity figures show what these public records alone establish.
• Patients are simulated: background genotypes drawn from gnomAD frequencies, ClinGen pathogenic or likely pathogenic variants placed in the positive cases, and matched negative controls. Annotation stores, simulated VCFs and model run logs are not published on this site.
• The language-model trial covers 24 variants and one model (Claude Opus (Claude Code subagent, model alias opus)). It identifies the kinds of unsupported claim; it does not estimate how often they occur.
• The literature step was measured on a sample of 96 variants; 36 of the 72 held-out sample variants had retrievable text. Functional and segregation findings from text count at supporting strength only, and second-allele observations are capped at two points.

Public records and simulated patients (public_records_and_simulated_patients). Figures come from ClinGen, ClinVar, gnomAD, HPO and published text, and from patients simulated out of those records. No patient data were used and no clinical performance was measured. No human auditor has signed, and this is not performance of the public call. A separate audit program passed 4 of 4 gates.

Audit record (JSON) · Run output (JSON) · 19-1 ACMG audit gate: method and replay record (KR) · 19-1 ACMG audit gate: cross-model review record (KR)

python3 -B -m acmg19.summarize --check
python3 -B -m acmg19.evaluate > /tmp/acmg19-recheck.json
cmp acmg19/result.json /tmp/acmg19-recheck.json
python3 -B -m unittest discover -s tests -p 'test_acmg19.py'
python3 -B -m acmg19.orchestrate --vcf acmg19/inputs/examples/PAH-02-P.vcf --case acmg19/inputs/examples/PAH-02-P.json --claims acmg19/inputs/examples/PAH-02-P.claims-from-memory.json

Synthetic replay · rare19

Role separation and audit gates on fixed synthetic cases

The runner annotates variants from a fixed VCF subset and ranks candidate genes from HPO phenotypes; a separate audit command rechecks evidence codes, ranks and input hashes. 2 of 4 synthetic cases reached the top five. A tie counts only if the whole tied group fits within five places.

Cases outside the top five

SIM-03, SIM-04

Ties at the cut-off

SIM-01, SIM-03

Limits

• The runner and the audit share the input parser and the fixed inputs. The audit computes the ranking separately; it is not a blinded environment with the reference answers withheld.

Synthetic inputs only (synthetic_only). Scope: synthetic inputs and evidence-code review. Patient VCFs, clinical performance and auditor sign-off are not included. A separate audit program passed 4 of 4 gates.

Audit record (JSON) · Run output (JSON) · 19-1 synthetic replay record (KR)

python3 -B -m rare19.orchestrate --output /tmp/rare19-run.json
python3 -B -m rare19.audit --run /tmp/rare19-run.json --output /tmp/rare19-audit.json

Sources and eligibility

• NCBI ClinVar

  Open

  Includes conflicting interpretations from submitters. Review status and public release dates should be tracked independently.

• gnomAD

  Public aggregate data

  Population frequencies and quality metadata describe the population sample. Case-level diagnosis requires its own reference.

• Human Phenotype Ontology

  Open

  Provides phenotype terms and gene annotations. Coverage gaps and release changes can affect rankings.

Boundary cases

• Gene ranks in synthetic cases containing known variants
• Ranking and evidence changes after missing or miscoded phenotypes
• Reference build mismatches, conflicting classifications and missing literature references

Common synthetic checks

• Top-five ranks and tie handling · 4/4 matched

Expected and observed values for these cases

What the official call asks

At least one clinician must be a principal or co-principal investigator. Papers, patents and stage-by-stage practical-use materials are required, and a disease indicator needs sensitivity and specificity data.

Proposal requirements, printed pages 105–107 · All twelve requirements

Interpretation and next evaluation

Evaluation can overstate performance when inputs already include the variant annotation or gene–disease association used as the reference answer.

Next check

Read the literature for every expert-pathogenic missense variant instead of a sample, so that functional, segregation and case findings enter the ledger. Then re-measure sensitivity on genes outside the current 40, with the condition that specificity does not fall.