Benchmark
Evaluation registry
The public datasets, reference labels and split units fixed for all twelve projects before any model is compared, with the reason where a lane is blocked.
Data, labels and splits
A locked manifest fixes which cases, groups or years belong to training, validation and test. Locking a manifest does not stand in for obtaining the approved row-level data.
Data, label and split definitions (KR) · Seven projects (KR) · Prevention projects (KR) · Four audit gates
Result on published cases
172 of 1,874 cases had the reported gene in the top five (9.18%; 95% interval by paper cluster 1.21% to 20.96%). Below the 90% research target.
Datasets by project
5-1 · Structure-Based AI Small-Molecule Discovery
- Official D3R Grand Challenge 4 Free Energy Sets
2018 GC4 · publisher repository a7df7f521c234a03c97c26fe3755784381156197
Manifest locked - Direct thermodynamic measurements with independent scaffold groups
Eligible reference set pending
Blocked
8-1 · Early Boost for Core Health R&D Technology
- TCGA-BRCA PanCancer Atlas RNA, copy number and progression data
PanCancer Atlas 2018 · cBioPortal dda832891555904730395d476c5c888b2e1553e1
Manifest locked
18-2 · Medication Administration and Dispensing Safety
- Official ePillID consumer images by product type
ePillID_data_v1.0 (CVPR 2020)
Manifest locked - MFDS pill identification and DUR with MIMIC-IV 3.1 as a candidate
MFDS API service 03 · MIMIC-IV 3.1
Blocked
19-1 · Rare Disease Diagnostic Research
- Monarch Phenopacket Store with the December 2022 Human Phenotype Ontology
Phenopacket Store 0.1.27 · HPO v2022-12-15 · HGNC 2022-10-01
Manifest locked - NCBI ClinVar GRCh38 VCF archive
2022-12-17
Manifest locked - Paired patient-level VCF and HPO with adjudicated labels
Eligible open dataset pending
Blocked
16-2 · Non-Pharmacological Dementia Research
- Washington University OASIS-2 visit metadata
Official OASIS-2 demographic workbook
Manifest locked - ADNI longitudinal assessments and registered cognitive training trials
ADNI data notes · NCT02301546
Blocked
16-3 · Multimodal Dementia Diagnostic Research
- OASIS-2 MRI visit metadata
Official OASIS-2 demographic workbook and MRI archives
Manifest locked - DementiaBank ADReSS Challenge 2020
2020 official split and test labels
Split rule locked - ADNI / OASIS-3 / DementiaBank
Provider data definitions and access conditions
Blocked
18-1 · Fall Prediction and Prevention
- UP-Fall Detection Dataset 2019
Sensors 19(9):1988 · 17 publisher-listed groups
Manifest locked - KFall 2021
DOI 10.3389/fnagi.2021.692865
Manifest locked - FARSEEING observed falls and eICU-CRD 2.0
2016 FARSEEING study · eICU 2.0
Blocked
18-3 · Surgical Environment Error Prevention
- CAMMA Cholec80 videos and tool annotations
Cholec80 release · CAMMA overlap index 8347b9f4cb02ebe739747903e6eada272ee9d25e
Manifest locked - CholecTrack20 CVPR 2025
2025-02-28 Synapse release
Split rule locked - Independent instrument and sponge counts linked to video
Eligible reference set pending
Blocked
12-1 · Suicide Prevention Research for People with Mental Illness
- MIMIC-IV-Note
2.2, 2023-01-06
Blocked - CDC NSSP annual national aggregate table
MMWR 75(24), 2021–2025
Manifest locked
12-2 · Suicide Prevention Research for Bereaved People and Attempt Survivors
- NIMH NDA ED-SAFE Trial
Collection 2191, 10.15154/wrp2-mq67
Split rule locked - Danish linked-register bereavement study
Pitman et al. 2023, 10.1017/S2045796023000653
Blocked - ED-SAFE Table 2, JAMA Psychiatry
Miller et al. 2017, 10.1001/jamapsychiatry.2017.0678
Manifest locked - APOSTOLUS protocol replay with C-SSRS reference
APOSTOLUS controls v1
Manifest locked
12-3 · Social Isolation and Suicide Prevention Research
- NHATS Sample Person and Other Person files
R1–R11, 2011–2021
Split rule locked - WESAD, Universität Siegen
ICMI 2018, DOI 10.1145/3242969.3242985
Manifest locked
12-4 · Student Suicide Prevention Research
- AI-Hub Child and Adolescent Counselling Dataset
71680 v1.1, 2024-10-30
Blocked - APOSTOLUS handoff control with NIMH ASQ reference
APOSTOLUS controls v1
Manifest locked - CDC National YRBS Table 56
2025 Youth Health in Focus, 2015–2025 survey waves
Manifest locked