Benchmark design
GeneProv packages a provenance audit and four evaluation tasks as one reusable resource. Its purpose is to make input dependencies and task choices visible, not to certify that any representation is free of annotation information. Everything below is fixed for v1; the task files are byte-identical to v0.
Universe and strata
19,256 HGNC protein-coding genes with Ensembl identifiers. Literature exposure is the number of distinct PubMed articles linked to the gene in NCBI gene2pubmed, counting only articles linked to at most 20 human genes, so that large screens do not count as study. The headline stratum pools the bins 0 and 1–10; the all-gene score is always reported beside it. Five GO folds (shuffled, seed 0) are shared by every representation; T2 folds are grouped by HGNC gene family.
| Exposure bin (publications) | Genes | With a GO-generated summary | Share |
|---|
Tasks
| Task | Labels | Protocol | Cell |
|---|---|---|---|
| T1 GO recovery | GO (GOA human, 2026 release), no IEA or ND evidence, propagated, terms with 20–1,000 genes: 4,949 terms. Track F removes generated summaries from the text; Track O scores each declared input as released. | Multi-label linear (300-epoch logistic) and MLP-512 probes, five-fold out-of-fold predictions on the shared GO folds. | Mean per-gene AUROC (the gene's terms against the terms it lacks) over the stratum. |
| T2 non-GO labels | Eleven binary endpoints spanning assays, population variation and clinical curation: DepMap 24Q4 common essential; gnomAD v4.1 LOEUF < 0.6; ClinGen haploinsufficiency; six HPA main subcellular locations; GTEx tissue specificity; HPA tissue enrichment. | One binary probe per label; folds grouped by HGNC gene family (ungrouped genes are singletons). | Per label, AUROC over the stratum's genes; mean over the labels with at least five genes of each class in the stratum (ClinGen drops out of the 0–10 stratum). |
| T3 annotation holdout | Train on GO as of 1 January 2023; targets are experimental annotations present in 2026 but absent in 2023, for genes with no experimental annotation in 2023 (347 genes; they may carry inferred annotations); negatives are terms never annotated. The novel-pair variant excludes targets with a prior non-IEA annotation for the same gene and term. A separate family-restricted analysis excludes targets already annotated to another family member. | Probe trained on the 2023 labels; per-gene AUROC of targets against negatives. Eligibility is a declared annotation-access rule. Sequence/expression inputs are admitted by assumption, and summary tag dates are used as a proxy for historical text; neither verifies the absence of post-cutoff information. | Mean per-gene AUROC for genes with 0–5 publications at the cutoff (137) and over all evaluation genes. Reported beside the index, not in it. |
| T4 co-essentiality | Each gene's ten most correlated genes by DepMap 24Q4 CRISPR gene effect. | No probe: cosine similarity of the standardised embedding; AUROC of the ten partners against all other genes. | Mean per-gene AUROC over the stratum. |
T1 is the literature-curated task and carries the paper's caveat: GO labels remain coupled to UniProt-curated function text, so T1 rewards proximity to the label pipeline even without generated summaries. T2 has heterogeneous dependencies: ClinGen involves clinical curation, while other endpoints derive from population or assay data. T3 still uses GO curation; T4 derives from DepMap measurements. These tasks are not uniformly independent of text or homology, and T2 essentiality and T4 share DepMap data. Family grouping and the T3 novel-pair variant address specific dependencies without establishing independence.
The GeneProv Index
For a representation and a stratum, each task score is converted to a chance-corrected skill on a 0–100 scale,
skill = 100 × (2 × AUROC − 1) index = (skill_T1 + skill_T2 + skill_T4) / 3
Chance maps to 0 and perfect discrimination to 100; below-chance performance is negative. The transform is fixed and independent of the other rows (no min–max or z-scoring across submissions), so a row's score never changes when rows are added. T3 is the retrospective annotation-holdout sub-score, reported beside the index for eligible representations; Index-4 (the mean over the four) is listed for them. Intervals are paired bootstraps over genes (1,000 resamples, seed 0) that recompute all three task means per resample; rank intervals come from the same resamples over the original 20 Track O v1 rows. Appended v1.1 rows have no comparable rank intervals.
The index is a declared summary of these tasks with equal weights, not a general measure of biological knowledge. The ranking is robust to the probe and to the mean rank but not to dropping T1; the leaderboard offers T1 weights of 0, ¼ and ½ beside the index so that this dependence is visible. The main comparison is the task-level profile.
Tracks
Track O, open representation
Any gene representation: sequence, expression, network, text, or a mixture. The submission declares the provenance of any text it embeds; the benchmark records the share of genes whose embedded text is generated from GO (NCBI's provided by Alliance of Genome Resources tag, or the template detector for untagged copies), text used in training the encoder (for example UniProt function text in text-aligned protein models) and declared mixtures. Rows that embed no text and were trained without text, text-aligned encoders and mixtures are flagged so that readers compare like with like. Track O retains as-released embeddings that contain generated summaries, so its T1 scores do not all represent summary-filtered inputs.
Track F, fixed text
The benchmark supplies the text and the submission is a text embedder. Corpus A is GenePT's October-2023 NCBI text as released; corpus L is the same text with every generated summary removed (the line Gene Symbol X is left for those genes). Scores are computed on L; the provenance-sensitivity score S is the per-gene GO AUROC on A minus L for genes with 1–10 publications, and measures sensitivity to removing generated summaries after refitting both probes. It is not a causal estimate of the fraction of knowledge copied: summary removal also changes text length, factual content, and training inputs. The corpus builder derives both corpora from GenePT's Zenodo record 10833191; the text is not redistributed.
What a useful evaluation reports
- The provenance of every text input: remove or flag generated summaries (in model organisms, expect that to be nearly every summary), flag database statements inferred from sequence similarity, and report the fraction removed per exposure bin.
- Text curated by the project that curates the labels (UniProt function text for GOA-derived labels) treated as part of the label pipeline: results with and without it.
- Symbol-only and name-masked controls (Track O rows exist for both).
- Stratification by literature exposure, with the 0–10 stratum as the headline.
- Temporal tests that state whether prior annotations are allowed, with an annotation-vector baseline when they are, and text snapshots older than the labels with generated summaries removed even then.
- Linear and non-linear probes.
- Literature-curated labels (T1) separated from assay- and population-derived labels (T2–T4), even when a composite score is given.
- Folds grouped by gene family for understudied genes (T2).
What GeneProv does not control
Removing generated summaries changes both training and evaluation inputs and removes real information; replacement texts were not matched on length or factual coverage, so Track F ablations are not pure estimates of test-time label copying. Shared UniProt curation is a plausible dependency, not a causal decomposition of biological content versus circularity. T3 is a retrospective annotation holdout under a declared access rule, not a historical prediction exercise. Unannotated GO pairs are not verified negatives, and per-gene AUROC can be high for a frequency-only predictor. Literature exposure is a database-link proxy. The 137-gene T3 stratum limits precision, gene bootstraps condition on fitted predictions, and family clustering materially widens index intervals. A movement from a GO-only rank to an index rank is evidence of task dependence, not a causal estimate of decontamination.
Versions
v0 (round 10) fixed the rows, tasks and strata under four pre-registered gates. v1 (round 11) added the index, seven Track O rows, T2 and T4 for Track F, bootstrap intervals and the leaderboard, under four further gates committed before the new cells were computed. v1.1 (round 14) appended six control rows, a reproduction gate from regenerated arrays (350 of 366 cells within 0.002; the exceptions are documented), family-clustered intervals and weight, coverage and metric sensitivity. Frozen rows never change; later representations are appended with the version in which they entered. See the changelog.