Submit a representation

Track O: any representation

1. Get the gene universe. Clone the research repository and locate genes_v1.tsv. It lists the 19,256 genes in a fixed order with HGNC symbol, Entrez and Ensembl identifiers, exposure bin, family and fold.

   git clone https://github.com/hwang595/gene-embedding-exposure.git
   cd gene-embedding-exposure
   head -3 round4/release/benchmark_v1/genes_v1.tsv

2. Embed every gene you can. Produce a float32 matrix of shape (19,256, d), one row per gene in genes_v1.tsv order, saved as a NumPy .npy file. If some genes cannot be embedded, fill their rows with zeros and supply a boolean mask of shape (19,256,) that is true where the row is valid; the harness substitutes the mean embedding for missing genes and reports coverage per exposure bin. Mixtures concatenate standardised components (the released mixture rows standardise each component over the universe before concatenation). Do not tune anything on the task labels: the folds are fixed and the probes are the only trained components.
3. Audit your text. If the representation embeds any gene text, run the template detector over it and report the share of genes whose text contains a generated summary, overall and in the 0–10 stratum. The detector reads NCBI's provenance tag when present and otherwise recognises the Alliance sentence templates; what it cannot see (paraphrased generated text, other pipelines) is your statement, not a verified field.

   python3 - <<'PY'
   import sys; sys.path.insert(0, "round4/scripts")
   from gene_text_audit import is_generated, strip_generated
   texts = {...}                       # your gene -> text mapping
   flags = {g: is_generated(t) for g, t in texts.items()}
   print(sum(flags.values()), "of", len(flags), "texts contain a generated summary")
   filtered = {g: strip_generated(t) for g, t in texts.items()}   # summary-filtered variant, if you want one
   PY

4. Write the declaration. Copy the template below, fill every field, and keep it with the embedding. The fields are what the leaderboard shows in the row detail; nothing else is inferred.

   representation: display name
   tag: short identifier, lowercase, no spaces
   modality: protein | rna | dna | single-cell | text | text-aligned protein | network | mixture
   embedding:
     url: where the .npy (and mask) can be downloaded
     shape: [19256, d]
     coverage: number of genes with a valid row
     standardised_components: for mixtures, list the components and how they were scaled
   text_embedded:
     any: yes | no
     source_and_date: e.g. NCBI Gene summaries, snapshot 2026-10-06; UniProt function text 2026_04
     generated_summary_share: share of embedded genes with a GO-generated summary (detector or tag), all genes and 0-10 stratum
     summary_filtered: yes | no (generated summaries removed before embedding?)
     curated_text_from_label_pipeline: e.g. UniProt function text included | excluded
   text_in_training:
     any: yes | no (e.g. UniProt function text in a text-aligned protein model)
     source_and_date: ...
   encoder: model and version used to embed text, if any
   training_data_and_date: as stated by the model's authors
   t3_eligibility:
     claim: eligible | not eligible
     reason: no input can contain functional annotation or text newer than 2023-01-01 because ...
   code_or_weights: URL
   licence: of the embedding file
   contact: name and e-mail

   Copy template
5. Open a submission issue. Use the GeneProv submission form in the repository's issue tracker and attach or link the embedding, the mask and the declaration. The maintainers run the frozen harness (linear and MLP probes on T1, T2 and T4; T3 when eligible), reply with the task profile and bootstrap intervals, and append the row to the next version of the table and to this site. Rows with an undeclared text source, or with text that fails the audit, are listed with that flag rather than rejected.
   Open the form (repository access required)Browse issues (repository access required)

Track F: a text embedder

1. Build the two corpora. Track F fixes the text, so the only input is the embedder. Download GenePT's text release (Zenodo record 10833191, NCBI_summary_of_genes.json) and run the corpus builder; it writes corpus A (as released) and corpus L (every generated summary removed, leaving Gene Symbol X).

   python3 round4/release/benchmark_v1/build_corpus.py /path/to/NCBI_summary_of_genes.json round4/release/benchmark_v1/genes_v1.tsv
   # Ambiguous historical aliases require review; see the builder help.
   # Use --allow-ambiguous-keys only to reproduce the historical mapping.
   # -> corpus_A.tsv, corpus_L.tsv (19,256 rows each, columns symbol, text)

2. Embed both corpora. One float32 .npy per corpus, rows in genes_v1.tsv order. Report the embedder name, version, pooling and any instruction prompt; do not fine-tune on the corpora or on the labels.
3. Submit. Open the submission form with both files. The row is scored on corpus L (index, T1, T2, T4) and receives S, the paired per-gene GO AUROC difference between corpus A and corpus L for genes with 1–10 publications. Track F rows have no rank intervals.

Aggregate prepared predictions locally

Install the package dependencies and prepare the keyed NPZ described in PREDICTION_SCHEMA.md. The schema requires the released gene order and fold identifiers, per-gene T1/T4 AUROCs and per-label T2 out-of-fold scores. The scorer validates these declarations, but cannot prove that fitting was out of fold. Declare whether the supplied AUROCs use the historical or corrected average-tie definition; corrected scores require a separate version rather than replacing historical cells.

python3 -m pip install -r round4/release/benchmark_v1/requirements.txt
python3 round4/release/benchmark_v1/score_predictions.py predictions.npz \
  --metric-definition average-ties --output scores.json

T3 is optional and requires --t3-eligible plus a separate input-provenance declaration. This flag cannot verify historical dates. The result includes task denominators and hashes of the predictions and release files; it does not supply index or rank intervals.

Review and versioning

• Frozen rows never change. The v1 table is the record evaluated in the paper. New rows are appended with their version label (v1.1 rows entered in round 14); the chance-corrected transform does not depend on other rows, so appending never moves an existing score.
• Reproduction. Each round that regenerates arrays runs a reproduction gate against the frozen cells and documents every exception. The round-14 gate reproduced 350 of 366 cells within 0.002 AUROC; the sixteen exceptions comprise twelve ESM2-650M cells from re-embedding under a different library version and four MLP T2 cells from regenerated text rows. They are listed rather than substituted into the frozen record.
• Provenance flags are displayed, not enforced. A representation with generated summaries in its text is listed with its share, as the GenePT releases are. A representation whose declaration is incomplete is listed as such.
• Compute. Runtime depends on the embedding size, hardware and tasks. The repository records the probe settings; agree on compute availability with the maintainers through the submission issue.

Questions

Which genes?

The 19,256 HGNC protein-coding genes in genes_v1.tsv; use the Entrez or Ensembl identifier to align, not the symbol alone.

What if coverage is low?

Missing genes receive the mean embedding and the coverage column says so; the row detail lists the missing genes by exposure bin. Report coverage for each exposure stratum, because universe-wide coverage can hide missingness among less-studied genes.

Can I submit an updated version of a model?

Yes, as a new row with its own tag and date; the earlier row stays.

Can I evaluate locally?

Prepared predictions can be aggregated locally using only the released package; see the command above. End-to-end evaluation from a new embedding still requires rebuilding the research pipeline. The driver round4/scripts/r7_10_eval.py records its probe protocol, and geneprov_index.py computes the index from three AUROCs.