# Submission tools Check and score a method before you open a pull request. Everything here runs offline and needs only Python with pymatgen (see `requirements.txt`) or the container. ## 1. Layout ``` predictions///rank01_*.cif ... rank20_*.cif # 20 ranked candidates per T180 target metadata.yaml # see example/metadata.yaml ``` Candidates are taken in rank order (the `rankNN_` prefix, otherwise natural file-name order); only the first 20 count. A candidate must parse, be ordered, have a finite positive volume, no atoms closer than 0.5 Å, and **exactly the element counts of the query's primitive-cell formula**. A candidate that fails is discarded but keeps its rank slot, so it counts as a failure. Targets without a valid candidate count as failures. ## 2. Validate ```bash python validate_submission.py predictions/my-method --testset-dir data/testsets/v3.1/T180 --metadata metadata.yaml --report validation.json ``` Exit status 0 = valid, 1 = errors, 2 = usage problem. `--strict` makes warnings fail too. ## 3. Score (the frozen accuracy evaluator) ```bash python score_submission.py predictions/my-method --testset-dir data/testsets/v3.1/T180 --metadata metadata.yaml --out out/ ``` Writes `scores.json`, `per_target.csv` and `validation.json`. Metrics: top-1/5/20 StructureMatcher recovery (`ltol` 0.2, `stol` 0.3, `angle_tol` 5°, primitive cell, scaled, no supercell), top-1 space group (spglib, `symprec` 0.1, also 0.01), consensus, and 95% bootstrap intervals over targets (10,000 resamples, seed 2026). Run on TCSPv3's native (unrelaxed, chemically ranked) predictions it reproduces their published scores exactly (122/154/163, space group 125, consensus 117 of 180); the leaderboard lists TCSPv3 relaxed and ranked with ORB-v3 (130/159/163). ## 3b. Re-rank with ORB-v3 (optional) Generative models are ranked on the leaderboard by relaxed ORB-v3 energy. If your method outputs an unranked pool, reproduce that step: ```bash python rerank_orb.py pool/ --out ranked/ --weights-path orb-v3-conservative-inf-omat-20250404.ckpt ``` `pool//*.cif` holds any number of raw candidates; each is relaxed with ORB-v3 (`orb_v3_conservative_inf_omat`, FIRE + ExpCellFilter, fmax 0.01 eV/Å, 500 steps) and the top 20 by relaxed energy per atom are written as `ranked//rankNN_*.cif` (plus `ranking.json`). On two CrystalFlow pools it reproduced the campaign's energies to 3·10⁻⁶ eV/atom (candidates within that tolerance may swap rank). Methods that rank by their own score, like TCSPv3, skip this step. Needs `requirements-ml.txt` and the ORB-v3 weights from Orbital Materials. ## 3c. Failure diagnosis (F0–F9) ```bash python diagnose_submission.py predictions/my-method --testset-dir data/testsets/v3.1/T180 --out out/ --mattersim-weights mattersim-v1.0.0-5M.pth [--raw-dir raw_top1/] ``` Runs the cascade of the Failure-diagnosis benchmark on the **raw top-1 candidate**: as generated (F0), symmetrized (F1), rescaled (F2), symmetry-constrained relax (F3a), free relax (F3b), relaxes below the reference (F8), then the symmetry, topology and coordination ladder (F4–F7, F9), with the frozen MatterSim v1.0.0-5M. The leaderboard diagnoses the generator output *before* relaxation, so if your pipeline relaxes its outputs, pass the unrelaxed rank-1 structures with `--raw-dir raw_top1//*.cif`; otherwise the submitted rank-1 candidate is diagnosed. Writes `diagnosis.json` (bucket counts, raw top-1, axis medians) and `diagnosis_per_target.csv`. On 24 TCSPv3 targets it assigned the same bucket as the paper's cascade for all 24, under both the leaderboard matcher and the looser frozen matcher. It takes a few minutes per ten targets on a laptop CPU (`--workers`). Needs `requirements-ml.txt` and the MatterSim weights. ## 4. Container (all steps) ```bash docker build -t csp-submission -f Dockerfile . # full image; --build-arg WITH_ML=0 for validation and scoring only docker run --rm -v $PWD/testset:/testset:ro -v $PWD/predictions/my-method:/submission:ro -v $PWD/metadata.yaml:/metadata.yaml:ro \ -v $PWD/weights:/weights:ro -v $PWD/out:/out csp-submission all # validate | score | diagnose | rerank | all # HPC: apptainer build csp-submission.sif Apptainer.def # apptainer run --bind $PWD/testset:/testset:ro,$PWD/predictions/my-method:/submission:ro,$PWD/out:/out csp-submission.sif score ``` `/testset` is the T180 folder of the release (`queries.json` and `cifs/`); `/weights` holds `mattersim-v1.0.0-5M.pth` and `orb-v3-conservative-inf-omat-20250404.ckpt`. For `rerank` also mount the raw pool at `/pool`. The images are written but have not been built or run yet. ## 5. Test ```bash CSP_TESTSET_DIR=/path/to/data/testsets/v3.1/T180 python -m unittest tests.test_submission_tools -v ``` ## 6. Add a method to the leaderboard (maintainers) ```bash python scripts/add_submission.py predictions/my-method/T180 --metadata metadata.yaml --set T180 [--diagnosis out/diagnosis.json] ``` Run from the site folder, once per test set (T46, T180, T240, T800). It validates and scores the predictions with the frozen evaluator, writes `submissions//.csv` and `submissions//meta.json`, and rebuilds `data/` and the pages (`--no-build` to skip). `--diagnosis` adds the raw top-1 and F0–F9 columns from `diagnose_submission.py` (T180 and T46). Submitted methods are tested against TCSPv3 with the same exact McNemar test, but their p-values are not Holm-corrected, so the published values of the campaign methods do not change. Delete `submissions//` and rebuild to remove a method. The test sets are read from the GitHub release (`../csp-leaderboard-github/testsets`), which carries the corrected mp-557387 query (`Sr4Tm2Nb2O12`, 1 November 2026).