Contribute a method

Add your CSP method to the leaderboard. At first we run the evaluator ourselves, so a submission is a pull request with your predictions and a metadata file.

1. Generate predictions

• For each query in data/testsets/v3.1/T180/queries.json give the method the primitive-cell formula (integer counts, e.g. Co2Te2) and nothing else: no lattice, coordinates, space group or Wyckoff information.
• Write 20 candidates per query as CIFs, ranked (rank 1 first), in predictions/<method>/<material-id>/rankNN_*.cif. State whether the ranking is a native score or an energy.
• Do not use any test target (or an equivalent structure) in training. Template methods must exclude same-composition templates of the target.

2. Describe the method

id: my-method
name: MyMethod
family: Diffusion / score-based        # or Flow / ODE, VAE + Langevin, Token sequence models, Symmetry-first (Wyckoff), Template substitution
paper_url: https://...
code_url: https://...
org: Your group, Your institution
checkpoint: name or hash of the checkpoint
training_data: MP-20 (train split) | MPTS-52 | other (describe)
mp20_trained: true
params: 12.3M
sampling: steps, batch size, seed, any tuning
ranker: native score | ORB-v3 energy | none
relaxed: none | ORB-v3 | other
compute: hardware and wall time per query

3. Check, score and submit

The submission tools (submission/ in the site repository) check your files and compute the leaderboard metrics offline.

# validate: layout, 20 ranked candidates, parsable ordered CIFs, exact element counts, metadata
python submission/validate_submission.py predictions/my-method --testset-dir data/testsets/v3.1/T180 --metadata metadata.yaml
# score with the frozen accuracy evaluator: top-1/5/20, space group, consensus, bootstrap intervals
python submission/score_submission.py predictions/my-method --testset-dir data/testsets/v3.1/T180 --metadata metadata.yaml --out out/
# optional: rank a raw pool with ORB-v3 (relax, then sort by energy per atom), as for the generative models on the leaderboard
python submission/rerank_orb.py pool/ --out predictions/my-method --weights-path orb-v3-conservative-inf-omat-20250404.ckpt
# failure diagnosis F0-F9 of the raw top-1 candidates, with the frozen MatterSim v1.0.0-5M
python submission/diagnose_submission.py predictions/my-method --testset-dir data/testsets/v3.1/T180 --out out/ --mattersim-weights mattersim-v1.0.0-5M.pth
# or in a container (Docker, or Apptainer for HPC; commands validate | score | diagnose | rerank | all)
git checkout -b add-my-method && git add . && git commit -m "Add MyMethod" && git push

On the native (unrelaxed, chemically ranked) TCSPv3 predictions the scorer reproduces the original TCSPv3 numbers exactly (122/154/163 of 180, space group 125, consensus 117; the leaderboard now lists the ORB-v3 relaxed and ranked version, 130/159/163), the diagnosis assigns the same F0–F9 bucket as the paper for the 24 targets checked, and the ORB-v3 re-ranking reproduces the campaign energies to 3·10^-6 eV/atom. The MatterSim and ORB-v3 weights are not bundled: download them from their providers. The diagnosis looks at the raw (unrelaxed) top-1 candidate; if your pipeline relaxes its outputs, pass the raw structures with --raw-dir.

Open a pull request. We re-run the frozen evaluator (StructureMatcher, the MatterSim-based F0–F9 cascade, ORB-v3 ranking where applicable), add your method to Community, and publish the per-target predictions and scores.

Rules

• One run per method is reported; extra seeds are welcome as additional entries labelled by seed.
• Candidates that fail validation (wrong element counts or cell size for the query) are discarded and count as failures.
• Training on the test targets or on structures that match them voids the entry; flagged entries are marked.
• Entries show their training corpus and a leakage badge; scores of models trained on larger corpora are not directly comparable.