Accuracy benchmark

T180: Cell size: 176 of the 180 targets have at most 20 atoms in the primitive cell, the size range of the MP-20 training data of the deep generative models; four have 22–28 atoms (K₆Na₂Fe₂Cl₁₂, Zn₄S₄O₁₆, Mg₁₈Au₆, Ge₁₂N₁₆). MaskGXT cannot generate cells of more than 20 atoms, so these four count as failures for it.

T46: Cell size: 45 of the 46 targets have at most 20 atoms in the primitive cell, the size range of the MP-20 training data of the deep generative models; one, Mg₁₈Au₆ (24 atoms, also in T180), is larger. MaskGXT cannot generate cells of more than 20 atoms, so this target counts as a failure for it.

T240: the 180 T180 targets (results from the T180 campaign) plus 60 additional quinary targets, run for the 16 campaign methods with the same protocol, resources and matcher. MaskGXT was run on the 60 additional targets afterwards, with its T180 protocol (20 of 30 native candidates, ORB-v3 single-point ranking, no relaxation); the 29 targets above its 20-atom limit count as failures, so it covers 211 of the 240 targets. Cell size: 211 of the 240 targets (88%) have at most 20 atoms in the primitive cell, the size range of the MP-20 training data of the deep generative models. The other 29 (4 among the T180 targets and 25 among the additional ones, up to 28 atoms) are included deliberately as a test of extrapolation beyond that training range. The T46-only search methods were not run. Space group (top-1 candidate, symprec 0.1 and 0.01) and consensus use the same definitions as for T180. Missing candidates count as failures; several methods return no candidate for many of the additional targets (TCSPv3 covers 39 of 60, uFlowCSP 13, CrystalFormer 9). p-values are paired exact McNemar tests against TCSPv3 at top-1, Holm-corrected over the 16 comparisons; intervals are 95% bootstrap (10,000 resamples, seed 2026).

T800: 800 targets from the MPTS-52 test split (1–6 elements, up to 52 sites), with no target, composition or structural overlap with T46/T180. TCSPv3 uses the full configuration of the T800 feature study in the TCSP3.0 paper, with its candidates ORB-v3 relaxed and ranked by ORB-v3 energy like the other methods. The deep generative models were run with the same protocol as on T180; failed generations count as misses. CrystalFormer was not run on T800. Cell size: 686 of the 800 targets (86%) have more than 20 atoms in the primitive cell (293 with 21–30, 189 with 31–40 and 204 with 41–52 atoms); only 114 have 20 or fewer. The deep generative models were trained on MP-20, whose cells have at most 20 atoms, so most T800 targets lie outside their training range, which explains their low scores here. MaskGXT cannot generate cells of more than 20 atoms, so its 683 larger targets count as misses (the other three are flagged targets withheld from every method). Each query’s own composition and structurally equivalent aliases are excluded from its templates; other T800 compounds remain eligible. The By-arity view shows binary, ternary and quaternary targets only (4 unary, 93 five-element and 14 six-element targets count in the totals). Intervals are 95% bootstrap (10,000 resamples, seed 2026).

How structures are compared. Structure matching uses the pymatgen StructureMatcher with ltol = 0.2, stol = 0.3, angle_tol = 5°, primitive_cell = True, scale = True and attempt_supercell = False, so a candidate must have the same number of atoms in the primitive cell as the reference (the reference is reduced to its primitive cell at 0.01 Å). Top-k counts a target if one of the first k ranked candidates matches. Space group is determined with spglib through pymatgen SpacegroupAnalyzer at symprec = 0.1 Å and compared by number with the reference's space group at the same tolerance, for the top-1 candidate only (the accuracy page also shows symprec = 0.01 Å). Consensus requires the top-1 candidate to pass both tests. EeqTop1% (energy-equivalent top-1 rate) is the share of all targets whose top-1 candidate lies within 10 meV/atom of the reference, |ΔE| ≤ 10 meV/atom, with both structures relaxed with ORB-v3 using the same settings; missing predictions count as failures. The F0–F9 diagnosis additionally reports a looser matcher (ltol 0.3, stol 0.5, angle_tol 10°) as a sensitivity check. Targets for which a method returned no valid candidate count as failures (MaskGXT: 4 targets above its 20-atom limit; Uni-3DAR and uFlowCSP: 1 target each; CrystalGRW: 2; CrystalFormer: 96). One run per method and target. Intervals are percentile bootstrap intervals over targets (percentage points).