subinium/Awesome-Scientific-LLM-Benchmarks

# Awesome Scientific LLM Benchmarks [](https://awesome.re) [](LICENSE) _Listed alphabetically within each domain._ ## Contents - [General / Multi-domain Science](#general--multi-domain-science) - [Mathematics](#mathematics) - [Physics & Astronomy](#physics--astronomy) - [Chemistry](#chemistry) - [Materials Science](#materials-science) - [Biology & Life Sciences](#biology--life-sciences) - [Agentic Science & AI Research](#agentic-science--ai-research) ### General / Multi-domain Science | Benchmark | Org | Year | Paper | Code | Stars | Description | |---|---|:--:|:--:|:--:|:--:|---| | **AGIEval** | Microsoft | 2023 | [paper](https://arxiv.org/abs/2304.06364) | [code](https://github.com/ruixiangcui/AGIEval) | [](https://github.com/ruixiangcui/AGIEval) | Human-centric benchmark from standardized exams like Gaokao, SAT, LSAT, and math competitions. | | **ARB** | DuckAI / Georgia Tech | 2023 | [paper](https://arxiv.org/abs/2307.13692) | [code](https://github.com/TheDuckAI/arb) | [](https://github.com/TheDuckAI/arb) | Advanced problems across mathematics, physics, biology, chemistry, and law, including symbolic and proof items graded by rubric. | | **ARC (AI2 Reasoning Challenge)** | Allen AI (AI2) | 2018 | [paper](https://arxiv.org/abs/1803.05457) | [code](https://github.com/allenai/ARC-Solvers) | [](https://github.com/allenai/ARC-Solvers) | 7,787 grade-school science exam multiple-choice questions split into Easy and Challenge sets. | | **C-Eval** | SJTU / HKUST | 2023 | [paper](https://arxiv.org/abs/2305.08322) | [code](https://github.com/hkust-nlp/ceval) | [](https://github.com/hkust-nlp/ceval) | 13,948 Chinese multiple-choice questions across 52 disciplines and four difficulty levels. | | **EMMA** | CUHK / Microsoft | 2025 | [paper](https://arxiv.org/abs/2501.05444) | [code](https://github.com/EMMA-Bench/EMMA) | [](https://github.com/EMMA-Bench/EMMA) | 2,788 problems demanding cross-modal reasoning across mathematics, physics, chemistry, and coding for multimodal models. | | **FrontierScience** | OpenAI | 2026 | [paper](https://arxiv.org/abs/2601.21165) | — | — | Expert-level science benchmark with an olympiad track (IPhO/IChO/IBO-level) and a rubric-graded PhD-level research track across physics, chemistry, and biology. | | **GAOKAO-Bench** | Fudan University | 2023 | [paper](https://arxiv.org/abs/2305.12474) | [code](https://github.com/OpenLMLab/GAOKAO-Bench) | [](https://github.com/OpenLMLab/GAOKAO-Bench) | 2,811 Chinese college-entrance-exam questions across subjects for LLM evaluation. | | **GPQA** | NYU / Anthropic / Cohere | 2023 | [paper](https://arxiv.org/abs/2311.12022) | [code](https://github.com/idavidrein/gpqa) | [](https://github.com/idavidrein/gpqa) | 448 expert-written graduate-level biology, physics, and chemistry Google-proof multiple-choice questions. | | **Humanity's Last Exam** | CAIS / Scale AI | 2025 | [paper](https://arxiv.org/abs/2501.14249) | [code](https://github.com/centerforaisafety/hle) | [](https://github.com/centerforaisafety/hle) | 2,500 expert questions across 100+ subjects at the frontier of human academic knowledge. | | **JEEBench** | IIT Delhi | 2023 | [paper](https://arxiv.org/abs/2305.15074) | [code](https://github.com/dair-iitd/jeebench) | [](https://github.com/dair-iitd/jeebench) | 515 challenging IIT-JEE Advanced physics, chemistry, and math problem-solving questions. | | **MMLU-Pro** | University of Waterloo (TIGER-Lab) | 2024 | [paper](https://arxiv.org/abs/2406.01574) | [code](https://github.com/TIGER-AI-Lab/MMLU-Pro) | [](https://github.com/TIGER-AI-Lab/MMLU-Pro) | 12K reasoning-focused ten-option questions across 14 academic and STEM domains. | | **MMMU** | IN.AI / University of Waterloo / OSU | 2023 | [paper](https://arxiv.org/abs/2311.16502) | [code](https://github.com/MMMU-Benchmark/MMMU) | [](https://github.com/MMMU-Benchmark/MMMU) | 11.5K college-level multimodal questions across six disciplines and 30 subjects. | | **OlympiadBench** | Tsinghua University / OpenBMB | 2024 | [paper](https://arxiv.org/abs/2402.14008) | [code](https://github.com/OpenBMB/OlympiadBench) | [](https://github.com/OpenBMB/OlympiadBench) | 8,476 olympiad-level bilingual multimodal math and physics problems with expert step annotations. | | **OlympicArena** | SJTU (GAIR) | 2024 | [paper](https://arxiv.org/abs/2406.12753) | [code](https://github.com/GAIR-NLP/OlympicArena) | [](https://github.com/GAIR-NLP/OlympicArena) | 11,163 olympiad-level problems across seven disciplines for multi-discipline cognitive reasoning. | | **OpenBookQA** | Allen AI (AI2) | 2018 | [paper](https://arxiv.org/abs/1809.02789) | [code](https://github.com/allenai/OpenBookQA) | [](https://github.com/allenai/OpenBookQA) | Elementary science multiple-choice questions requiring core facts plus broad common-sense knowledge. | | **PaperMind** | University of Illinois Urbana-Champaign | 2026 | [paper](https://arxiv.org/abs/2604.21304) | [code](https://github.com/Yanjun-Zhao/PaperMind) | [](https://github.com/Yanjun-Zhao/PaperMind) | Multimodal benchmark evaluating agent-oriented reasoning and critique over real research papers across seven domains via grounding, experimental interpretation, cross-source evidence, and critical-assessment tasks. | | **QASC** | Allen AI (AI2) | 2020 | [paper](https://arxiv.org/abs/1910.11473) | [code](https://github.com/allenai/qasc) | [](https://github.com/allenai/qasc) | 9,980 grade-school science questions requiring retrieval and composition of two facts. | | **SciAssess** | DP Technology (deepmodeling) | 2024 | [paper](https://arxiv.org/abs/2403.01976) | [code](https://github.com/deepmodeling/SciAssess) | [](https://github.com/deepmodeling/SciAssess) | Scientific literature analysis over biology, chemistry, materials, and medicine at three levels. | | **SciBench** | UCLA | 2023 | [paper](https://arxiv.org/abs/2307.10635) | [code](https://github.com/mandyyyyii/scibench) | [](https://github.com/mandyyyyii/scibench) | 692 open-ended college-level chemistry, physics, and math problems requiring multi-step reasoning. | | **ScienceQA** | UCLA / Allen AI / ASU | 2022 | [paper](https://arxiv.org/abs/2209.09513) | [code](https://github.com/lupantech/ScienceQA) | [](https://github.com/lupantech/ScienceQA) | ~21K multimodal multiple-choice science questions with lecture and chain-of-thought explanations. | | **SciEval** | Fudan University (OpenDFM) | 2023 | [paper](https://arxiv.org/abs/2308.13149) | [code](https://github.com/OpenDFM/SciEval) | [](https://github.com/OpenDFM/SciEval) | ~18K multi-level questions testing scientific knowledge across chemistry, physics, and biology. | | **SciFact** | Allen AI (AI2) | 2020 | [paper](https://arxiv.org/abs/2004.14974) | [code](https://github.com/allenai/scifact) | [](https://github.com/allenai/scifact) | 1,409 expert-written scientific claims verified against research abstracts, with rationales. | | **SciKnowEval** | Zhejiang University (HICAI) | 2024 | [paper](https://arxiv.org/abs/2406.09098) | [code](https://github.com/HICAI-ZJU/SciKnowEval) | [](https://github.com/HICAI-ZJU/SciKnowEval) | Tens of thousands of problems across five cognitive levels in biology, chemistry, physics, and materials. | | **SciQ** | Allen AI (AI2) | 2017 | [paper](https://arxiv.org/abs/1707.06209) | — | — | 13,679 crowdsourced science multiple-choice questions across physics, chemistry, and biology with evidence. | | **SDE** | Cornell / Princeton / Stanford / MIT / Toronto | 2025 | [paper](https://arxiv.org/abs/2512.15567) | [code](https://github.com/deepprinciple/lm-evaluation-harness) | [](https://github.com/deepprinciple/lm-evaluation-harness) | Scenario-grounded scientific-discovery benchmark of 43 scenarios and 1,125 questions across biology, chemistry, materials, and physics, plus 8 project-level hypothesis, experiment-design, and interpretation tasks. | | **SuperGPQA** | ByteDance Seed / M-A-P | 2025 | [paper](https://arxiv.org/abs/2502.14739) | [code](https://github.com/SuperGPQA/SuperGPQA) | [](https://github.com/SuperGPQA/SuperGPQA) | 26,529 graduate-level questions spanning 285 disciplines, including under-evaluated long-tail fields. | | **TheoremQA** | University of Waterloo (TIGER-Lab) | 2023 | [paper](https://arxiv.org/abs/2305.12524) | [code](https://github.com/TIGER-AI-Lab/TheoremQA) | [](https://github.com/TIGER-AI-Lab/TheoremQA) | 800 questions applying 350+ theorems across math, physics, EE/CS, and finance. | | **Xiezhi** | Fudan University | 2023 | [paper](https://arxiv.org/abs/2306.05783) | [code](https://github.com/MikeGu721/XiezhiBenchmark) | [](https://github.com/MikeGu721/XiezhiBenchmark) | 249,587 questions spanning 516 disciplines across 13 categories, continuously updated. | [Back to top](#contents) ### Mathematics | Benchmark | Org | Year | Paper | Code | Stars | Description | |---|---|:--:|:--:|:--:|:--:|---| | **CHAMP** | MIT | 2024 | [paper](https://arxiv.org/abs/2401.06961) | [code](https://github.com/YilunZhou/champ-dataset) | [](https://github.com/YilunZhou/champ-dataset) | 270 high-school competition problems annotated with concepts and problem-specific hints. | | **CombiBench** | Moonshot AI / Numina | 2025 | [paper](https://arxiv.org/abs/2505.03171) | [code](https://github.com/MoonshotAI/CombiBench) | [](https://github.com/MoonshotAI/CombiBench) | 100 competition combinatorics problems formalized in Lean4, a domain underrepresented by existing formal benchmarks. | | **FIMO** | Peking University / Huawei | 2023 | [paper](https://arxiv.org/abs/2309.04295) | [code](https://github.com/liuchengwucn/FIMO) | [](https://github.com/liuchengwucn/FIMO) | 149 IMO-shortlist problems formalized in Lean with informal statements for olympiad-level theorem proving. | | **FormalMATH** | SphereLab / M-A-P | 2025 | [paper](https://arxiv.org/abs/2505.02735) | [code](https://github.com/Sphere-AI-Lab/FormalMATH-Bench) | [](https://github.com/Sphere-AI-Lab/FormalMATH-Bench) | 5,560 formally verified Lean4 statements from olympiad to undergraduate across algebra, calculus, and number theory. | | **FrontierMath** | Epoch AI | 2024 | [paper](https://arxiv.org/abs/2411.04872) | — | — | Hundreds of unpublished expert-crafted research-level math problems resistant to guessing. | | **GSM8K** | OpenAI | 2021 | [paper](https://arxiv.org/abs/2110.14168) | [code](https://github.com/openai/grade-school-math) | [](https://github.com/openai/grade-school-math) | 8.5K grade-school arithmetic word problems requiring multi-step reasoning. | | **HARDMath** | Harvard | 2024 | [paper](https://arxiv.org/abs/2410.09988) | [code](https://github.com/sarahmart/HARDMath) | [](https://github.com/sarahmart/HARDMath) | Graduate applied-math problems requiring asymptotic and approximation methods where leading LLMs score below 45%. | | **HARP** | UCL | 2024 | [paper](https://arxiv.org/abs/2412.08819) | [code](https://github.com/aadityasingh/HARP) | [](https://github.com/aadityasingh/HARP) | US competition math problems (AMC/AIME/USAMO) with human-written ground-truth solutions. | | **IMO-Bench** | Google DeepMind | 2025 | [paper](https://arxiv.org/abs/2511.01846) | [code](https://github.com/google-deepmind/superhuman) | [](https://github.com/google-deepmind/superhuman) | Olympiad-level suite testing final answers, proof-writing, grading, and Lean formal proofs, vetted by IMO medalists. | | **Lean Workbook** | Shanghai AI Lab | 2024 | [paper](https://arxiv.org/abs/2406.03847) | — | — | About 57K formal-informal Lean4 problem pairs auto-formalized from competition math. | | **LeanDojo** | Caltech / NVIDIA | 2023 | [paper](https://arxiv.org/abs/2306.15626) | [code](https://github.com/lean-dojo/LeanDojo) | [](https://github.com/lean-dojo/LeanDojo) | 98,734 theorems and proofs from Lean mathlib with premise annotations for retrieval-augmented theorem proving. | | **MATH** | UC Berkeley | 2021 | [paper](https://arxiv.org/abs/2103.03874) | [code](https://github.com/hendrycks/math) | [](https://github.com/hendrycks/math) | 12,500 competition mathematics problems (AMC/AIME level) with step-by-step solutions. | | **MATH-500** | OpenAI | 2023 | [paper](https://arxiv.org/abs/2305.20050) | [code](https://github.com/openai/prm800k) | [](https://github.com/openai/prm800k) | 500-problem held-out subset of MATH widely used for LLM evaluation. | | **MathBench** | Shanghai AI Laboratory | 2024 | [paper](https://arxiv.org/abs/2405.12209) | [code](https://github.com/open-compass/MathBench) | [](https://github.com/open-compass/MathBench) | Hierarchical bilingual benchmark spanning arithmetic to college-level theory and application. | | **MATHCHECK** | XJTLU / HKUST | 2024 | [paper](https://arxiv.org/abs/2407.08733) | [code](https://github.com/PremiLab-Math/MathCheck) | [](https://github.com/PremiLab-Math/MathCheck) | Checklist benchmark testing task generalization and reasoning robustness beyond end-to-end answer accuracy. | | **MathOdyssey** | NetMind.AI | 2024 | [paper](https://arxiv.org/abs/2406.18321) | [code](https://github.com/protagolabs/odyssey-math) | [](https://github.com/protagolabs/odyssey-math) | 387 expert-crafted high-school, university, and olympiad-level math problems. | | **MathQA** | UW / Allen AI | 2019 | [paper](https://arxiv.org/abs/1905.13319) | [code](https://github.com/math-QA/math-QA) | [](https://github.com/math-QA/math-QA) | 37K multiple-choice math word problems annotated with executable operation programs. | | **MathVerse** | CUHK MMLab / Shanghai AI Lab | 2024 | [paper](https://arxiv.org/abs/2403.14624) | [code](https://github.com/ZrrSkywalker/MathVerse) | [](https://github.com/ZrrSkywalker/MathVerse) | 2,612 visual math problems in six diagram/text variants probing whether MLLMs truly interpret figures. | | **MathVista** | UCLA / University of Washington / Microsoft | 2023 | [paper](https://arxiv.org/abs/2310.02255) | [code](https://github.com/lupantech/MathVista) | [](https://github.com/lupantech/MathVista) | Mathematical reasoning benchmark combining visual contexts (figures, charts, diagrams) with problems. | | **miniF2F** | OpenAI | 2021 | [paper](https://arxiv.org/abs/2109.00110) | [code](https://github.com/openai/miniF2F) | [](https://github.com/openai/miniF2F) | 488 olympiad-level (AMC/AIME/IMO) problems formalized across multiple proof assistants. | | **Omni-MATH** | Peking University | 2024 | [paper](https://arxiv.org/abs/2410.07985) | [code](https://github.com/KbsdJames/Omni-MATH) | [](https://github.com/KbsdJames/Omni-MATH) | 4,428 olympiad-level competition problems across 33+ subdomains and difficulty tiers. | | **ProofNet** | Yale University | 2023 | [paper](https://arxiv.org/abs/2302.12433) | [code](https://github.com/zhangir-azerbayev/ProofNet) | [](https://github.com/zhangir-azerbayev/ProofNet) | 371 undergraduate theorems for autoformalization and formal proving in Lean. | | **Putnam-AXIOM** | Stanford | 2025 | [paper](https://arxiv.org/abs/2508.08292) | — | — | Putnam competition problems plus programmatically perturbed variations giving contamination-resistant unseen instances. | | **PutnamBench** | UT Austin | 2024 | [paper](https://arxiv.org/abs/2407.11214) | [code](https://github.com/trishullab/PutnamBench) | [](https://github.com/trishullab/PutnamBench) | 1,600+ Putnam competition problems formalized in Lean, Isabelle, and Coq. | | **We-Math** | BUPT / Tencent | 2024 | [paper](https://arxiv.org/abs/2407.01284) | [code](https://github.com/We-Math/We-Math) | [](https://github.com/We-Math/We-Math) | 6.5K visual math problems decomposed into 67 hierarchical knowledge concepts to diagnose reasoning versus memorization. | [Back to top](#contents) ### Physics & Astronomy | Benchmark | Org | Year | Paper | Code | Stars | Description | |---|---|:--:|:--:|:--:|:--:|---| | **ABench-Physics** | Zhejiang University / Ant Group | 2025 | [paper](https://arxiv.org/abs/2507.04766) | — | — | 500 hard static plus dynamic-variant physics problems testing reasoning and generalization robustness. | | **Astro-QA** | ACMIS Lab | 2025 | [paper](https://www.nature.com/articles/s41597-025-04613-9) | [code](https://github.com/ACMISLab/Astro-QA) | [](https://github.com/ACMISLab/Astro-QA) | About 2,700 bilingual astronomy questions across six types spanning astrophysics, celestial mechanics, and astrometry. | | **AstroMLab 1** | AstroMLab Collaboration | 2024 | [paper](https://arxiv.org/abs/2407.11194) | — | — | 4,425 astronomy multiple-choice questions from Annual Reviews testing LLM astronomical knowledge. | | **AstroVisBench** | NSF-Simons CosmicAI Institute | 2025 | [paper](https://arxiv.org/abs/2505.20538) | [code](https://github.com/NSF-Simons-CosmicAI-Institute/AstroVisBench) | [](https://github.com/NSF-Simons-CosmicAI-Institute/AstroVisBench) | Evaluates LLMs on end-to-end astronomy computing workflows and scientific result visualization. | | **AtmosSci-Bench** | HKUST | 2025 | [paper](https://arxiv.org/abs/2502.01159) | [code](https://github.com/Relaxed-System-Lab/AtmosSci-Bench) | [](https://github.com/Relaxed-System-Lab/AtmosSci-Bench) | Atmospheric-science problems spanning dynamics, physics, hydrology, geophysics, and oceanography via templated questions. | | **CMPhysBench** | Chinese Academy of Sciences (IOP) | 2025 | [paper](https://arxiv.org/abs/2508.18124) | [code](https://github.com/CMPhysBench/CMPhysBench) | [](https://github.com/CMPhysBench/CMPhysBench) | 520+ graduate condensed-matter physics problems with a partial-credit metric; top models score below 30%. | | **CritPt** | Argonne National Laboratory / UIUC (50+ physicists, 30+ institutions) | 2025 | [paper](https://arxiv.org/abs/2509.26574) | [code](https://github.com/CritPt-Benchmark/CritPt) | [](https://github.com/CritPt-Benchmark/CritPt) | 71 composite research-project challenges and 190 modular checkpoints authored by 50+ active physicists, auto-graded via numerical, symbolic, and code evaluations. | | **DiscoverPhysics** | Princeton / NYU / Flatiron Institute / Polymathic AI | 2026 | [paper](https://arxiv.org/abs/2605.26087) | [code](https://github.com/SampsonML/DiscoverPhysics) | [](https://github.com/SampsonML/DiscoverPhysics) | 22 simulated worlds with non-standard physics where an agent proposes initial conditions, observes noisy N-body trajectories, and submits a natural-language law plus a Python implementation. | | **Gravity-Bench-v1** | University of Toronto | 2025 | [paper](https://arxiv.org/abs/2501.18411) | [code](https://github.com/NolanKoblischke/GravityBench) | [](https://github.com/NolanKoblischke/GravityBench) | Agents plan budgeted observations of simulated binary-star systems to discover the underlying gravitational physics. | | **HiPhO** | CUHK / Shanghai AI Lab | 2025 | [paper](https://arxiv.org/abs/2509.07894) | [code](https://github.com/SciYu/HiPhO) | [](https://github.com/SciYu/HiPhO) | 13 recent physics olympiad exams with step-level grading and human medal-based comparison for (M)LLMs. | | **LLM-SRBench** | Virginia Tech / CMU | 2025 | [paper](https://arxiv.org/abs/2504.10415) | [code](https://github.com/deep-symbolic-mathematics/llm-srbench) | [](https://github.com/deep-symbolic-mathematics/llm-srbench) | 239 problems across four science domains testing genuine LLM equation discovery over memorized formulas. | | **NewtonBench** | HKUST | 2025 | [paper](https://arxiv.org/abs/2510.07172) | [code](https://github.com/HKUST-KnowComp/NewtonBench) | [](https://github.com/HKUST-KnowComp/NewtonBench) | 324 tasks across twelve physics domains testing LLM agents discovering scientific laws via interactive experimentation. | | **PHYBench** | Peking University | 2025 | [paper](https://arxiv.org/abs/2504.16074) | [code](https://github.com/phybench-official/phybench) | [](https://github.com/phybench-official/phybench) | 500 original physics problems from high-school to olympiad level with expression-edit-distance scoring. | | **PhysGym** | KAUST | 2025 | [paper](https://arxiv.org/abs/2507.15550) | [code](https://github.com/principia-ai/PhysGym) | [](https://github.com/principia-ai/PhysGym) | 97 simulated interactive physics-discovery problems with four controlled levels of prior knowledge. | | **PHYSICS** | Yale University | 2025 | [paper](https://arxiv.org/abs/2503.21821) | [code](https://github.com/yale-nlp/Physics) | [](https://github.com/yale-nlp/Physics) | 1,297 expert-annotated university-level physics problems across six core areas with automated evaluation. | | **PhysReason** | Xi'an Jiaotong University | 2025 | [paper](https://arxiv.org/abs/2502.12054) | [code](https://github.com/dxzxy12138/PhysReason) | [](https://github.com/dxzxy12138/PhysReason) | 1,200 physics problems with step-level automatic scoring for multi-step physics reasoning. | | **PRL-Bench** | Shanghai Jiao Tong University | 2026 | [paper](https://arxiv.org/abs/2604.15411) | — | — | End-to-end physics research benchmark built from ~100 recent Physical Review Letters papers across five subfields, each turned into a long-horizon task scored by an LLM-as-judge. | | **TPBench** | University of Wisconsin-Madison | 2025 | [paper](https://arxiv.org/abs/2502.15815) | — | — | 57 theoretical physics problems in high-energy theory and cosmology, undergraduate to research level. | | **UGPhysics** | HKUST | 2025 | [paper](https://arxiv.org/abs/2502.00334) | [code](https://github.com/YangLabHKUST/UGPhysics) | [](https://github.com/YangLabHKUST/UGPhysics) | 5,520 bilingual undergraduate physics problems across 13 subjects with rule-based judging. | [Back to top](#contents) ### Chemistry | Benchmark | Org | Year | Paper | Code | Stars | Description | |---|---|:--:|:--:|:--:|:--:|---| | **ChemBench** | LAMALab, University of Jena (Jablonka group) | 2024 | [paper](https://doi.org/10.1038/s41557-025-01815-x) | [code](https://github.com/lamalab-org/chembench) | [](https://github.com/lamalab-org/chembench) | 2,700+ curated questions probing LLM chemical knowledge and reasoning against expert chemists. | | **ChemCoTBench** | IDEA Research / PKU / CUHK | 2025 | [paper](https://arxiv.org/abs/2505.21318) | [code](https://github.com/IDEA-XL/ChemCoTBench) | [](https://github.com/IDEA-XL/ChemCoTBench) | Step-wise chemical reasoning across molecular understanding, editing, optimization, and reaction prediction. | | **ChemEval** | USTC | 2024 | [paper](https://arxiv.org/abs/2409.13989) | [code](https://github.com/USTC-StarTeam/ChemEval) | [](https://github.com/USTC-StarTeam/ChemEval) | Multi-level chemistry benchmark spanning 42 tasks across four progressive difficulty levels for LLMs. | | **ChemLLMBench** | University of Notre Dame et al. | 2023 | [paper](https://arxiv.org/abs/2305.18365) | [code](https://github.com/ChemFoundationModels/ChemLLMBench) | [](https://github.com/ChemFoundationModels/ChemLLMBench) | Eight chemistry tasks including reaction prediction, retrosynthesis, and molecule captioning for LLMs. | | **ChemSafetyBench** | Peking University | 2024 | [paper](https://arxiv.org/abs/2411.16736) | — | — | Safety benchmark testing LLM handling of hazardous-chemical property, legality, and synthesis queries. | | **MaCBench** | LAMALab (Jena) / IIT Delhi | 2024 | [paper](https://arxiv.org/abs/2411.16955) | [code](https://github.com/lamalab-org/macbench) | [](https://github.com/lamalab-org/macbench) | 1,100+ image-question pairs probing multimodal LLM limitations across chemistry and materials workflows. | | **MolCap-Arena** | Genentech / UIUC | 2024 | [paper](https://arxiv.org/abs/2411.00737) | [code](https://github.com/Genentech/molcap-arena) | [](https://github.com/Genentech/molcap-arena) | Battle-based benchmark scoring 20+ LLMs on molecule captions augmenting molecular property prediction. | | **MoleculeQA** | IDEA Research | 2024 | [paper](https://arxiv.org/abs/2403.08192) | [code](https://github.com/IDEA-XL/MoleculeQA) | [](https://github.com/IDEA-XL/MoleculeQA) | 62K QA pairs over 23K molecules evaluating factual accuracy of LLM molecular comprehension. | | **MolTextQA** | UIUC | 2024 | [paper](https://openreview.net/forum?id=tFfqvKE2J5) | [code](https://github.com/siddharthal/MolTextQA) | [](https://github.com/siddharthal/MolTextQA) | 500K QA pairs over 240K PubChem molecules evaluating molecule-structure-to-text understanding and retrieval. | | **MOOSE-Chem** | NTU Singapore / Shanghai AI Lab | 2024 | [paper](https://arxiv.org/abs/2410.07076) | [code](https://github.com/ZonglinY/MOOSE-Chem) | [](https://github.com/ZonglinY/MOOSE-Chem) | Tests whether LLMs rediscover unseen chemistry hypotheses from 51 Nature/Science-level papers given background information. | | **ScholarChemQA** | KAUST / Notre Dame | 2024 | [paper](https://arxiv.org/abs/2407.16931) | [code](https://github.com/iriscxy/chemmatch) | [](https://github.com/iriscxy/chemmatch) | 40K abstract-derived chemistry research questions exposing LLM limits in comprehending scholarly literature. | | **SMolInstruct / LlaSMol** | OSU NLP Group | 2024 | [paper](https://arxiv.org/abs/2402.09391) | [code](https://github.com/OSU-NLP-Group/LLM4Chem) | [](https://github.com/OSU-NLP-Group/LLM4Chem) | Large-scale instruction dataset of 14 small-molecule chemistry tasks used to train and evaluate LlaSMol. | | **TOMG-Bench** | HK PolyU / Shanghai AI Lab | 2024 | [paper](https://arxiv.org/abs/2412.14642) | [code](https://github.com/phenixace/TOMG-Bench) | [](https://github.com/phenixace/TOMG-Bench) | Evaluates LLMs on text-based open-domain molecule generation across editing, optimization, and customized generation. | [Back to top](#contents) ### Materials Science | Benchmark | Org | Year | Paper | Code | Stars | Description | |---|---|:--:|:--:|:--:|:--:|---| | **ALDbench** | Argonne National Laboratory | 2024 | [paper](https://arxiv.org/abs/2412.10477) | [code](https://github.com/aldsim/aldbench) | [](https://github.com/aldsim/aldbench) | Expert open-ended QA benchmark evaluating LLMs on atomic-layer-deposition synthesis for accuracy and specificity. | | **AtomWorld** | USTC / Shanghai AI Laboratory / UNSW | 2025 | [paper](https://arxiv.org/abs/2510.04704) | [code](https://github.com/theAfish/AtomWorldBench) | [](https://github.com/theAfish/AtomWorldBench) | Evaluates LLM spatial reasoning on crystalline materials via ten atomic-structure editing actions across four modeling categories over CIF files with verifiable metrics. | | **LLM4Mat-Bench** | Princeton (Vertaix) | 2024 | [paper](https://arxiv.org/abs/2411.00177) | [code](https://github.com/vertaix/LLM4Mat-Bench) | [](https://github.com/vertaix/LLM4Mat-Bench) | Largest benchmark for LLM materials property prediction over 1.9M crystals and 45 properties. | | **MaScQA** | IIT Delhi (M3RG) | 2024 | [paper](https://doi.org/10.1039/D3DD00188A) | [code](https://github.com/M3RG-IITD/MaScQA) | [](https://github.com/M3RG-IITD/MaScQA) | 650 undergraduate-level materials science questions across 14 categories for evaluating LLM knowledge. | | **MatCha** | CUHK-Shenzhen | 2025 | [paper](https://arxiv.org/abs/2509.09307) | [code](https://github.com/FreedomIntelligence/MatCha) | [](https://github.com/FreedomIntelligence/MatCha) | 1,500 expert questions across 21 tasks for materials characterization image understanding by multimodal models. | | **MatSci-NLP** | Mila / Universite de Montreal | 2023 | [paper](https://arxiv.org/abs/2305.08264) | [code](https://github.com/BangLab-UdeM-Mila/NLP4MatSci-ACL23) | [](https://github.com/BangLab-UdeM-Mila/NLP4MatSci-ACL23) | Seven materials-science NLP tasks evaluating language models via a unified text-to-schema approach. | | **MatSciBench** | UCLA | 2025 | [paper](https://arxiv.org/abs/2510.12171) | [code](https://github.com/Jun-Kai-Zhang/MatSciBench) | [](https://github.com/Jun-Kai-Zhang/MatSciBench) | College-level materials-science reasoning benchmark of 1,340 problems spanning six fields, including multimodal questions. | | **MatText** | LamaLab (Jena) / Intel Labs | 2024 | [paper](https://arxiv.org/abs/2406.17295) | [code](https://github.com/lamalab-org/MatText) | [](https://github.com/lamalab-org/MatText) | Benchmarking framework evaluating language models on crystal property prediction across nine text representations. | | **MatViX** | Duke University | 2024 | [paper](https://arxiv.org/abs/2410.20494) | [code](https://github.com/ghazalkhalighinejad/matvix) | [](https://github.com/ghazalkhalighinejad/matvix) | Benchmarks vision-language models on multimodal information extraction from visually-rich materials-science articles. | | **MatVQA** | Mila / Universite de Montreal | 2025 | [paper](https://arxiv.org/abs/2505.18319) | — | — | 1,325 questions testing multimodal models on materials imagery like microscopy and diffraction with multi-step reasoning. | [Back to top](#contents) ### Biology & Life Sciences | Benchmark | Org | Year | Paper | Code | Stars | Description | |---|---|:--:|:--:|:--:|:--:|---| | **BioCoder** | Yale University (Gerstein Lab) | 2023 | [paper](https://arxiv.org/abs/2308.16458) | [code](https://github.com/gersteinlab/BioCoder) | [](https://github.com/gersteinlab/BioCoder) | Fuzz-tested benchmark evaluating LLMs on generating bioinformatics code with cross-file dependencies and domain knowledge. | | **BioLLMBench** | UCLA | 2023 | [paper](https://www.biorxiv.org/content/10.1101/2023.12.19.572483) | — | — | 2,160 runs evaluating GPT-4, Bard, and LLaMA across 24 bioinformatics tasks and metrics. | | **BioMaze** | Peking University | 2025 | [paper](https://arxiv.org/abs/2502.16660) | [code](https://github.com/zhao-ht/BioMaze) | [](https://github.com/zhao-ht/BioMaze) | 5.1K real-research pathway problems testing LLM reasoning over biological pathways and perturbations. | | **BioML-bench** | ScienceMachine | 2025 | [paper](https://www.biorxiv.org/content/10.1101/2025.09.01.673319) | [code](https://github.com/science-machine/biomlbench) | [](https://github.com/science-machine/biomlbench) | AI agents build end-to-end biomedical ML pipelines across protein, omics, imaging, and drug tasks. | | **Biomni** | Stanford | 2025 | [paper](https://www.biorxiv.org/content/10.1101/2025.05.30.656746) | [code](https://github.com/snap-stanford/Biomni) | [](https://github.com/snap-stanford/Biomni) | General biomedical agent with Biomni-Eval: 433 instances across 10 reasoning tasks. | | **BioMysteryBench** | Anthropic | 2026 | [paper](https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench) | — | — | 99 expert-written bioinformatics tasks over raw datasets, judged on the final biological conclusion, not the path. | | **BioPlanner** | FutureHouse / Francis Crick / Oxford | 2023 | [paper](https://arxiv.org/abs/2310.10632) | [code](https://github.com/bioplanner/bioplanner) | [](https://github.com/bioplanner/bioplanner) | BioProt dataset automatically evaluating LLMs on generating biology experimental protocols as executable pseudocode. | | **BioProBench** | Peking University | 2025 | [paper](https://arxiv.org/abs/2505.07889) | [code](https://github.com/YuyangSunshine/bioprotocolbench) | [](https://github.com/YuyangSunshine/bioprotocolbench) | Tests LLMs on biological-protocol QA, step ordering, error correction, generation, and reasoning across 556K instances. | | **BixBench** | FutureHouse | 2025 | [paper](https://arxiv.org/abs/2503.00096) | [code](https://github.com/Future-House/BixBench) | [](https://github.com/Future-House/BixBench) | Bioinformatics agents tackle 53 real analysis scenarios with ~300 open-answer research questions. | | **CellVoyager** | Stanford | 2025 | [paper](https://www.biorxiv.org/content/10.1101/2025.06.03.657517) | [code](https://github.com/zou-group/CellVoyager) | [](https://github.com/zou-group/CellVoyager) | CellBench: 76 scRNA-seq studies test agents predicting which analyses the authors performed. | | **GeneBench-Pro** | OpenAI | 2026 | [paper](https://openai.com/index/introducing-genebench-pro/) | — | — | 129 messy, judgment-intensive computational-biology problems across ten genomics domains on realistic datasets. | | **GeneGPT** | NCBI | 2023 | [paper](https://arxiv.org/abs/2304.09667) | [code](https://github.com/ncbi/GeneGPT) | [](https://github.com/ncbi/GeneGPT) | Teaches LLMs to call NCBI Web APIs; adds GeneHop, evaluated on GeneTuring. | | **GeneTuring** | Columbia University | 2023 | [paper](https://www.biorxiv.org/content/10.1101/2023.03.11.532238) | [code](https://github.com/Winnie09/GeneTuring) | [](https://github.com/Winnie09/GeneTuring) | 16 genomics tasks, 1,600 questions probing LLM genomic knowledge and reasoning. | | **Genome-Bench** | Princeton / Stanford | 2025 | [paper](https://arxiv.org/abs/2505.19501) | — | — | 3,000+ genome-engineering multiple-choice questions mined from expert forum discussions for evaluating LLM reasoning. | | **GenoTEX** | UIUC | 2024 | [paper](https://arxiv.org/abs/2406.15341) | [code](https://github.com/Liu-Hy/GenoTEX) | [](https://github.com/Liu-Hy/GenoTEX) | Expert-curated gene-expression tasks: dataset selection, preprocessing, and gene-trait statistical analysis. | | **HealthBench** | OpenAI | 2025 | [paper](https://arxiv.org/abs/2505.08775) | — | — | 5,000 multi-turn physician-crafted health conversations scored by rubric criteria for LLM performance and safety. | | **LAB-Bench** | FutureHouse | 2024 | [paper](https://arxiv.org/abs/2407.10362) | [code](https://github.com/Future-House/LAB-Bench) | [](https://github.com/Future-House/LAB-Bench) | 2,400+ MCQs across literature, figures, databases, protocols, and DNA/protein sequence tasks. | | **LABBench2** | FutureHouse (Edison Scientific) | 2026 | [paper](https://arxiv.org/abs/2604.09554) | [code](https://github.com/EdisonScientific/labbench2) | [](https://github.com/EdisonScientific/labbench2) | Successor to LAB-Bench with ~1,900 biology-research tasks in realistic contexts (literature, figures, protocols, databases), giving a sharp difficulty jump over LAB-Bench. | | **LifeSciBench** | OpenAI | 2026 | [paper](https://openai.com/index/introducing-life-sci-bench/) | — | — | 750 expert-authored free-response life-science research tasks across seven biological domains, rubric-graded. | | **ProteinLMBench** | Shanghai Jiao Tong University | 2024 | [paper](https://arxiv.org/abs/2406.05540) | [code](https://github.com/tsynbio/ProteinLMDataset) | [](https://github.com/tsynbio/ProteinLMDataset) | 944 verified MCQs assessing LLM comprehension of protein sequences and descriptions. | | **VirBench** | Anthropic (with NCBI, Broad Institute, Pachter Lab) | 2026 | [paper](https://arxiv.org/abs/2606.06749) | — | — | 120 curated viral-sequence retrieval queries across ~40 pathogens testing whether LLM agents pull correct ground-truth sequence data from NCBI Virus. | [Back to top](#contents) ### Agentic Science & AI Research | Benchmark | Org | Year | Paper | Code | Stars | Description | |---|---|:--:|:--:|:--:|:--:|---| | **AAAR-1.0** | Penn State et al. | 2024 | [paper](https://arxiv.org/abs/2410.22394) | [code](https://github.com/RenzeLou/AAAR-1.0) | [](https://github.com/RenzeLou/AAAR-1.0) | Expert-annotated tasks assessing LLMs on equation inference, experiment design, and paper-weakness identification. | | **AstaBench** | Allen AI (AI2) | 2025 | [paper](https://arxiv.org/abs/2510.21652) | [code](https://github.com/allenai/asta-bench) | [](https://github.com/allenai/asta-bench) | 2,400+ problems across eleven benchmarks evaluating agents over the full scientific research pipeline. | | **BLADE** | University of Washington | 2024 | [paper](https://arxiv.org/abs/2408.09667) | [code](https://github.com/behavioral-data/BLADE) | [](https://github.com/behavioral-data/BLADE) | Twelve datasets and research questions evaluating agents' analytical decisions against expert ground truth. | | **CORE-Bench** | Princeton | 2024 | [paper](https://arxiv.org/abs/2409.11363) | [code](https://github.com/siegelz/core-bench) | [](https://github.com/siegelz/core-bench) | 270 tasks from 90 papers testing agents on computationally reproducing published scientific results. | | **Curie** | University of Michigan | 2025 | [paper](https://arxiv.org/abs/2502.16069) | [code](https://github.com/Just-Curieous/Curie) | [](https://github.com/Just-Curieous/Curie) | Framework and 46-question benchmark for rigorous, automated scientific experimentation across four CS domains. | | **DiscoveryBench** | Allen AI (AI2) | 2024 | [paper](https://arxiv.org/abs/2407.01725) | [code](https://github.com/allenai/discoverybench) | [](https://github.com/allenai/discoverybench) | 264 real plus 903 synthetic tasks for data-driven hypothesis discovery across six domains. | | **DiscoveryWorld** | Allen AI (AI2) | 2024 | [paper](https://arxiv.org/abs/2406.06769) | [code](https://github.com/allenai/discoveryworld) | [](https://github.com/allenai/discoveryworld) | Simulated environment with 120 tasks requiring full cycles of hypothesis, experiment, and analysis. | | **DSBench** | UT Dallas / Tencent AI Lab | 2024 | [paper](https://arxiv.org/abs/2409.07703) | [code](https://github.com/LiqiangJing/DSBench) | [](https://github.com/LiqiangJing/DSBench) | 540 realistic data-analysis and data-modeling tasks sourced from ModelOff and Kaggle competitions. | | **EXP-Bench** | University of Michigan | 2025 | [paper](https://arxiv.org/abs/2505.24785) | — | — | Benchmarks agents on designing, implementing, and analyzing end-to-end AI research experiments from publications. | | **HeurekaBench** | EPFL (MLBio Lab) | 2026 | [paper](https://arxiv.org/abs/2601.01678) | [code](https://github.com/mlbio-epfl/HeurekaBench) | [](https://github.com/mlbio-epfl/HeurekaBench) | Builds benchmarks of open-ended research questions grounded in real studies and their code to evaluate end-to-end AI co-scientist agents (instantiated as sc-HeurekaBench in single-cell biology). | | **InnovatorBench** | GAIR-NLP (SJTU) | 2025 | [paper](https://arxiv.org/abs/2510.27598) | [code](https://github.com/GAIR-NLP/InnovatorBench) | [](https://github.com/GAIR-NLP/InnovatorBench) | Evaluates agents on end-to-end innovative LLM-research tasks like data construction, loss design, and reward design. | | **LiveIdeaBench** | Renmin University of China | 2024 | [paper](https://arxiv.org/abs/2412.17596) | [code](https://github.com/x66ccff/liveideabench) | [](https://github.com/x66ccff/liveideabench) | Benchmarks scientific idea generation from minimal keyword context across five dimensions: originality, feasibility, fluency, flexibility, and clarity. | | **LMR-Bench** | UT Dallas | 2025 | [paper](https://arxiv.org/abs/2506.17335) | [code](https://github.com/du-nlp-lab/LMR-Bench) | [](https://github.com/du-nlp-lab/LMR-Bench) | 28 tasks from 23 NLP papers testing whether agents reproduce masked research code, verified by unit tests. | | **MLAgentBench** | Stanford | 2023 | [paper](https://arxiv.org/abs/2310.03302) | [code](https://github.com/snap-stanford/MLAgentBench) | [](https://github.com/snap-stanford/MLAgentBench) | 13 ML experimentation tasks where agents read, write, and execute code to improve performance. | | **MLE-bench** | OpenAI | 2024 | [paper](https://arxiv.org/abs/2410.07095) | [code](https://github.com/openai/mle-bench) | [](https://github.com/openai/mle-bench) | 75 Kaggle ML-engineering competitions testing agents against human leaderboards. | | **MLGym** | Meta | 2025 | [paper](https://arxiv.org/abs/2502.14499) | [code](https://github.com/facebookresearch/MLGym) | [](https://github.com/facebookresearch/MLGym) | Gym framework with 13 open-ended AI research tasks spanning vision, NLP, RL, and game theory. | | **MLR-Bench** | National University of Singapore | 2025 | [paper](https://arxiv.org/abs/2505.19955) | [code](https://github.com/chchenhui/mlrbench) | [](https://github.com/chchenhui/mlrbench) | 201 workshop-derived tasks evaluating agents across idea generation, experimentation, and paper writing. | | **MLRC-Bench** | University of Michigan | 2025 | [paper](https://arxiv.org/abs/2504.09702) | [code](https://github.com/yunx-z/MLRC-Bench) | [](https://github.com/yunx-z/MLRC-Bench) | Benchmarks agents on proposing and coding novel methods for seven recent ML research competition problems. | | **Paper2Code** | KAIST / DeepAuto.ai | 2025 | [paper](https://arxiv.org/abs/2504.17192) | [code](https://github.com/going-doer/Paper2Code) | [](https://github.com/going-doer/Paper2Code) | Multi-agent framework and benchmark generating functional code repositories that reproduce ML research papers. | | **PaperBench** | OpenAI | 2025 | [paper](https://arxiv.org/abs/2504.01848) | — | — | Agents replicate 20 ICML 2024 papers from scratch, graded on 8,316 rubric criteria. | | **RE-Bench** | METR | 2024 | [paper](https://arxiv.org/abs/2411.15114) | [code](https://github.com/METR/RE-Bench) | [](https://github.com/METR/RE-Bench) | Seven open-ended ML research-engineering environments comparing agents against 61 human experts. | | **ResearchBench** | Shanghai AI Lab / NTU | 2025 | [paper](https://arxiv.org/abs/2503.21248) | — | — | Benchmarks scientific discovery via inspiration retrieval, hypothesis composition, and ranking across twelve disciplines. | | **ResearchCodeBench** | Stanford University | 2025 | [paper](https://arxiv.org/abs/2506.02314) | [code](https://github.com/PatrickHua/ResearchCodeBench) | [](https://github.com/PatrickHua/ResearchCodeBench) | Challenges LLMs to implement novel contributions from recent ML papers by completing TODO code snippets. | | **RExBench** | Boston University | 2025 | [paper](https://arxiv.org/abs/2506.22598) | [code](https://github.com/tinlaboratory/RExBench) | [](https://github.com/tinlaboratory/RExBench) | Twelve tasks where agents autonomously implement novel research extensions to existing papers' codebases. | | **SciAgentGym** | Fudan NLP Group | 2026 | [paper](https://arxiv.org/abs/2602.12984) | [code](https://github.com/CMarsRover/SciAgentGYM) | [](https://github.com/CMarsRover/SciAgentGYM) | Agentic science benchmark pairing an environment of 1,780 domain-specific tools across natural-science disciplines with a tiered suite from elementary tool actions to long-horizon workflows. | | **SciCode** | UIUC / Princeton / Argonne National Lab | 2024 | [paper](https://arxiv.org/abs/2407.13168) | [code](https://github.com/scicode-bench/SciCode) | [](https://github.com/scicode-bench/SciCode) | 80 real research coding problems across 16 natural-science subfields, split into 338 subproblems. | | **ScienceAgentBench** | Ohio State (OSU-NLP) | 2024 | [paper](https://arxiv.org/abs/2410.05080) | [code](https://github.com/OSU-NLP-Group/ScienceAgentBench) | [](https://github.com/OSU-NLP-Group/ScienceAgentBench) | 102 data-driven discovery tasks from 44 peer-reviewed papers, evaluating agents that write Python programs. | | **Scientist-Bench** | University of Hong Kong | 2025 | [paper](https://arxiv.org/abs/2505.18705) | [code](https://github.com/HKUDS/AI-Researcher) | [](https://github.com/HKUDS/AI-Researcher) | Evaluates fully autonomous idea-to-paper research systems across CV, NLP, data mining, and IR against expert papers. | | **SciReplicate-Bench** | King's College London | 2025 | [paper](https://arxiv.org/abs/2504.00255) | [code](https://github.com/xyzCS/SciReplicate-Bench) | [](https://github.com/xyzCS/SciReplicate-Bench) | Benchmarks agents on reproducing executable code for algorithms described in 36 recent research papers. | | **SUPER** | Allen AI (AI2) | 2024 | [paper](https://arxiv.org/abs/2409.07440) | [code](https://github.com/allenai/super-benchmark) | [](https://github.com/allenai/super-benchmark) | Evaluates agents on setting up and executing tasks from research code repositories. | [Back to top](#contents) ## Contributing Found a missing benchmark or an error? Edit **only** [`data/benchmarks.yaml`](data/benchmarks.yaml) and open a PR — the README, website, and examples regenerate automatically. See [CONTRIBUTING.md](CONTRIBUTING.md) for the schema and inclusion bar. ## License Released under the [MIT License](LICENSE). Each entry links to its original paper and code; benchmark metadata is drawn from public sources.

标签：DLL 劫持, 人工智能, 大语言模型, 数据评估, 测试基准, 用户模式Hook绕过, 科学推理, 自动化攻击, 防御加固