ReplicationBench evaluates whether AI agents can replicate real astrophysics research papers from scratch. The dataset consists of 111 astrophysics research…
AttributionMIT License - Copyright (c) 2025 Christine Ye and contributors
Based on Exercism coding exercises, this benchmark evaluates how effectively LLMs can translate natural language coding requests into executable code that…
AttributionAll exercise content is copyright (c) Exercism. These exercises are used in accordance with Exercism's open source licenses.
Terminal-Bench is a benchmark designed to evaluate LLM agent performance in real terminal environments. Tasks span file manipulation, software installation,…
AttributionMike Merrill, Alex Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly…
R2E-Gym is a benchmark and interactive environment suite that measures an LLM agent's ability to read instructions, explore an environment, and then execute…
AttributionNaman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, Ion Stoica
R2E-Gym is the largest procedurally-curated executable gym environment for training real-world software engineering agents. It consists of over 8,100 tasks acro…
AttributionNaman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica.
SWE-Lancer is a large-scale benchmark of 1,488 real freelance software engineering tasks sourced from Upwork, totaling $1M in contracts. It evaluates models…
AttributionSamuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke (all from OpenAI)
CruxEval is a benchmark designed to test fine-grained reasoning and debugging skills of large language models. It includes tasks such as identifying algorithmic…
AttributionAlex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, Sida I. Wang
BigCodeBench is a large-scale benchmark created to evaluate realistic, developer-grade coding abilities of LLMs. Tasks include software engineering problems, bu…
SWE-smith is a toolkit designed to generate a large number of SWE-bench style task instances for any given Python repository. It allows users to create training dat…
AttributionJohn Yang, Kilian Leret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmi…
LegalBench is a collaboratively built benchmark of 162 tasks measuring legal reasoning in LLMs, spanning issue-spotting, rule-recall, rule-application, and interpretation.
AttributionNeel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, et al. (Stanford CRFM / HazyResearch)
MedQA evaluates medical knowledge and clinical reasoning with multiple-choice questions drawn from US medical licensing board exams (USMLE), covering diagnosis, treatment, and pharmacology.
FinQA is a numerical-reasoning benchmark over real financial reports: each question requires multi-step arithmetic grounded in tables and text from S&P 500 earnings filings.