14 Standard LLM Benchmarks to Know in 2025

Massive Language Fashions (LLMs) have confirmed themselves as a formidable instrument, excelling in each deciphering and producing textual content that mimics human language. However, the widespread availability of those fashions introduces the complicated process of precisely assessing their efficiency. Right here, LLM benchmarks take middle stage, offering systematic evaluations to measure a mannequin’s talent in duties like language understanding and superior reasoning. This text explores their important position, highlights famend examples, and examines their limitations, providing a full image of their affect on language expertise.

Benchmarks are important for evaluating Massive Language Fashions (LLMs), serving as a regular for measuring and evaluating efficiency. They provide a constant option to assess expertise, from primary language comprehension to superior reasoning and programming.

What Are LLM Benchmarks?

LLM benchmarks are structured exams designed to judge the efficiency of language fashions on particular duties. They assist reply important questions resembling:

• Can this LLM successfully deal with coding duties?
• How nicely does it present related solutions in a dialog?
• Is it able to fixing complicated reasoning issues?

Key Options of LLM Benchmarks

• Standardized Exams: Every benchmark consists of a set of duties with identified right solutions, permitting for constant analysis.
• Various Areas of Evaluation: Benchmarks can deal with numerous expertise, together with:
  • Language comprehension
  • Math problem-solving
  • Coding talents
  • Conversational high quality
  • Security and moral concerns

What’s the Want for LLM Benchmarks?

Standardization and Transparency in Analysis

• Comparative Consistency: Benchmarks facilitate direct comparisons amongst LLMs, guaranteeing evaluations are clear and reproducible.
• Efficiency Snapshot: They provide a fast evaluation of a brand new LLM’s capabilities relative to established fashions.

Progress Monitoring and Refinement

• Monitoring Progress: Benchmarks help in observing mannequin efficiency enhancements over time, aiding researchers in refining their fashions.
• Uncovering Limitations: These instruments can pinpoint areas the place fashions fall quick, guiding future analysis and growth efforts.

Mannequin Choice

• Knowledgeable Decisions: For practitioners, benchmarks change into an important reference when selecting fashions for particular duties, guaranteeing well-informed selections for functions like chatbots or buyer help techniques.

Working of LLM Benchmarks

Right here’s the step-by-step course of:

• Dataset Enter and Testing
  • Benchmarks present quite a lot of duties for the LLM to finish, resembling answering questions or producing code.
  • Every benchmark features a dataset of textual content inputs and corresponding “floor reality” solutions for analysis.
• Efficiency Analysis and Scoring: After finishing the duties, the mannequin’s responses are evaluated utilizing standardized metrics, resembling accuracy or BLEU scores, relying on the duty sort.
• LLM Rating and Leaderboards: Fashions are ranked based mostly on their scores, usually displayed on leaderboards that combination outcomes from a number of benchmarks.

Reasoning Benchmarks

1. ARC: The Abstraction and Reasoning Problem

The Abstraction and Reasoning Corpus (ARC) benchmarks machine intelligence by drawing inspiration from Raven’s Progressive Matrices. It challenges AI techniques to establish the following picture in a sequence based mostly on a number of examples, selling few-shot studying that mirrors human cognitive talents. By emphasizing generalization and leveraging “priors”—intrinsic data concerning the world—ARC goals to advance AI towards human-like reasoning. The dataset follows a structured curriculum, systematically guiding techniques by more and more complicated duties whereas measuring efficiency by prediction accuracy. Regardless of progress, AI nonetheless struggles to achieve human-level efficiency, highlighting the continued want for developments in AI analysis.

The Abstraction and Reasoning Corpus features a various set of duties that each people and synthetic intelligence techniques can clear up. Impressed by Raven’s Progressive Matrices, the duty format requires members to establish the following picture in a sequence, testing their cognitive talents.

2. Large Multi-discipline Multimodal Understanding (MMMU)

The Large Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark evaluates multimodal fashions on college-level data and reasoning duties. It consists of 11.5K questions from exams, quizzes, and textbooks throughout six disciplines: Artwork & Design, Enterprise, Science, Well being & Drugs, Humanities & Social Science, and Tech & Engineering.

These questions span 30 topics and 183 subfields, incorporating 30 heterogeneous picture sorts like charts, diagrams, maps, and chemical constructions. MMMU focuses on superior notion and reasoning with domain-specific data, difficult fashions to carry out expert-level duties, and goals to measure notion, data, and reasoning expertise in Massive Multimodal Fashions (LMMs). Analysis of present fashions, together with GPT-4V, reveals substantial room for enchancment, even with superior fashions solely reaching round 56% accuracy. A extra sturdy model of the benchmark, MMMU-Professional, has been launched for enhanced analysis.

Sampled MMMU examples from every self-discipline. The questions and pictures want expert-level data to grasp and motive.

3. GPQA: A Difficult Benchmark for Superior Reasoning

GPQA is a dataset of 448 multiple-choice questions in biology, physics, and chemistry, designed to problem consultants and superior AI. Area consultants with PhDs create and validate the questions to make sure prime quality and problem. Consultants obtain 65% accuracy (74% with retrospectively recognized errors), whereas non-experts with PhDs in different fields rating solely 34%, regardless of unrestricted web entry, proving the questions are “Google-proof.” Main AI fashions like GPT-4 attain simply 39% accuracy. GPQA helps analysis on scalable oversight for AI surpassing human talents, serving to people extract truthful info even on subjects past their experience.

Initially, a query is crafted, after which an skilled in the identical area supplies their reply and suggestions, which can embrace instructed revisions to the query. Subsequently, the query author revises the query based mostly on the skilled’s suggestions. This revised query is then despatched to a different skilled in the identical area and three non-expert validators with experience in different fields. We contemplate skilled validators’ settlement (*) once they both reply appropriately initially or, after seeing the right reply, they supply a transparent rationalization of their preliminary mistake or display a radical understanding of the query author’s rationalization.

4. Measuring Large Multitask Language Understanding (MMLU)

The Large Multitask Language Understanding (MMLU) benchmark, designed to measure a textual content mannequin’s data acquired throughout pretraining. MMLU evaluates fashions on 57 various duties, together with elementary arithmetic, US historical past, laptop science, legislation, and extra. It’s formatted as multiple-choice questions, making analysis easy.

The benchmark goals to be a extra complete and difficult take a look at of language understanding than earlier benchmarks, requiring a mix of data and reasoning. The paper presents outcomes for a number of fashions, displaying that even giant pretrained fashions battle on MMLU, suggesting important room for enchancment in language understanding capabilities. Moreover, the paper explores the affect of scale and fine-tuning on MMLU efficiency.

This process requires understanding detailed and dissonant eventualities, making use of acceptable

authorized precedents, and selecting the right rationalization. The inexperienced checkmark is the bottom reality.

Coding Benchmarks

5. HumanEval: Evaluating Code Era from Language Fashions

HumanEval is a benchmark designed to judge the purposeful correctness of code generated by language fashions. It consists of 164 programming issues with a perform signature, docstring, and a number of other unit exams. These issues assess expertise in language understanding, reasoning, algorithms, and easy arithmetic. In contrast to earlier benchmarks that relied on syntactic similarity, HumanEval evaluates whether or not the generated code truly passes the supplied unit exams, thus measuring purposeful correctness. The benchmark highlights the hole between present language fashions and human-level code era, revealing that even giant fashions battle to provide right code persistently. It serves as a difficult and sensible take a look at for assessing the capabilities of code-generating language fashions.

Under are three illustrative issues from the HumanEval dataset, accompanied by the chances {that a} single pattern from Codex-12B passes unit exams: 0.9, 0.17, and 0.005. The immediate introduced to the mannequin is displayed on a white background, whereas a profitable model-generated completion is highlighted on a yellow background. Though it doesn’t assure drawback novelty, all issues have been meticulously crafted by hand and never programmatically copied from current sources, guaranteeing a novel and difficult dataset.

6. SWE-Bench

SWE-bench is a benchmark designed to judge giant language fashions (LLMs) on their skill to resolve real-world software program points discovered on GitHub. It consists of two,294 software program engineering issues sourced from actual GitHub points and corresponding pull requests throughout 12 common Python repositories. The duty includes offering a language mannequin with a codebase and a problem description, difficult it to generate a patch that resolves the difficulty. The mannequin’s proposed resolution is then evaluated towards the repository’s testing framework. SWE-bench focuses on assessing a complete “agent” system, which incorporates the AI mannequin and the encompassing software program scaffolding accountable for producing prompts, parsing output, and managing the interplay loop2. A human-validated subset referred to as SWE-bench Verified consisting of 500 samples ensures the duties are solvable and supplies a clearer measure of coding brokers’ efficiency

SWE-bench sources process cases from real-world Python repositories by connecting GitHub points to merge pull request options that resolve associated exams. Supplied with the difficulty textual content and a codebase snapshot, fashions generate a patch that’s evaluated towards actual exams

7. SWE-Lancer

SWE-Lancer is a benchmark developed to judge the capabilities of frontier language fashions (LLMs) in finishing real-world freelance software program engineering duties sourced from Upwork, with a complete worth of $1 million. It consists of over 1,400 duties that vary from easy bug fixes, valued at $50, to complicated characteristic implementations price as much as $32,000. The benchmark assesses two kinds of duties: Particular person Contributor (IC) duties, the place fashions generate code patches verified by end-to-end exams by skilled engineers, and SWE Supervisor duties, the place fashions choose one of the best implementation proposals from a number of choices. The findings point out that even superior fashions battle to unravel most duties, highlighting the hole between present AI capabilities and real-world software program engineering wants. By linking mannequin efficiency to financial worth, SWE-Lancer goals to foster analysis into the financial implications of AI in software program growth.

The analysis course of for IC SWE duties includes a rigorous evaluation the place the mannequin’s efficiency is totally examined. The mannequin is introduced with a set of duties, and it should generate options that fulfill all relevant exams to earn the payout. This analysis movement ensures that the mannequin’s output will not be solely right but in addition complete, assembly the excessive requirements required for real-world software program engineering duties.

8. Stay Code Bench 

LiveCodeBench is a novel benchmark designed to supply a holistic and contamination-free analysis of Massive Language Fashions (LLMs) on code-related duties by addressing the constraints of current benchmarks. It makes use of issues sourced from weekly coding contests on platforms like LeetCode, AtCoder, and CodeForces, tagged with launch dates to forestall contamination, and evaluates LLMs on self-repair, code execution, and take a look at output prediction, along with code era. With over 500 coding issues printed between Could 2023 and Could 2024, LiveCodeBench options high-quality issues and exams, balanced drawback problem, and has revealed potential overfitting to HumanEval amongst some fashions, highlighting the various strengths of various fashions throughout various coding duties.

LiveCodeBench affords a complete analysis method by presenting numerous coding eventualities. Coding is a posh process, and we suggest assessing Massive Language Fashions (LLMs) by a collection of analysis setups that seize a spread of coding-related expertise. Past the standard code era setting, we introduce three extra eventualities: self-repair, code execution, and a novel take a look at output prediction process.

9. CodeForces

CodeForces is a novel benchmark designed to judge the competition-level code era talents of Massive Language Fashions (LLMs) by instantly interfacing with the CodeForces platform. This method ensures correct analysis by entry to hidden take a look at circumstances, help for particular judges, and a constant execution surroundings. CodeForces introduces a standardized Elo ranking system, aligned with CodeForces’ personal ranking system however with diminished variance, permitting for direct comparability between LLMs and human rivals. Analysis of 33 LLMs revealed important efficiency variations, with OpenAI’s o1-mini reaching the very best Elo ranking of 1578, inserting it within the high ninetieth percentile of human members. The benchmark reveals the progress made by superior fashions and the appreciable room for enchancment in most present LLMs’ aggressive programming capabilities. The CodeForces benchmark and its Elo calculation logic are publicly accessible.

CodeForces presents a variety of programming challenges, and every drawback is fastidiously structured to incorporate important elements. These elements usually embrace: 1) a descriptive title, 2) a time restrict for the answer, 3) a reminiscence restrict for this system, 4) an in depth drawback description, 5) the enter format, 6) the anticipated output format, 7) take a look at case examples to information the programmer, and eight) an elective notice offering extra context or hints. One such drawback, titled “CodeForces Drawback E,” may be accessed on the URL: https://codeforces.com/contest/2034/drawback/E. This drawback is fastidiously crafted to check a programmer’s expertise in a aggressive coding surroundings, difficult them to create environment friendly and efficient options throughout the given time and reminiscence constraints.

10. TAU-Bench

τ-bench actively evaluates language brokers on their skill to work together with (simulated) human customers and programmatic APIs whereas adhering to domain-specific insurance policies. In contrast to current benchmarks that always characteristic simplified instruction-following setups, τ-bench emulates dynamic conversations between a person (simulated by language fashions) and a language agent outfitted with domain-specific API instruments and coverage tips. This benchmark employs a modular framework that features reasonable databases and APIs, domain-specific coverage paperwork, and directions for various person eventualities with corresponding floor reality annotations. A key characteristic of τ-bench is its analysis course of, which compares the database state on the finish of a dialog with the annotated purpose state, permitting for an goal measurement of the agent’s decision-making.

The benchmark additionally introduces a brand new metric, go^ok, to judge the reliability of agent conduct over a number of trials, highlighting the necessity for brokers that may act persistently and comply with guidelines reliably in real-world functions. Preliminary experiments present that even state-of-the-art perform calling brokers battle with complicated reasoning, coverage adherence, and dealing with compound requests.

τ-bench is an progressive benchmark the place an agent engages with database API instruments and an LM-simulated person to perform duties. It evaluates the agent’s functionality to assemble and convey pertinent info to and from customers by a number of interactions, whereas additionally testing its skill to unravel intricate points in real-time, guaranteeing adherence to tips outlined in a domain-specific coverage doc. Within the τ-airline process, the agent should reject a person’s request to alter a primary financial system flight based mostly on area insurance policies after which suggest another resolution—canceling and rebooking. This process requires the agent to use zero-shot reasoning in a posh surroundings that includes databases, guidelines, and person intents.

Language Understanding and Query Answering Benchmark

11. SuperGLUE

SuperGLUE assesses the capabilities of Pure Language Understanding (NLU) fashions by a complicated benchmark, providing a extra demanding analysis than its predecessor, GLUE. Whereas retaining two of GLUE’s most difficult duties, SuperGLUE introduces new and extra intricate duties that require deeper reasoning, commonsense data, and contextual understanding. It expands past GLUE’s sentence and sentence-pair classifications to incorporate duties like query answering and coreference decision. SuperGLUE designers create duties that college-educated English audio system can handle, however these duties nonetheless exceed the capabilities of present state-of-the-art techniques. This benchmark supplies complete human baselines for comparability and affords a toolkit for mannequin analysis. SuperGLUE goals to measure and drive progress in the direction of growing general-purpose language understanding applied sciences.

The event set of SuperGLUE duties affords a various vary of examples, every introduced in a novel format. These examples usually embrace daring textual content to point the precise format for every process. The mannequin enter integrates the italicized textual content to offer important context or info. It specifically marks the underlined textual content throughout the enter, usually highlighting a selected focus or requirement. Lastly, it makes use of the monospaced font to characterize the anticipated output, showcasing the anticipated response or resolution.

12. HelloSwag

HellaSwag is a benchmark dataset for evaluating commonsense pure language inference (NLI). It challenges machines to finish sentences based mostly on given contexts. Developed by Zellers et al., it incorporates 70,000 issues. People obtain over 95% accuracy, whereas high fashions rating under 50%. The dataset makes use of Adversarial Filtering (AF) to generate deceptive but believable incorrect solutions, making it tougher for fashions to search out the appropriate completion. This highlights the constraints of deep studying fashions like BERT in commonsense reasoning. HellaSwag emphasizes the necessity for evolving benchmarks that preserve AI techniques challenged in understanding human-like eventualities.

Fashions like BERT usually battle to finish sentences in HellaSwag, even once they come from the identical distribution because the coaching information. The inaccurate endings, although contextually related, fail to fulfill human requirements of correctness and plausibility. For instance, in a WikiHow passage, choice A advises drivers to cease at a pink mild for less than two seconds, which is clearly flawed and impractical.

Arithmetic Benchmarks

13. MATH Dataset

The MATH dataset, launched within the article, incorporates 12,500 difficult arithmetic competitors issues. It evaluates the problem-solving talents of machine studying fashions. These issues come from competitions like AMC 10, AMC 12, and AIME, protecting numerous problem ranges and topics resembling pre-algebra, algebra, quantity concept, and geometry. In contrast to typical math issues solvable with identified formulation, MATH issues require problem-solving strategies and heuristics. Every drawback features a step-by-step resolution, serving to fashions be taught to generate reply derivations and explanations for extra interpretable outputs.

This instance consists of various mathematical issues with generated options and corresponding floor reality options. The newest AIME, held on February sixth, rapidly gained curiosity within the math neighborhood. Folks shared issues and options on YouTube, on-line boards, and blogs quickly after the examination. This fast dialogue highlights the neighborhood’s enthusiasm for these challenges. For instance, the primary drawback’s generated resolution is right and clearly defined, displaying a profitable mannequin output. In distinction, the second drawback, involving combinatorics and a determine, challenges the mannequin, resulting in an incorrect resolution.

14. AIME 2025

The American Invitational Arithmetic Examination (AIME) is a prestigious math competitors and the second stage in choosing the U.S. group for the Worldwide Arithmetic Olympiad. Most members are highschool college students, however some gifted center schoolers qualify annually. The Mathematical Affiliation of America conducts this examination.

The maths neighborhood rapidly took curiosity within the latest AIME on February sixth, sharing and discussing issues and options throughout YouTube, boards, and blogs quickly after the examination. This fast evaluation displays the neighborhood’s enthusiasm for these difficult competitions.

This picture denotes an instance drawback and resolution from the AIME 2025 paper. This benchmark focuses on the mathematical reasoning capabilities of an LLM.

Conclusion

Builders create and prepare new fashions nearly every single day on giant datasets, equipping them with numerous capabilities. LLM benchmarks play an important position in evaluating these fashions by answering important questions, resembling which mannequin is finest for writing code, which one excels in reasoning, and which one handles NLP duties most successfully. Subsequently, evaluating fashions on these benchmarks turns into a compulsory step. As we quickly progress towards AGI, researchers are additionally creating new benchmarks to maintain up with developments.