I Tried Making my Personal (Dangerous) LLM Benchmark to Cheat in Escape Rooms

Lately, DeepSeek introduced their newest mannequin, R1, and article after article got here out praising its efficiency relative to value, and the way the discharge of such open-source fashions might genuinely change the course of LLMs endlessly. That’s actually thrilling! And likewise, too massive of a scope to write down about… however when a mannequin like DeepSeek comes out of nowhere with a metal chair, boasting related efficiency ranges to different fashions, what does efficiency actually imply on this context?

In case you comply with AI releases, you’ve seen this dance earlier than. Each new mannequin drops with its graphs exhibiting the way it’s someway concurrently higher than GPT-4 on math issues whereas being smaller and extra environment friendly. However what precisely are these benchmarks measuring? How are they created? And extra importantly, how can we lower by means of the hype to create our personal benchmarks for particular use circumstances?

I wished to study extra about LLM Benchmarking.

Half 1: What’s a Benchmark? (in 3 seconds)

TL:DR — The SATs (a number of, really) for LLMs.

Half 1.1: What’s a Benchmark? (in additional than 3 seconds)

Earlier than we dive into the nitty-gritty of particular benchmarks, let’s take a second to unpack what we even imply by “LLM Benchmark.” As a result of calling them the “SATs for AI” feels each proper and likewise barely oversimplified.

LLM benchmarks are, at their core, structured exams used to measure how effectively giant language fashions carry out on sure duties. These duties might be something from figuring out if a press release is true or false, to summarizing a authorized doc, to producing legitimate Python capabilities. Consider them as curated impediment programs specifically designed by AI researchers to check each related muscle these fashions might need. These frameworks sometimes present a dataset of inputs with identified right outputs, permitting for constant comparability between fashions.

Trendy benchmarks make use of varied analysis methodologies. Classification metrics like accuracy work for duties with discrete right solutions, whereas overlap-based metrics (BLEU, ROUGE) consider free-form textual content technology. Some benchmarks use purposeful testing for code technology, or make use of different LLMs as judges to judge response high quality.

A typical benchmark often comes packaged as:

• A standardized dataset of questions, prompts, or duties (with right or reference solutions).
• An analysis protocol specifying how you can measure success, like accuracy, F1 rating, BLEU/ROUGE for textual content technology, or go/fail charges for coding duties.
• A leaderboard or some type of comparative scoreboard, typically with massive flashy graphs.

Some actually well-known benchmarks embrace MMLU for testing multitask language understanding, TruthfulQA for assessing factual accuracy, and HumanEval for measuring coding capabilities. Outcomes are fairly typically revealed on public leaderboards, which let’s individuals carry out some clear comparability between completely different fashions.

What Makes a Good Benchmark?

1. A Clear Activity Definition: We wish duties which might be unambiguous. The extra easy and well-specified the problem, the simpler it’s to belief the outcomes.
2. Information Integrity: The take a look at set shouldn’t be floating round within the coaching knowledge. As a result of if the mannequin’s seen the very same query 50 instances earlier than, the analysis is about as helpful as giving a math quiz to somebody who already has the reply key.
3. Quantifiable Metrics: You want an ordinary for scoring efficiency — like what number of instances the mannequin’s code passes take a look at circumstances or how shut the generated abstract is to a “ground-truth” abstract.
4. Activity Variety & Issue: If a benchmark is simply too straightforward, everybody simply ACES it on day one, and we study… effectively, nothing. If it’s too area of interest (like “We take a look at solely the mannequin’s skill to rely the digits of Pi for 20 minutes”), that’s additionally not so useful.

Life Ain’t All about The Grades

Benchmarks seize solely a slice of what LLMs can do. In the actual world, your chatbot would possibly have to juggle area data, maintain observe of dialog context, abide by your organization’s insurance policies, and produce fluent, non-offensive replies. No single standardized take a look at on the market totally covers that. As we’ll see within the upcoming case research, the design and execution of a benchmark can closely form the image you get of your mannequin’s efficiency… and generally lead you astray in the event you’re not cautious with the way you measure success.

Now that we have now a way of what Llm Benchmarks are designed to perform (and the place they may fall brief), let’s discover a few examples to see how individuals really construct and use them in observe — with combined outcomes!

Case Examine #1: Leetcode as an LLM Benchmark

As a pupil within the tech area, the phrase “Leetcode” popping up throughout my seek for cool benchmarks raised by blood strain by a statistically vital quantity. In contrast to Leetcode, which sucks, the paper “Efficiency Examine of LLM-Generated Code on Leetcode” was very attention-grabbing — it asks a deceptively easy query: can we use Leetcode to benchmark LLM code technology? Their findings reveal each the promise and pitfalls of this method.

The Benchmark Design

The researchers constructed a three-stage validation system. Native exams catch primary errors, Leetcode’s choose verifies correctness, and a customized benchmarking setup measures efficiency. This setup revealed one thing essential: benchmarking code efficiency is tougher than it appears.

After they in contrast native measurements to Leetcode’s metrics, they discovered solely a 0.28 correlation. Leetcode’s measurements confirmed a lot larger variation (0.089 vs 0.035 domestically). Even worse, Leetcode’s rankings proved unstable — an identical options might drop from the 77th to 54th percentile simply based mostly on submission timing.

The Actual Issues

Three main points emerged that problem Leetcode’s viability as a benchmark:

Information Contamination: Utilizing public issues dangers LLMs having seen the options throughout coaching. The researchers had to make use of solely issues from 2023 to mitigate this.

Platform Instability: Leetcode’s metrics drift over time — reminiscence measurements confirmed a -0.24 correlation with take a look at date. This makes reproducible benchmarking practically unimaginable.

Measurement Reliability: The weak correlation between native and platform measurements raises questions on what we’re really testing.

What It Means for LLM Benchmarking

This research doesn’t simply critique Leetcode — it highlights what we’d like in a code technology benchmark: reproducible measurements, dependable efficiency metrics, and assured training-test separation. Till we have now platforms constructed particularly for this goal, we should be extraordinarily cautious about utilizing competitors platforms as benchmarks.

So! We all know that not all benchmarks are viable benchmarks — what a couple of extra mainstream one?

Case Examine #2: SuperGLUE — Constructing a Higher Language Understanding Benchmark

The SuperGLUE paper tackles an enchanting drawback in AI benchmarking: what do you do when fashions get too good at your exams? When GLUE turned inadequate (with fashions surpassing human efficiency), the researchers needed to rethink how we measure language understanding.

The Benchmark Design

SuperGLUE’s core innovation is its process choice methodology. The researchers collected process proposals from the NLP group and filtered them by means of a rigorous course of: every process wanted clear analysis metrics, public coaching knowledge, and — most significantly — vital headroom between machine and human efficiency.

This resulted in eight duties (I’ve simplified the desk from the doc right here, it’s rather less readable however you must get the sense of what the questions are asking):

What makes these duties particular is their variety in format. In contrast to GLUE’s give attention to sentence classification, SuperGLUE consists of coreference decision, studying comprehension, and extra com plex reasoning duties. Every process measures completely different features of language understanding whereas sustaining clear, quantifiable metrics.

Half 2: Let’s Construct a Bodily Reasoning Benchmark: To Cheat at Escape Rooms

After taking a look at some benchmarks like SuperGLUE and Leetcode, I had an thought: what if we examined LLMs on one thing utterly completely different — bodily reasoning… by means of escape room puzzles?

It’s a reasonably legitimate thought — escape rooms poses potentialities and penalties for failure — screw up one too many puzzles, and your pals will assume you’re fairly silly, and relegate you to spectator responsibility. Fortunately for us nevertheless, they (or the poor workers) don’t know you can sneak a cellphone into an escape room — and simply who to ask for the solutions. Immediately, LLMs face off in opposition to the puzzles of a bodily escape room.

Be aware: That is NOT a rigorous tutorial benchmark (please don’t cite this in papers, why would you even wish to do this?), and even near it, and it’s simply speculated to be a enjoyable solution to take a look at LLM benchmarking and analysis. Please don’t destroy my prompts, I’m conscious they’re unhealthy.

Why Bodily Reasoning?

For actual, although… most LLM benchmarks give attention to linguistic duties (like SuperGLUE) or code technology (like Leetcode). And for good purpose — these are well-defined domains with clear analysis metrics. However real-world drawback fixing typically requires understanding bodily rules and their interactions. The well-known “Can GPT-4 do physics?” debates often focus on mathematical problem-solving, not sensible bodily reasoning.

present benchmarks taught me a couple of key rules:

1. Clear analysis metrics are essential (from SuperGLUE’s task-specific scores)
2. Issues ought to have unambiguous options (from HumanEval’s take a look at circumstances)
3. The benchmark ought to take a look at distinct capabilities (from MMLU’s topic classes)

Designing the Issues

I settled on escape room puzzles for 2 causes. First, they naturally mix bodily reasoning with clear targets. Second, they’ve unambiguous success situations — both you remedy it by means of the supposed approach, otherwise you don’t. Third, and most significantly, they let me embrace “pink herrings” — irrelevant gadgets that take a look at if the LLM can establish what issues bodily. Fourth, I simply actually like doing escape rooms (did I point out that already?),

I’m conscious that that is greater than two causes, but when LLMs can’t rely what number of rs’ there are in strawberry, I’m allowed to mess up from time to time too.

Right here’s how I structured the 5 core issues:

Fluid Dynamics (FLUID_001) (Ping pong ball caught in a tube)

• Assessments understanding of buoyancy and fluid displacement
• Impressed by traditional physics issues however in sensible context
• Consists of deliberately irrelevant gadgets (like squishy meals fashions)

Mild Properties (UV_001) (UV mild on a push numebr lock)

• Assessments understanding of UV fluorescence and materials properties
• Combines a number of bodily rules (mild, materials science)
• Requires understanding of environmental situations

Mechanical Understanding (CIPHER_001) (A cipher ring)

• Assessments spatial reasoning and mechanical alignment
• No pink herrings — exams for correlating a dial to a cypher wheel
• Requires understanding rotational symmetry

Drive Software (VAC_001) (Can caught in gap)

• Assessments understanding of vacuum forces and floor adhesion
• A number of potential resolution approaches
• Requires understanding pressure multiplication

Collaborative Physics (COLLAB_001) (Can two individuals shimmy a key?)

• Assessments understanding of bodily constraints in multi-agent situations
• Requires combining a number of bodily rules
• Assessments understanding of software creation and friction

Sounds actually fancy… however it’s just a few primary bodily puzzles. You may entry them on my GitHub.

The Technical Half

The benchmark implementation has three foremost parts:

Downside Definition Layer

Issues are outlined in a structured JSON format that enforces constant analysis:

{
    "problem_id": "FLUID_001",
    "setup": {
        "situation": "A ping pong ball is on the backside of a slender tube...",
        "available_items": ["bottle of water", "squishy food models"...],
        "constraints": ["tube too narrow for manual retrieval"]
    },
    "physical_principles": ["buoyancy", "fluid displacement"],
    "red_herrings": ["squishy food models", "milk carton"],
    "resolution": {
        "steps": ["pour water into tube", "allow ball to float"],
        "key_insights": ["water displaces air", "ping pong ball less dense"]
    }
}

This construction attracts from SuperGLUE’s design — every part is clearly separated and machine-readable. The physical_principles subject explicitly lists what’s being examined, whereas red_herrings helps in scoring the LLM’s skill to disregard irrelevant data.

2. Analysis Framework

The analysis system makes use of Python’s asyncio for concurrent testing, with retry logic for a little bit bit extra API stability:

@retry(cease=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
async def evaluate_response(self, standards: JudgingCriteria) -> Dict:
    """Consider a mannequin's response utilizing GPT-4 as choose."""
    async with aiohttp.ClientSession() as session:
        # ... analysis logic

The scoring system appears at three parts:

Bodily Understanding Rating (PUS) ∈ [0,2]

• Measures understanding of related bodily rules
• Calculated as normalized sum of demonstrated rules

Answer Path Rating (SPS) ∈ [0,2]

• Evaluates completeness and correctness of resolution steps
• Considers sensible feasibility of proposed options

Pink Herring Dealing with (RHH) ∈ {0,1}

• A Binary rating for avoiding irrelevant gadgets
• Assessments skill to give attention to bodily related elements

And sure, there are additionally so many different scoring strategies, higher and worse, that might be used! For instance, RHH might be about how many irrelevant gadgets are used within the resolution, or it might be a measure of how viable the use is… the purpose is that choosing these metrics are sometimes instances fairly arbitrary, however are very essential to creating your benchmark is credible, which mine could be very a lot not.

Moreover, I didn’t wish to rewrite any code after. Sue me.

3. Mannequin Interface Layer

The benchmark helps a number of LLM backends by means of a standard interface:

class ModelInterface:
    """Interface for various LLM APIs."""
    async def generate_response(self, immediate: str) -> str:
        elevate NotImplementedError

class GPT4Interface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # GPT-4 particular implementation
class ClaudeInterface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # Claude particular implementation

Two fashions… I can’t actually afford any extra, please perceive.

Let’s Check It!

So after some refilling of API balances, I used to be able to go. I ran the benchmark, and after completely not losing credit on like, 5 completely different errors, the outcomes had been in. They usually had been… attention-grabbing! I used GPT4 and Claude 3.5 Haiku (I’m conscious these aren’t equally spec-ed fashions) as our victims.

The Problem: A can is caught in a gap that nearly precisely matches its quantity. A mix is written on the underside of the can that must be retrieved.

"VAC_001": {
        "problem_id": "VAC_001",
        "setup": {
            "available_items": [
                "ruler",
                "notebook",
                "suction cup",
                "UV light"
            ],
            "constraints": [
                "can fits too tightly for finger grip",
                "hole is same diameter as can",
                "can is clean and dry"
            ]
        },
        "physical_principles": [
            "vacuum suction",
            "surface adhesion",
            "force multiplication"
        ],
        "red_herrings": [
            "ruler",
            "notebook",
            "UV light"
        ],
        "resolution": {
            "steps": [
                "inspect suction cup and can surface",
                "ensure can surface is clean",
                "firmly press suction cup onto can top",
                "create vacuum seal",
                "pull can out using suction cup"
            ],
            "key_insights": [
                "suction cup creates vacuum seal",
                "vacuum force allows extraction",
                "other items don't provide necessary force"
            ]
        }

GPT-4’s Answer

Bodily Ideas: The principle bodily precept concerned on this resolution is the idea of air strain differential, which is similar precept that enables a suction cup to stay to a floor. Air strain is larger on one aspect of the suction cup (outdoors) in comparison with the opposite (inside), and this distinction in strain causes the cup to stay to the floor.

Answer Steps:

1. Connect the suction cup to the highest of the can. The flat floor of the can ought to present an excellent contact for the suction cup to stick to.
2. As soon as the suction cup is firmly connected, gently pull upwards. The air strain differential ought to trigger the suction cup (and thus the can) to be lifted out of the opening.
3. As soon as the can is out of the opening, take away the suction cup from the highest of the can.
4. Use the UV mild to light up the mixture written on the underside of the can. Some inks solely develop into seen below UV mild, which is why this step is critical.
5. Write down the mixture on the pocket book to make sure it’s not forgotten.

Required Gadgets:

• Suction cup: That is required to stick to the can and pull it out of the opening, because the can matches too tightly for finger grip.
• UV mild: That is required to disclose the mixture written on the underside of the can, as it might be written in ink that solely turns into seen below UV mild.
• Pocket book: That is required to file the mixture as soon as it’s seen, guaranteeing it’s not forgotten.”

Not unhealthy in any respect! GPT-4 manages to determine the bodily precept behind the answer (even when it is vitally wordy), and manages to step-by-step describe how you can remedy the issue. It does find yourself utilizing the UV mild and pocket book, however that’s extra of an issue on my finish, not punishing using pointless gadgets within the immediate sufficient. Additionally, the makes use of for these things appear decently cheap… so why not!

Haiku-3.5’s Answer

Bodily Ideas: The important thing bodily rules concerned on this situation are the idea of quantity and using suction to extract the can from the opening. For the reason that can matches virtually precisely into the opening, it’s seemingly that the can is caught as a result of tight match and the dearth of any air gaps or area between the can and the opening. This tight match creates a vacuum-like impact, making it troublesome to grip and pull the can out.

Answer Steps:

1. Measure the diameter of the can utilizing the ruler. This may assist decide the dimensions of the opening and ensure that the can matches tightly inside it.
2. Examine the underside of the can to find the mixture written on it. Use the UV mild to light up the underside of the can and make the mixture seen, if it’s not already clearly seen.
3. Connect the suction cup to the highest of the can. The suction cup will create a short lived seal on the can’s floor, permitting you to grip and pull it out of the opening.
4. Gently pull the can upwards utilizing the suction cup. The suction pressure must be sturdy sufficient to beat the vacuum-like impact and extract the can from the opening.
5. As soon as the can is out, file the mixture written on the underside of the can within the pocket book.

Required Gadgets:

• Ruler: To measure the diameter of the can and ensure the tight match within the gap.
• UV mild: To light up the underside of the can and make the mixture seen, if it’s not already clearly seen.
• Suction cup: To create a short lived seal on the can’s floor and supply the required grip to tug the can out of the opening.
• Pocket book: To file the mixture written on the underside of the can.

Alternatively, Haiku’s resolution was.. actually an answer of all time.

First, it desires to measure the diameter of the can utilizing the ruler??? To find out the dimensions of the opening and ensure that the can matches tightly inside it? Why would we have to do that? And do we’d like a ruler for that?

Second, it tells us to examine the underside of the can to find the mixture, when the complete drawback is about not with the ability to pull the can out of the opening conventionally. This would possibly simply be a difficulty of order, however now I really perceive my buddies’ emotions at any time when I’d inform them “simply repair it man” to their quite a few issues.

However it will definitely does get the answer. So… not the worst.

Right here’s a flowery radar graph of the outcomes!

We see that each fashions are fairly related of their capabilities, with GPT-4 being barely higher in bodily understanding and resolution path, and Haiku being barely higher in pink herring dealing with. General although, each fashions sort of suck. Dang.

There are additionally solely… 5 questions.

In case you’d wish to see the total breadth of questions, they’re on my GitHub.

LLM-as-a-Choose

By the way in which, the tactic I used to generate the evaluations, LLM-as-a-judge, has gained vital traction within the AI group, notably after the work of Zheng et al. of their 2023 paper “Judging LLM-as-a-Choose.” The approach has confirmed remarkably efficient, attaining over 80% settlement with human evaluators in duties starting from code evaluation to dialogue high quality analysis!

Right here’s the place my experiment will get sort of cool (arguably, perhaps, subjectively) — I used this system and had GPT-4 choose different LLMs’ bodily reasoning skills. Sure, I’m utilizing an AI to evaluate different AIs.

Why does this work? Effectively, judging a response is definitely a less complicated process than producing one. When GPT-4 generates an answer to a bodily puzzle, it must:

• Perceive the bodily rules concerned
• Plan a sequence of steps
• Contemplate all constraints
• Generate a coherent rationalization

However when judging, it solely must test if particular standards are met in an present resolution. The analysis immediate could be very targeted:

def _create_evaluation_prompt(self, standards: JudgingCriteria) -> str:
    return f"""You're an skilled choose evaluating an LLM's understanding of bodily reasoning puzzles.

Consider based mostly on three standards:
2. Bodily Understanding Rating (0-2): Does the answer accurately apply related bodily rules?
3. Answer Path Rating (0-2): Are the steps full and possible?
4. Pink Herring Dealing with (0-1): Does it keep away from utilizing irrelevant gadgets?
Situation: {standards.situation}
Bodily Ideas Required: {standards.correct_principles}
Answer Given: {standards.model_response}
"""

To validate this method, I adopted the validation framework instructed by Zheng et al., performing spot-checks of GPT-4’s evaluations in opposition to my very own judgments. Surprisingly (or maybe unsurprisingly, given the broader analysis on LLM analysis), it was remarkably constant in figuring out each right bodily understanding and flawed reasoning.

Is that this good? Completely not. There’s one thing philosophically bizarre about utilizing one LLM to judge one other. However in observe, it could possibly work surprisingly effectively — identical to how I moan and groan in regards to the visible presentation of a dish on Masterchef, whereas setting my kitchen aflame attempting to microwave a scorching canine.

What I Realized

Constructing this benchmark taught me a number of issues about benchmark design:

Clear Metrics Matter: Even for advanced duties like bodily reasoning, you want unambiguous scoring standards.

Pink Herrings Are Highly effective: Together with irrelevant gadgets reveals loads about an LLM’s reasoning course of.

Context Management is Laborious: Guaranteeing LLMs don’t “hallucinate” extra bodily context is difficult.

Is that this an ideal benchmark? Not even shut. Please don’t rub it in. Is it scientifically rigorous? Undoubtedly not. However it’s been an enchanting exploration into a facet of LLM capabilities, and generally the most effective we will study can come from simply attempting issues out and seeing what occurs.

Now, in the event you’ll excuse me, I shall be sneaking in a cellphone with an web connection into my subsequent escape room, for causes that I’m legally unmotivated to reveal.

[1] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, “Judging LLM-as-a-Choose with MT-Bench and Chatbot Enviornment,” Proceedings of the thirty seventh Convention on Neural Data Processing Programs (NeurIPS 2023), Datasets and Benchmarks Observe (2023)

[2] T. Coignion, C. Quinton, R. Rouvoy, “A Efficiency Examine of LLM-Generated Code on Leetcode,” In twenty eighth Worldwide Convention on Analysis and Evaluation in Software program Engineering (EASE 2024), Salerno, Italy (2024)

[3] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Bowman, “SuperGLUE: A Stickier Benchmark for Basic-Goal Language Understanding Programs,” In thirty third Convention on Neural Data Processing Programs (NeurIPS 2019), Vancouver, Canada (2019)

[5] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Tune, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z.F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao et al., “DeepSeek-R1: Incentivizing Reasoning Functionality in LLMs by way of Reinforcement Studying,” arXiv preprint arXiv:2501.12948 (2025)

[6] Until in any other case said, all photos are created by the writer.