An agent with a spreadsheet engine beats one without

Our results are now on the official SpreadsheetBench leaderboard: 91.25% on the verified 400 set, using Claude Sonnet.

The rest of this post is how we got here: what SpreadsheetBench is, what six months of running the benchmark taught us - and why an agent with a spreadsheet engine beats an agent without one.

We're also releasing @grid-is/agent-tools: the tools our spreadsheet agent uses to read, write, and understand workbooks - complementing our free-for-evaluation GRID engine.

What is SpreadsheetBench?

SpreadsheetBench is a benchmark for evaluating how well LLMs solve real spreadsheet tasks. It was introduced in a 2024 NeurIPS paper by researchers from Renmin University of China, Tsinghua University, and Zhipu.AI (Zeyao Ma, et al.). The dataset holds 912 tasks scraped from real Excel forum threads. Scoring is done by comparing cells: the cells your agent produced against the cells an expert says should be there.

In late 2025 the researchers released a 400-task subset in which every task has been reviewed by experts, the "verified 400", in collaboration with Shortcut (see blog). Their objective was to create a subset of tests with fewer correct interpretations, optimized for automated evaluation. By removing overly ambiguous and simple tasks they created, in our opinion, a subset which gives a much cleaner signal, and it's the set all our published numbers refer to.

Why we ran it

We build a spreadsheet engine. This is a tricky thing to do, because a modern spreadsheet is a combination of decades of features piled on top of one another, none particularly well documented. Unless you spend your days knee-deep in a workbook, you may not have thought about the engine underneath. You just use Excel, and the calculation engine is invisible.

When you've built something invisible, it's hard to tell people what you've done. You can say "We've written a spreadsheet engine, from scratch, that runs the calculations itself, manages the dependency graph, handles the awkward edges of Excel semantics, and exposes itself cleanly to code and to language models", and people will nod politely and then ask you if you have any interesting hobbies.

So how exactly should we show off our engine? We have demos, we have case studies, standard stuff. And then we discovered SpreadsheetBench. 912 real-world problems, nabbed from Excel forums, scored by comparing cells to cells. The tasks are the sort of thing a colleague sends you at half past four on a Friday: combine these two sheets by matching on text in two columns, oh and by the way there's a totals row, oh and by the way new rows will be added later, so please make it survive that. A great test for an engine.

This would help us test our hypothesis: an agent with a spreadsheet engine is better than an agent without. If you ask an LLM to solve a spreadsheet problem, it will often reach for the OpenPyXL Python library. But OpenPyXL is designed to read and write .xlsx files, not to provide a spreadsheet engine. It can't run calculations, it can't run formulas, it can't pivot tables. Instead, the LLM needs to write Python code to calculate values, and at that point a spreadsheet problem becomes a programming problem. It tries to solve the programming problem and hope the translation was lossless. It usually isn't.

Meanwhile our engine, sitting quietly on the other side of the room, can just evaluate the formula. It can solve the spreadsheet problem. We think an engine provides a good foundation for LLMs to solve the sort of problems people have in spreadsheets.

There was another reason too. We wanted to see our own engine and harness work on problems we hadn't chosen. It is very easy, when you build something, to end up testing it on the things it happens to be good at. SpreadsheetBench would help us get out of our comfort zone and find out if our spreadsheet engine could stand up against unforeseen problems.

How we benefitted

Our first tests, in December 2025, ran against the online MCP API we were hosting at the time, and they were surprisingly slow. That turned out to be the token-hungry nature of spreadsheets, not the service. So we quickly built an agent backend with the engine running directly inside it: a fuller exercise in what we believed our customers would want to build, and a faster way to iterate.

We'd been developing that engine for more than seven years, but we had only just taught it to save modified .xlsx files. Until then it had powered our first product, Calculator Studio, which turned spreadsheets into visual, interactive calculators that read and calculated across more than 200,000 workbooks but never wrote changes back to a file. Durable editing was brand new.

The first tests bore that out. They surfaced plenty of weaknesses in our export path (enough, we reckon, to account for something like 10–15% of the failures) and left us around 75% to start, depending on the tasks, model, and prompt.

We started from the prompts and tools we'd been using for our LLM showcase. Our tools at the time reflected that history: mostly read-oriented (detecting data labels, understanding workbook structure), with the write path barely out of the oven. Running hundreds of forum-sourced tasks through it, then round-tripping the results back out to .xlsx for scoring, shook out bugs at a rate no test suite we could have written ourselves would have matched. The benchmark quietly became a development tool.

It also changed how we run things. Waiting a few hours for a sequential run of hundreds of tasks to surface one regression is no way to live, so we built our own runner: tasks execute in parallel against the agent and are evaluated while the run is still going. Full 400-task runs are our high-fidelity check, exercising the whole export and roundtrip path, but they aren't cheap in tokens, so we don't run them constantly. Instead we used a handful of full runs to rate each task's difficulty, then picked 100 tasks representative of the overall spread and ran that smaller set far more often. It gives a fast, clear signal while we iterate on tools and prompts, and lets us reserve the full 400 for less frequent, higher-fidelity checks.

Once the score settled into a reliable 85–90%, the benchmark's job changed. Since February this year we mostly stopped working on the agent itself: our customers needed pivot tables, charts, and other core engine work more than they needed a better benchmark score, so that's where our efforts were directed. The benchmark stayed on as a regression suite, guarding the score while we evolve engine features, tools, and skills the benchmark doesn't measure at all.

The numbers

Over the next few months these regression runs accumulated the following results:

The trend is really two stories. The big jump from sonnet-4-5 to sonnet-4-6 was mostly our own doing: those runs carry far more advanced GRID tooling and engine capabilities. The improvement since then is split between smaller tooling refinements and the model itself: sonnet-5 sits at 90% or above in six of its seven runs.

It's worth noting that we'd probably need more than 6 or 7 runs for a given configuration of model + GRID tools to understand how well we do. A typical run fails 35–45 of the 400 tasks, but most of that is churn, the run-to-run noise we measure below. Just 14 tasks failed in all seven of the recent claude-sonnet-5 runs; put the other way, 96.5% of the verified 400 passed at least once within that window. Widen the window to every run we've recorded and only three tasks have never passed: 99.25% solved at least once, and the remaining three are arguably defective tasks, not lack of capability.

A note on error bars, because leaderboards don't usually draw them. Every eval we'd built before this one was single-turn, and single-turn evals at temperature 0 are fairly stable, across both Claude and GPT models. Nothing had prepared us for how stochastic a 20-plus-turn agentic run is: early on, re-running a day's failed tasks lifted the score from 82-something to 85.7%, and a subsequent full re-run erased those gains.

So we measured it properly. We ran the same agent build, configuration and model (claude-sonnet-4-6) five times over a fixed 100-task sample: 12 of the 100 tasks flipped between pass and fail across the runs. An earlier four-run group on claude-sonnet-4-5 showed the same: 14 of 100 flipped. Both groups ran with temperature=0 and top_k=1, sampling pinned to the single most likely token at every step, so the flips survive even with sampling nailed down.

The take-away: a single-run score on this benchmark carries several points of noise, and comparisons between agents that differ by a point or two are mostly comparisons of luck. We're not the only ones to notice: the GPT for Work team's write-up of their own run reaches the same conclusion, attributing most of their misses to variance rather than a capability gap, with several failing tasks flipping to pass on re-runs.

We standardize on Sonnet because that's the model businesses actually deploy at scale. It can be a real matter of economics: Sonnet is about 40% cheaper, on a workload as token-hungry as spreadsheets. A leaderboard doesn't record what a score costs to serve. In our controlled head-to-head with identical agent build on verified 400 and only the model swapped, Opus 4.8 scored one point higher than Sonnet 5 (91.0% vs 90.0%). A second Opus run came back at 90.0%. The noise shows that the choice has negligible effect on score.

Reproducibility

We're publishing our leaderboard run in full, every transcript and every output file. It lands in the same ~90% band as the rest of our runs; a single number here means little, which is rather the point. A score without its transcripts is just a claim, and we'd like publishing them to become the norm. Dealglass, whose Tetra sits ahead of us on the leaderboard, led the way here: their spreadsheet-agents dataset includes session logs for their baseline GPT-5.4 and Claude Opus 4.6 runs, alongside the eval results for their own agent.

Two tasks from a recent run show what having the engine in the loop looks like, and why it matters:

• 48643 wants a column of accumulated wealth over 39 years: each year's deposit added to the previous year's total, grown by that year's rate. The calculation is inherently recursive, each row depends on the one above it, so there's no closed-form shortcut. Our agent evaluated the candidate formula against the live workbook first, then wrote it to the top cell and filled it down the column in one call. The engine recalculated the whole 39-row chain, and the agent read the computed values back to confirm the compounding behavior. Without an engine the agent would have to reimplement 39 rows of compounding arithmetic in Python and hope the rounding matches, or write the formulas blind.

• 48982 is a debugging task: a running-counter formula in B323 displays "000" and Excel complains about a circular reference, and the asker wants to know why. The display is a red herring (the cell's number format is literally 000), and the real bug is in the formula, =IF(A323="E","",MAX(B$7:B377)+1). The MAX range includes row 323 itself, so the cell feeds its own calculation. Our agent found this by reading the formula and its format, narrowed the range to B$7:B322, and evaluated the fixed cell in place to confirm that it now computed a value. A circular reference only exists, and only gets caught, when something is actually running the calculation.

What the benchmark is, and isn't

The good parts we covered up top: forum-sourced tasks with the underspecified shape of real work, plus expert review on the verified 400. It's the best public signal we've found for spreadsheet agent capability.

But there are limitations:

• The reference harness assumes your agent writes Python. It generates code, runs it against the file, opens the result in Excel to recalculate, and then compares cells. That's a sensible pipeline if OpenPyXL is your only option, but unnecessary if your agent has an engine. Ours reads and edits cells directly and calculates natively.

• Some tasks have more than one valid solution, so getting them reliably right can mean you've overfit your prompts and tools rather than solved anything general. Only three of the 400 tasks have never passed in any of our runs, on any model. Every other task has passed at least once; most single-run failures are just tasks that fail one run and pass the next. We've shared our notes on some of these.

• Run-to-run volatility (see above) made us decide it's more valuable as a regression and optimization suite than anything else. That's how we use it: as the thing that tells us when an engine change broke the write path, not as a number to chase.

There is also plenty a cell-comparison benchmark can't see: formatting and styles aren't scored at all, and comparing cells, or exact ranges, limits the tasks to those with a single, fixed position where the answer can be checked. SpreadsheetBench V2 addresses both, making it more representative of real spreadsheet work. That makes V2 the version of the benchmark we believe in, and the one our effort is aimed at next. Its leaderboard is young (so far it holds only the researchers' own model baselines, with the best at 34.9%), so nobody has really run at it yet.

Try it for yourself

The tooling layer the agent uses (the tools for reading, writing, and understanding workbooks that all these runs battle-tested) is now available as @grid-is/agent-tools. The engine itself is already available for non-commercial use, and together with our viewer and editor it's what a number of companies already license to build spreadsheet capability into their own products, increasingly for AI-agent workflows. The newest example: a financial-modeling agent one of our clients released on the engine's latest version just weeks ago. We're also strategically open-sourcing parts of the stack, more on that later.

And credit is overdue. SpreadsheetBench has given us far more value than a number to put in a blog post: it has been a working development tool for half a year, and will be for the foreseeable future.

Everything here is public:

• The agent tools: docs.grid.is/agent-tools

• The engine, free for non-commercial use: grid.is

• Our leaderboard run, transcripts and outputs: github.com/GRID-is/spreadsheetbench-results

• Install the tools into Claude Code or Codex: GRID skill marketplace

• An agent with a spreadsheet engine beats an agent without one. Now you have the tools to check.