The GRID Spreadsheet Benchmark: how close do spreadsheet engines get to Excel?

An agent with a spreadsheet engine beats one without

Our results are now on the official SpreadsheetBench leaderboard: 91.25% on the verified 400 set, using Claude Sonnet. The rest of this post is how we got here: what SpreadsheetBench is, what six months of running the benchmark taught us - and why an agent with a spreadsheet engine beats an agent without one. We're also releasing @grid-is/agent-tools: the tools our spreadsheet agent uses to read, write, and understand workbooks - complementing our free-for-evaluation GRID engine. What is SpreadsheetBench? SpreadsheetBench is a benchmark for evaluating how well LLMs solve real spreadsheet tasks. It was introduced in a 2024 NeurIPS paper by researchers from Renmin University of China, Tsinghua University, and Zhipu.AI (Zeyao Ma, et al.). The dataset holds 912 tasks scraped from real Excel forum threads. Scoring is done by comparing cells: the cells your agent produced against the cells an expert says should be there. In late 2025 the researchers released a 400-task subset in which every task has been reviewed by experts, the "verified 400", in collaboration with Shortcut (see blog). Their objective was to create a subset of tests with fewer correct interpretations, optimized for automated evaluation. By removing overly ambiguous and simple tasks they created, in our opinion, a subset which gives a much cleaner signal, and it's the set all our published numbers refer to. Why we ran it We build a spreadsheet engine. This is a tricky thing to do, because a modern spreadsheet is a combination of decades of features piled on top of one another, none particularly well documented. Unless you spend your days knee-deep in a workbook, you may not have thought about the engine underneath. You just use Excel, and the calculation engine is invisible. When you've built something invisible, it's hard to tell people what you've done. You can say "We've written a spreadsheet engine, from scratch, that runs the calculations itself, manages the dependency graph, handles the awkward edges of Excel semantics, and exposes itself cleanly to code and to language models", and people will nod politely and then ask you if you have any interesting hobbies. So how exactly should we show off our engine? We have demos, we have case studies, standard stuff. And then we discovered SpreadsheetBench. 912 real-world problems, nabbed from Excel forums, scored by comparing cells to cells. The tasks are the sort of thing a colleague sends you at half past four on a Friday: combine these two sheets by matching on text in two columns, oh and by the way there's a totals row, oh and by the way new rows will be added later, so please make it survive that. A great test for an engine. This would help us test our hypothesis: an agent with a spreadsheet engine is better than an agent without. If you ask an LLM to solve a spreadsheet problem, it will often reach for the OpenPyXL Python library. But OpenPyXL is designed to read and write .xlsx files, not to provide a spreadsheet engine. It can't run calculations, it can't run formulas, it can't pivot tables. Instead, the LLM needs to write Python code to calculate values, and at that point a spreadsheet problem becomes a programming problem. It tries to solve the programming problem and hope the translation was lossless. It usually isn't. Meanwhile our engine, sitting quietly on the other side of the room, can just evaluate the formula. It can solve the spreadsheet problem. We think an engine provides a good foundation for LLMs to solve the sort of problems people have in spreadsheets. There was another reason too. We wanted to see our own engine and harness work on problems we hadn't chosen. It is very easy, when you build something, to end up testing it on the things it happens to be good at. SpreadsheetBench would help us get out of our comfort zone and find out if our spreadsheet engine could stand up against unforeseen problems. How we benefitted Our first tests, in December 2025, ran against the online MCP API we were hosting at the time, and they were surprisingly slow. That turned out to be the token-hungry nature of spreadsheets, not the service. So we quickly built an agent backend with the engine running directly inside it: a fuller exercise in what we believed our customers would want to build, and a faster way to iterate. We'd been developing that engine for more than seven years, but we had only just taught it to save modified .xlsx files. Until then it had powered our first product, Calculator Studio, which turned spreadsheets into visual, interactive calculators that read and calculated across more than 200,000 workbooks but never wrote changes back to a file. Durable editing was brand new. The first tests bore that out. They surfaced plenty of weaknesses in our export path (enough, we reckon, to account for something like 10–15% of the failures) and left us around 75% to start, depending on the tasks, model, and prompt. We started from the prompts and tools we'd been using for our LLM showcase. Our tools at the time reflected that history: mostly read-oriented (detecting data labels, understanding workbook structure), with the write path barely out of the oven. Running hundreds of forum-sourced tasks through it, then round-tripping the results back out to .xlsx for scoring, shook out bugs at a rate no test suite we could have written ourselves would have matched. The benchmark quietly became a development tool. It also changed how we run things. Waiting a few hours for a sequential run of hundreds of tasks to surface one regression is no way to live, so we built our own runner: tasks execute in parallel against the agent and are evaluated while the run is still going. Full 400-task runs are our high-fidelity check, exercising the whole export and roundtrip path, but they aren't cheap in tokens, so we don't run them constantly. Instead we used a handful of full runs to rate each task's difficulty, then picked 100 tasks representative of the overall spread and ran that smaller set far more often. It gives a fast, clear signal while we iterate on tools and prompts, and lets us reserve the full 400 for less frequent, higher-fidelity checks. Once the score settled into a reliable 85–90%, the benchmark's job changed. Since February this year we mostly stopped working on the agent itself: our customers needed pivot tables, charts, and other core engine work more than they needed a better benchmark score, so that's where our efforts were directed. The benchmark stayed on as a regression suite, guarding the score while we evolve engine features, tools, and skills the benchmark doesn't measure at all. The numbers Over the next few months these regression runs accumulated the following results: Model Runs Best Median claude-sonnet-5 7 91.25% 90.75% claude-opus-4-8 2 91.0% 90.5% claude-sonnet-4-6 7 90.0% 89.0% claude-sonnet-4-5 6 86.75% 85.6% The trend is really two stories. The big jump from sonnet-4-5 to sonnet-4-6 was mostly our own doing: those runs carry far more advanced GRID tooling and engine capabilities. The improvement since then is split between smaller tooling refinements and the model itself: sonnet-5 sits at 90% or above in six of its seven runs. It's worth noting that we'd probably need more than 6 or 7 runs for a given configuration of model + GRID tools to understand how well we do. A typical run fails 35–45 of the 400 tasks, but most of that is churn, the run-to-run noise we measure below. Just 14 tasks failed in all seven of the recent claude-sonnet-5 runs; put the other way, 96.5% of the verified 400 passed at least once within that window. Widen the window to every run we've recorded and only three tasks have never passed: 99.25% solved at least once, and the remaining three are arguably defective tasks, not lack of capability. A note on error bars, because leaderboards don't usually draw them. Every eval we'd built before this one was single-turn, and single-turn evals at temperature 0 are fairly stable, across both Claude and GPT models. Nothing had prepared us for how stochastic a 20-plus-turn agentic run is: early on, re-running a day's failed tasks lifted the score from 82-something to 85.7%, and a subsequent full re-run erased those gains. So we measured it properly. We ran the same agent build, configuration and model (claude-sonnet-4-6) five times over a fixed 100-task sample: 12 of the 100 tasks flipped between pass and fail across the runs. An earlier four-run group on claude-sonnet-4-5 showed the same: 14 of 100 flipped. Both groups ran with temperature=0 and top_k=1, sampling pinned to the single most likely token at every step, so the flips survive even with sampling nailed down. The take-away: a single-run score on this benchmark carries several points of noise, and comparisons between agents that differ by a point or two are mostly comparisons of luck. We're not the only ones to notice: the GPT for Work team's write-up of their own run reaches the same conclusion, attributing most of their misses to variance rather than a capability gap, with several failing tasks flipping to pass on re-runs. We standardize on Sonnet because that's the model businesses actually deploy at scale. It can be a real matter of economics: Sonnet is about 40% cheaper, on a workload as token-hungry as spreadsheets. A leaderboard doesn't record what a score costs to serve. In our controlled head-to-head with identical agent build on verified 400 and only the model swapped, Opus 4.8 scored one point higher than Sonnet 5 (91.0% vs 90.0%). A second Opus run came back at 90.0%. The noise shows that the choice has negligible effect on score. Reproducibility We're publishing our leaderboard run in full, every transcript and every output file. It lands in the same ~90% band as the rest of our runs; a single number here means little, which is rather the point. A score without its transcripts is just a claim, and we'd like publishing them to become the norm. Dealglass, whose Tetra sits ahead of us on the leaderboard, led the way here: their spreadsheet-agents dataset includes session logs for their baseline GPT-5.4 and Claude Opus 4.6 runs, alongside the eval results for their own agent. Two tasks from a recent run show what having the engine in the loop looks like, and why it matters: 48643 wants a column of accumulated wealth over 39 years: each year's deposit added to the previous year's total, grown by that year's rate. The calculation is inherently recursive, each row depends on the one above it, so there's no closed-form shortcut. Our agent evaluated the candidate formula against the live workbook first, then wrote it to the top cell and filled it down the column in one call. The engine recalculated the whole 39-row chain, and the agent read the computed values back to confirm the compounding behavior. Without an engine the agent would have to reimplement 39 rows of compounding arithmetic in Python and hope the rounding matches, or write the formulas blind. 48982 is a debugging task: a running-counter formula in B323 displays "000" and Excel complains about a circular reference, and the asker wants to know why. The display is a red herring (the cell's number format is literally 000), and the real bug is in the formula, =IF(A323="E","",MAX(B$7:B377)+1). The MAX range includes row 323 itself, so the cell feeds its own calculation. Our agent found this by reading the formula and its format, narrowed the range to B$7:B322, and evaluated the fixed cell in place to confirm that it now computed a value. A circular reference only exists, and only gets caught, when something is actually running the calculation. What the benchmark is, and isn't The good parts we covered up top: forum-sourced tasks with the underspecified shape of real work, plus expert review on the verified 400. It's the best public signal we've found for spreadsheet agent capability. But there are limitations: The reference harness assumes your agent writes Python. It generates code, runs it against the file, opens the result in Excel to recalculate, and then compares cells. That's a sensible pipeline if OpenPyXL is your only option, but unnecessary if your agent has an engine. Ours reads and edits cells directly and calculates natively. Some tasks have more than one valid solution, so getting them reliably right can mean you've overfit your prompts and tools rather than solved anything general. Only three of the 400 tasks have never passed in any of our runs, on any model. Every other task has passed at least once; most single-run failures are just tasks that fail one run and pass the next. We've shared our notes on some of these. Run-to-run volatility (see above) made us decide it's more valuable as a regression and optimization suite than anything else. That's how we use it: as the thing that tells us when an engine change broke the write path, not as a number to chase. There is also plenty a cell-comparison benchmark can't see: formatting and styles aren't scored at all, and comparing cells, or exact ranges, limits the tasks to those with a single, fixed position where the answer can be checked. SpreadsheetBench V2 addresses both, making it more representative of real spreadsheet work. That makes V2 the version of the benchmark we believe in, and the one our effort is aimed at next. Its leaderboard is young (so far it holds only the researchers' own model baselines, with the best at 34.9%), so nobody has really run at it yet. Try it for yourself The tooling layer the agent uses (the tools for reading, writing, and understanding workbooks that all these runs battle-tested) is now available as @grid-is/agent-tools. The engine itself is already available for non-commercial use, and together with our viewer and editor it's what a number of companies already license to build spreadsheet capability into their own products, increasingly for AI-agent workflows. The newest example: a financial-modeling agent one of our clients released on the engine's latest version just weeks ago. We're also strategically open-sourcing parts of the stack, more on that later. And credit is overdue. SpreadsheetBench has given us far more value than a number to put in a blog post: it has been a working development tool for half a year, and will be for the foreseeable future. Everything here is public: The agent tools: docs.grid.is/agent-tools The engine, free for non-commercial use: grid.is Our leaderboard run, transcripts and outputs: github.com/GRID-is/spreadsheetbench-results Install the tools into Claude Code or Codex: GRID skill marketplace An agent with a spreadsheet engine beats an agent without one. Now you have the tools to check.

Spreadsheet superpowers, for humans and agents.

GRID took a big step this weekend. Our engine - the most sophisticated spreadsheet engine outside of Google and Microsoft - is now free for evaluation and non-commercial use, for any developer: human or agent. We've built it from scratch over the last 8 years, grounded in deep research into how spreadsheets really work, and we license it commercially to some of the leading AI companies in the world. What it does: More deeply compatible with Excel and Google Sheets than anything else out there: Runs 99.5%+ of real-world spreadsheet models, backed by tens of thousands of compatibility tests on every build. Fast enough to recalculate even the most complex models at "animation speed". Runs wherever you need it: 100% on-premises, in your own cloud, or entirely in the browser. The setup is yours to control, and the data never leaves your environment. Other parts of the stack are free too: the viewer (near pixel-perfect rendering of any spreadsheet) and the editor (a lightweight editing layer on top). Demand at the intersection of spreadsheets and AI is strong, and software development is changing fast. Hard-gated enterprise licensing is too slow for a world where coding agents do the integrating. We'd rather let agents, and the developers directing them, integrate and prove out their ideas first, and talk commercials later. For a personal project, all we ask is attribution. Spreadsheets run the world. Now you can run them. Give it a spin or point your agent at: grid.is/engine

Spreadsheets ran the world. Then AI showed up.

I wore this shirt for years. Time to change the tense. For decades, the spreadsheet has been where the world does its thinking with numbers. It quietly became the most widely used programming language in the world, and hundreds of millions of people who would never call themselves programmers build working models in it every day. That era is ending, and not only for spreadsheets. Authoring is moving to AI across documents, slide decks, web apps, and yes - spreadsheets. Instead of starting with a blank slate, we describe what we want in plain language and let a machine draft the artifact. The handmade spreadsheet, built up cell by cell by a person, will become a rarity. Before long it will feel as quaint as hand-coding a web page in Notepad. Where the calculations go Once a machine is doing the authoring, a spreadsheet is no longer the obvious format for the response. Where the calculation ends up instead depends on the kind of task, and on how much rides on getting it right. Consumer tasks (splitting a bill, a home budget, the cost of a trip). People used to open a spreadsheet for this. Now they ask AI and get back a report or a small web app. The spreadsheet never appears at all. Operational tasks (team dashboards, sales pipelines, the startup model everyone argues over). These are moving to generated dashboards and web artifacts. A spreadsheet might sit underneath as the data source, but the logic that drives them is increasingly written as code, in Python or JavaScript, rather than in cells. High-stakes tasks (financial planning, M&A models, anything audited or regulated). Here the spreadsheet prevails, and will for a long time yet. Excel is the language of finance. The models get checked cell by cell by people who have to answer for them. Nobody trusts that to a black box. Seen this way, AI is not so much killing the spreadsheet as exposing what it is: a tool built first for the author, not for the audience. Its real gift is to the person building the model, who gets a live feedback loop where every intermediate value sits in front of them as they work. The reader is handed the finished wall of numbers with the reasoning collapsed inside it. It became the default way to deliver a calculation largely because it was what everyone already had and knew how to open, not because it was ever the best vehicle for the reader. What every calculation needs But, whatever the format, two things will always matter for any calculation: Quality. A calculation has to be correct, deterministic, verifiable and repeatable, something you can open up and inspect rather than simply believe. Legibility. A human has to be able to understand the model they are being asked to trust. As they say in finance, if you understand the spreadsheet, you understand the deal: the model is the deal, and trusting it means being able to read it. Today's AI is uneven on both counts. Ask it for a number in chat and you get neither: a confident answer, not verifiable, with no way to see how it was reached, which leaves you trusting it blindly. The moment it produces something runnable instead, a snippet of code, quality comes within reach, because the computation is deterministic and can be checked. Legibility is where it still fails. AI tends to show its work as Python or JavaScript, which almost no one outside of engineering can actually read. The spreadsheet's edge here is easy to misread. Cell for cell, a formula is no more legible than the equivalent line of code; the syntax was never the point. What it has is familiarity: orders of magnitude more people already read spreadsheets than read code, and in finance that gap is close to absolute. This doesn't make the spreadsheet inherently clearer than code. It just makes it the calculation that the most people can actually check, and that is exactly the property we are now at risk of throwing away. What comes next Calculations themselves are not going anywhere. If anything, we are about to produce far more of them than ever before. What is changing is that the spreadsheet is no longer the right format for all of them. In high-stakes finance, where every number has to be checked and someone has to answer for it, it will hold on for years yet. Everywhere else, its grip is already slipping. Familiarity is something a new format cannot inherit, but the rest of legibility, letting people follow the working and check it for themselves, is something you can design in from the start. That gap, between an explosion of machine-made calculations and our basic need to trust the numbers, is one of the largest and least-noticed openings AI is creating. Whoever works out how to make calculations that are correct, legible, and trustworthy, in whatever shape the audience needs, will build the next thing that quietly runs the world.