Numbers don’t lie — but AI might

Large language models (LLMs) sometimes generate information that sounds convincing, but is completely made-up or incorrect. In AI terms, this is called a hallucination. An LLM might confidently state a fact or number that has no basis in reality — without any indication of the answer being made up. This can make it hard for users to realize when they’re reading (or worse, trusting) incorrect information. Businesses need to be aware of this risk when they‘re relying on factual details or calculations in their AI projects: Just because an LLM’s answer is detailed and confident, that doesn’t guarantee it’s accurate. This is especially true when it comes to numbers, calculations and spreadsheet logic — areas where LLMs are notoriously unreliable.

How common are these hallucinations?

Studies show that when LLMs like GPT-4o and Claude-3.5 are asked short, factual-based questions (pub quiz style), they can hallucinate up to 60% of their answers. On the other hand, when the models are given a task with context, for example summarizing a document, GPT-4o only hallucinated 1.5% of the output. This shows that giving an LLM relevant context to work from can greatly reduce the risk of hallucinations, though it does not eliminate hallucinations entirely.

The research mentioned focuses purely on text — but what about numbers? A study found that modern LLMs fail frequently on many basic numerical problems, reasoning through a math problem but then confidently outputting an incorrect result. The study showed that for easy cases, like adding two small numbers together, the models provided correct results with near 100% accuracy. As soon as the tasks became slightly more complex, with larger numbers and decimals for example, accuracy quickly decreased and the vast majority of answers were hallucinated results.

Text vs. numbers

The thing is: we might be asking too much of a language model. At their core, LLMs predict linguistic patterns. The fundamental operation of GPT models such as those running on ChatGPT and Claude can — as this study elegantly puts it — be distilled down to answering a single question:

Given a sequence of words, what word is likely to come next?

Why should an LLM excel in calculations, given how they’re built?

It’s clear that providing context significantly reduces the risk of hallucinations in text-based answers from LLMs. If you’re working on a AI project, you’ve probably implemented contextual grounding and made good progress on keeping hallucinations at bay. But what about numbers? Calculations play a critical role across various industries — whether you’re involved in finance, insurance, e-commerce or SaaS, numbers are integral to your daily operations. However, relying solely on an LLM means that including calculations in your AI projects is off the table. Fortunately, modern LLMs support function calls or tool integrations, enabling them to leverage external tools. At GRID, we’ve made it possible to integrate our spreadsheet engine to LLMs, unlocking the possibility for LLMs to offer reliable calculations — something that’s not possible without GRID today.

By leaving the language to the language models, and the calculations to a spreadsheet engine, you reduce the risk of hallucinations drastically.

How can the calculations be trusted?

When using GRID’s spreadsheet engine in your AI projects, there are two key things to get right to ensure your calculations are reliable:

1. The calculations you’re asking for exist in a spreadsheet.

2. In your prompt, make sure the LLM never answers without asking the spreadsheet for the calculations — meaning, the LLM doesn’t attempt to calculate itself.

If you ensure that these two things are in place, you can trust the calculated output the LLM provides. The good thing is, while you’re testing the integration, you can always verify that the calculations are correct by going back to your spreadsheet.

But wait — can’t LLMs calculate using code?

If you’re an engineer (or someone that can write and understand code) and you’re building your AI project from scratch, coding the calculations is also an option. Instead of relying on the LLM’s internal math reasoning, developers can prompt the LLM to create executable code, and then run these calculations through an external computational environment to ensure accuracy.

For those that are not in a position to write the calculations in code, using a spreadsheet instead is now an option thanks to GRID. If the calculations you want to integrate to your AI project are complex, or you already have a spreadsheet with your calculations, GRID is the answer.

With GRID, you can turn your spreadsheets to RESTful APIs and integrate them to your AI projects. There’s no need to leave out calculations in your AI projects because LLMs can’t calculate — you just have to use the right tools to make sure your calculations are reliable and verifiable.

Is your team building AI products? We’d love to hear from you at [email protected] 👋🏻