Does ChatGPT Keep the Numbers Right When It Summarizes? I Checked 9 Summaries

A smiling robot feeds a long page into a small machine that turns a tall stack of documents into a short bullet-point card, while a person checks the result with a magnifying glass

Written by

in

,

· Published

Summaries are one of the first things people ask a chatbot to do: a report, an email thread, a long notice. But if a summary has a wrong number in it, you may not notice until it matters. So I asked Claude to run a small test with ChatGPT: give it short texts full of figures, ask for three kinds of summary, and check every number against the original.

The short version

In 9 summaries, ChatGPT did not get a single figure wrong. The problem was different: it left things out. When I asked for 3 bullet points, it kept 96% of the facts. When I asked for one sentence, it kept 37%. In 3 of the 9 summaries it also turned an “about” estimate into a firm number.

How I tested

Claude ran the prompts on November 8, 2026, in ChatGPT without logging in, using a new chat each time. I wrote three short texts myself, about 95 words each, so I knew every fact in them. The bakery, the library and the solar company are invented. Each text was summarized three ways:

  • “Summarize this in no more than 50 words.”
  • “Summarize this in 3 bullet points.”
  • “Summarize this in one sentence.”

I counted how many of the facts in each text survived and checked each figure by hand. A few answers did not load in time, so I ran those prompts again and used the new answer. That gives 9 summaries in total, one run each.

The three texts

Text A (bakery, 18 facts): The Rivera Bakery opened on March 4, 2019 in Valencia with 6 employees. In 2021 it moved to a larger shop at Calle Mayor 12 and grew to 14 employees. Sales were €182,000 in 2022 and €231,000 in 2023, an increase of 27%. In 2023 it started selling online, which was 18% of sales by December. The owner, Marta Rivera, plans to open a second shop in Alicante in spring 2025 and hire 8 more people. The bakery’s best seller is a rye loaf with walnuts, sold at €4.20, with about 310 loaves per week.

Text B (library, 13 facts): The Northgate Community Library will close for renovation from January 13 to February 21. During the closure, the 48,000 books will stay on site, but the 2,500 most requested titles will move to the Elm Street branch, open Monday to Saturday from 9:00 to 19:00. Late fees, normally €0.25 per day, will not be charged during the closure. The renovation costs €640,000, of which the city pays 70% and the rest comes from donations. The library will reopen on February 24 with a new children’s area and 30 more study seats, bringing the total to 120.

Text C (solar company, 15 facts): Brightline Solar reported its results for 2023. Revenue fell from €4.2 million in 2022 to €3.8 million in 2023, while costs rose from €3.1 million to €3.6 million. As a result, profit dropped from €1.1 million to €0.2 million, a fall of 82%. The company sold 1,240 installations in 2023 compared with 1,510 in 2022, and its average price per installation rose from €2,780 to €3,065. CEO Daniel Ortega blamed a 9-month delay in subsidy payments and said the company will not pay dividends in 2024, although it expects revenue of about €4.5 million next year.

Results: how many facts survived

Text“No more than 50 words”“3 bullet points”“One sentence”
A (18 facts)12187
B (13 facts)8136
C (15 facts)5134
Total (46 facts)25 (54%)44 (96%)17 (37%)

The pattern is easy to see: the more room I gave, the more facts survived. The one-sentence summaries kept the “headline” (the 82% drop in profit, for example) but lost most of the rest, like the actual sales figures.

What went wrong, and what did not

  • No wrong figures: every number that appeared in a summary matched the original. That includes the 82% drop in profit in the solar text.
  • Estimates became exact: the original says “about 310 loaves” and “about €4.5 million”. Three summaries dropped the “about” and wrote 310 loaves and €4.5 million as if they were exact. The other summaries kept “approximately” or left the figure out.
  • A number I never wrote: one summary said 30% of the library’s cost came from donations. My text only said “the rest”. It is correct, but it is a figure ChatGPT worked out itself, so you would have to check to know.
  • Vague wording: one summary said “Online sales reached 18%”, without saying 18% of what. The original says 18% of total sales.
  • The word limit: for “no more than 50 words”, the three answers had 53, 42 and 44 words by my count. One went slightly over, which fits what I found in my word-limit test.

What I would do with this

  • Do not use one-sentence summaries for anything where numbers matter. They keep the story and drop the figures.
  • Three or more bullet points worked best in this test. If a fact matters, make sure it is there.
  • Check every figure against the original before you copy it anywhere, especially if the original says “about” or “approximately”.
  • Be careful with numbers the summary calculated that are not in the original. They may be right, but you cannot tell without checking.

This is the same habit I came away with from my other tests. In my test of AI references, checking was what separated the real from the invented, and in my test of repetition the instructions you give change the result.

What this test does not show

  • It is 9 summaries, one run each, on one day, with ChatGPT without an account. Another day, another model version or a paid plan could give different results.
  • The texts are short, invented and written in plain English. Long documents, PDFs, tables or other languages are a different test, and I did not run it.
  • I did not try adding “keep every number” to the prompt. That is the first thing I would test next.
  • I only tested ChatGPT, so I cannot say how Claude, Gemini or other chatbots compare.
About the author
I’m Christian Serrano, and I run AIVERSUSIA. I sell products online and use AI tools in that work every day. I test tools with the same prompts and publish what happened, including mistakes and limits. Claude helps me write articles and run tests, and I review and approve everything before it is published. More about me · Editorial policy