Recompute the averages and rates an AI stated from a CSV
Take an AI's mean, median and rates back as JSON, then recompute the same keys with pandas. Covers a rounding tolerance and re-asking only wrong figures.
An AI says "mean 126.54, return rate 26.7%" and the only way to know is to calculate it yourself. Get the figures as JSON with fixed keys, let pandas recompute the same keys, and flag the ones that disagree. Checking citations and textual claims is covered in citation verification and the claim–evidence table; this post is about numbers only.
[1] Make a test CSV: 60 orders (order id, region, amount, returned flag) from random numbers. The seed is fixed, so you get the same file.
import csv, randomrandom.seed(7)rows = [[f"A{1000+i}", random.choice(["north", "south", "east", "west"]), round(random.uniform(12, 240), 2), random.choice(["yes", "no", "no", "no"])] for i in range(1, 61)]with open("orders.csv", "w", newline="") as f: w = csv.writer(f) w.writerow(["order_id", "region", "amount", "returned"]) w.writerows(rows)$ python3 gen.py$ head -4 orders.csvorder_id,region,amount,returnedA1001,east,228.11,noA1002,north,28.52,yesA1003,east,144.88,no[2] Install pandas in a virtual environment for the check.
$ python3 -m venv .venv$ .venv/bin/pip install -q pandas$ .venv/bin/python -c "import pandas; print(pandas.__version__)"3.0.6[3] Write the prompt. Pin the key names, state the rounding, ask for one JSON object. The stated rounding is what lets you set the tolerance in step 5.
Analyze the CSV inside <data>. Do not use any tools. Report these figures as a single JSON object and nothing else. Use these keys exactly:rows, mean_amount, median_amount, max_amount, returned_pct, mean_amount_north, mean_amount_south, mean_amount_east, mean_amount_west, total_amount returned_pct is the percentage of rows where returned is "yes". Round money to 2 decimals and percentages to 1 decimal. <data>[4] Append the CSV and run. --tools "" turns tools off so the model cannot run code. The point of the test is the numbers the model states from the text alone.
$ (cat head.txt orders.csv; echo "</data>") > prompt.txt$ claude -p --model claude-sonnet-5-5 --max-turns 1 --tools "" < prompt.txt > out1.txt$ cat out1.txt{"rows":60,"mean_amount":126.54,"median_amount":139.33,"max_amount":238.43,"returned_pct":26.7,"mean_amount_north":115.54,"mean_amount_south":126.5,"mean_amount_east":128.69,"mean_amount_west":135.55,"total_amount":7592.57}[5] Write the check script. For every key in the answer it computes the true value with pandas and passes the stated value if it is within half a unit of the last digit. Do not round the true value first and compare with ==: when the true value is exactly halfway, such as 31.25, both 31.2 and 31.3 are valid roundings.
#!/usr/bin/env python3"""usage: recheck.py data.csv answer.json — recompute each stated figure with pandas and compare"""import json, sysimport pandas as pd df = pd.read_csv(sys.argv[1])stated = json.load(open(sys.argv[2], encoding="utf-8"))yes = df["returned"] == "yes" actual = { "rows": len(df), "mean_amount": df["amount"].mean(), "median_amount": df["amount"].median(), "max_amount": df["amount"].max(), "returned_pct": yes.mean() * 100, "total_amount": df["amount"].sum(), "std_amount": df["amount"].std(), # pandas default is n-1 "p90_amount": df["amount"].quantile(0.9), # linear interpolation "mean_amount_returned": df.loc[yes, "amount"].mean(), "mean_amount_not_returned": df.loc[~yes, "amount"].mean(), "amount_over_200_count": int((df["amount"] > 200).sum()),}for region, g in df.groupby("region"): actual[f"mean_amount_{region}"] = g["amount"].mean() actual[f"returned_pct_{region}"] = (g["returned"] == "yes").mean() * 100 DIGITS = {"rows": 0, "amount_over_200_count": 0}DIGITS.update({k: 1 for k in actual if "pct" in k})bad = 0for key, got in stated.items(): if key not in actual: print(f"UNKNOWN {key}"); bad += 1; continue nd = DIGITS.get(key, 2) want = round(actual[key], nd) ok = abs(got - actual[key]) <= 0.5 * 10 ** -nd + 1e-9 # within half a unit of the last digit bad += not ok print(f"{'OK ' if ok else 'DIFF'} {key:<26} stated={got:<9} actual={want}")print(f"{len(stated)} figures, {bad} differ")sys.exit(1 if bad else 0)[6] Run it. All ten figures match.
$ .venv/bin/python -I recheck.py orders.csv out1.txt; echo exit $?OK rows stated=60 actual=60OK mean_amount stated=126.54 actual=126.54OK median_amount stated=139.33 actual=139.33OK max_amount stated=238.43 actual=238.43OK returned_pct stated=26.7 actual=26.7OK mean_amount_north stated=115.54 actual=115.54OK mean_amount_south stated=126.5 actual=126.5OK mean_amount_east stated=128.69 actual=128.69OK mean_amount_west stated=135.55 actual=135.55OK total_amount stated=7592.57 actual=7592.5710 figures, 0 differexit 0A second prompt on the same file asked for the standard deviation, 90th percentile, mean by returned flag, return rate per region and the count over 200 (out2.txt). Of nine figures, only returned_pct_east differed: the model said 31.3, pandas rounded to 31.2.
$ .venv/bin/python -I -c "import pandas as pd; d=pd.read_csv('orders.csv'); e=d[d.region=='east']; print(len(e), (e.returned=='yes').sum(), (e.returned=='yes').mean()*100)"16 5 31.255 of 16 is 31.25. Python's round(31.25, 1) gives 31.2, but 31.3 is not a wrong answer. Because the tolerance is half a unit, the script in step 5 marks this line OK (all nine pass). Rounding the true value first and comparing with == would report it as a false alarm.
[7] Check that a wrong answer produces DIFF. This uses out3.txt, the answer for a 300-row file (orders300.csv, same generator with seed 11), with two values edited by hand. The errors are not the model's.
$ sed 's/"mean_amount_south":109.78/"mean_amount_south":105.78/;s/"returned_pct":25.3/"returned_pct":27.3/' out3.txt > out3-bad.json$ .venv/bin/python -I recheck.py orders300.csv out3-bad.json | grep -v '^OK'; echo exit ${PIPESTATUS[0]}DIFF returned_pct stated=27.3 actual=25.3DIFF mean_amount_south stated=105.78 actual=109.7810 figures, 2 differexit 1[8] Re-ask only the keys that disagree. Do not hand over the script's true values. Say only that another tool disagrees, and have the model recompute those keys from scratch. If you give it the answer you cannot tell whether it recomputed or copied.
Recompute only these figures from the CSV inside <data>, from scratch. Do not use any tools.A separate script disagrees with an earlier answer for them, so do not reuse that answer.Output a single JSON object with exactly these keys and nothing else: mean_amount_south, returned_pctRound money to 2 decimals and percentages to 1 decimal. <data>$ (cat reask-head.txt orders300.csv; echo "</data>") > prompt4.txt$ claude -p --model claude-sonnet-5-5 --max-turns 1 --tools "" < prompt4.txt > out4.txt$ cat out4.txt{"mean_amount_south": 109.78, "returned_pct": 25.3}$ .venv/bin/python -I recheck.py orders300.csv out4.txt; echo exit $?OK mean_amount_south stated=109.78 actual=109.78OK returned_pct stated=25.3 actual=25.32 figures, 0 differexit 0Check the re-asked answer with the same script. If it still shows DIFF, stop asking the model and use the pandas value.