Weighted decision matrix with ChatGPT, recalculated in a sheet

The same notes and the same prompt, run twice, put Tool A first and then Tool B. The arithmetic was right both times (3.65 / 3.50 / 2.45 and 3.35 / 3.90 / 2.35 both recompute exactly). What changed was what the model chose: the chat-integration weight was 30% in one run and 25% in the other, and B's chat score went from 2 to 3.

Three fictional collaboration tools (A: $9, official chat plugin; B: $12, webhooks you build yourself, five permission levels; C: $7, no chat integration) were scored three ways, two runs each.

In (1) the model said so itself: "Weights and 1–5 scores (5 is best) are my assumptions, since the notes don't give priorities." That table cannot answer "who decided chat is worth 30%?" in a meeting. Its winner flipped because the model picked different weights and scores: chat weighted 30% vs 25%, B's chat score 2 vs 3, Tool A's onboarding 3 vs 2. In (2) every run put B first, but B's chat score moved between 2 and 3, so the total moved between 74 and 80 and the gap between 3 and 9 points; only the 3-point gap earned a "conditional recommendation". Only (3) gave the same numbers on all four runs (two English, two Korean), and with a 1-point gap it was labeled a conditional recommendation.

Prompt (3)

Anthropic's prompting best practices suggest ending with an instruction to verify the answer against criteria, which catches math errors well. The "Check:" line is there for that reason, but it is not what you trust.

<instructions>Score the three {options} in <notes> against the criteria and weights in <criteria>.- Use the given weights as they are. Do not change them or add criteria.- Score each item by <rubric>. Evidence is a short quote from the notes. If the notes give no basis, write NA as the score.- Total = Σ(weight × score) ÷ 5 (out of 100), one decimal place.- Output one CSV: tool,criterion,weight,score,evidence (one row per tool × criterion), then a tool,total table.- Before finishing, check that the weights sum to 100 and recompute each total from the scores; report it in one line starting with "Check:".- If the gap between first and second place is 5 points or less, write "conditional recommendation" and name the one criterion that could flip the order.</instructions> <criteria>criterion,weight{criterion},{weight}…</criteria> <rubric>price (per user per month): $7 or less 5 · $10 or less 4 · $15 or less 2{criterion}: {condition} 5 · {condition} 3 · {condition} 2…</rubric> <notes>{notes per option}</notes>

tool,totalTool B,70.0Tool A,69.0Tool C,53.0…**Conditional recommendation:** Tool B leads Tool A by only 1.0 point. Chat integration is the criterion that could flip the order. If building B's webhooks turns out to be impractical and B is scored 1 ("none") instead of 2, B drops to 64.0 and A moves into first place.

The other three runs of (3) named a different criterion each time: permissions (A from 2 to 3 gives 74.0), onboarding (A from 2 to 3 gives 71.0), and, in the second English run, no scenario that actually flips the order, only "if B's webhook build were judged equivalent to an official plugin, B's lead would grow". The arithmetic holds in the two that named one. "Name one criterion" is the part that varies, so read it.

Recompute

[1] Paste the CSV into a sheet. For each tool's five rows (weights in column C, scores in column D), compute the total:

=SUMPRODUCT(C2:C6,D2:D6)/5

[2] When comparing several runs, a script is faster. It prints the model's total next to the recomputed one.

#!/usr/bin/env python3"""usage: recalc.py out.txt — recompute weighted totals from the model's CSV and compare with its own totals"""import sys, csvfrom collections import defaultdictrows = [r for r in csv.reader(open(sys.argv[1], encoding='utf-8')) if r]score_rows = [r for r in rows if len(r) == 5 and r[2].strip().isdigit()]told = {r[0]: float(r[1]) for r in rows if len(r) == 2 and r[1].replace('.', '').isdigit()}weights, total = {}, defaultdict(float)for tool, crit, w, s, _ in score_rows:    weights[crit] = int(w)    if s.strip().upper() != 'NA':        total[tool] += int(w) * int(s) / 5print('weight sum:', sum(weights.values()))print('rows with extra commas:', sum(1 for r in rows if len(r) > 5 and r[2].strip().isdigit() and r[3].strip().isdigit()))for tool in total:    flag = 'OK ' if abs(total[tool] - told.get(tool, -1)) < 0.05 else 'DIFF'    print(f'{flag} {tool}: recomputed {total[tool]:.1f} / model {told.get(tool)}')r = sorted(total.values(), reverse=True)print(f'gap 1st-2nd: {r[0] - r[1]:.1f}')

$ python3 -I recalc.py out.p3.en.1.txtweight sum: 100rows with extra commas: 0OK  Tool A: recomputed 69.0 / model 69.0OK  Tool B: recomputed 70.0 / model 70.0OK  Tool C: recomputed 53.0 / model 53.0gap 1st-2nd: 1.0

Seven of the eight runs of (2) and (3) came back OK like this. The eighth (the second Korean run of (3)) printed DIFF:

$ python3 -I recalc.py out.p3.ko.2.txtweight sum: 100rows with extra commas: 2DIFF A: recomputed 53.0 / model 69.0OK  B: recomputed 70.0 / model 70.0DIFF C: recomputed 33.0 / model 53.0gap 1st-2nd: 17.0

The model's totals were right; the script's input was not. That run wrapped the evidence column in 「 」 instead of CSV quotes, and the commas in 9,000 and 7,000 created an extra column, so two price rows (16 points for A, 20 for C) were dropped (69−16=53, 53−20=33). That is why the script now counts rows with extra commas. When you see DIFF, look at that number before blaming the model's math. A rule telling the model to wrap the evidence column in double quotes should prevent it, but that was not tested. Run the script every time anyway. An NA adds nothing to that tool's total, so a low total can mean missing notes rather than a weak option.

[3] Read the evidence column against the notes. Without a rubric, as in (2), the model invents its own scale. Both English runs gave email-only onboarding a 3 and docs-and-videos a 2; both Korean runs gave them 2 and 3. If the column cites something the notes never said (contract terms, security certifications), delete that row and ask again. A scale like that is yours to set, as the <rubric> in (3).