Sort Your Wrong Answers into an Error-Type Table

Paste eight missed questions into an AI and it re-solves them first. If it solves one wrong, it names the wrong cause. Give it the correct answer, the student's answer and one line of the student's working, and ask for the cause only.

[1] Build the input table: problem, correct answer, student answer, one line of working. The problems are made up.

No | Problem | Correct | Student answer | Student working1 | Solve 3x − 5 = 10 | x=5 | x=5/3 | 3x=10−5, x=5/32 | Evaluate (−3)² − 2×(−4) | 17 | −1 | (−3)²=−9, −9+8=−13 | Perimeter of a rectangle 8 cm by 5 cm | 26 cm | 40 cm | 8×5=404 | y=2x+3. Find y when x=−2 | −1 | 7 | 2×(−2)=4, 4+3=75 | Three consecutive whole numbers sum to 51. Find the largest | 18 | 17 | 51÷3=176 | Least common multiple of 12 and 18 | 36 | 6 | 12=2×2×3, 18=2×3×3, shared 2×3=67 | Simplify 2(x+3) − (x−4) | x+10 | x+2 | 2x+6−x−4=x+28 | Distance walked in 30 minutes at 4 km/h | 2 km | 120 km | 4×30=120

[2] Write the prompt. Each of the three cause names gets a one-line definition, plus one example row that is not in the data. OpenAI's prompt engineering guide describes the two ideas used here: show the classification you want with an input/output example, and put reference material directly in the prompt. The "Quote" column exists for step 5.

Below is a table of the problems a student got wrong on a practice test. Classify why each one was missed. Rules:- Take the correct answer and the student's answer as given. Do not solve the problems again.- Choose exactly one cause per problem from: calculation slip / concept gap / missed condition.  - calculation slip: the method is right, but the arithmetic or a sign slipped.  - concept gap: the student used a wrong rule or mixed up two concepts.  - missed condition: the student did not answer what was asked (misread the question, ignored a unit or a condition).- In "Quote", copy a piece of the student's working line character for character. It must be a substring of that line.- "Next rule" is one short instruction the student can follow next time, specific to this problem. Never leave it empty.- Output two Markdown tables and nothing else.  Table 1 columns: No | Cause | Quote | Next rule  Table 2 columns: Cause | Count Example (not part of the data below):| No | Cause | Quote | Next rule || 0 | calculation slip | 7×6=48 | Divide the product back to check it | <wrong_answers>

[3] Append the data and the closing tag, then run it. --tools "" turns off built-in tools, --strict-mcp-config drops connected MCP tools, and --setting-sources "" skips your settings and CLAUDE.md files.

$ (cat head.en.txt data.en.txt; echo "</wrong_answers>") > prompt.en.txt$ claude -p --model claude-sonnet-5-5 --max-turns 1 --tools "" --setting-sources "" --strict-mcp-config < prompt.en.txt > out.en.1.txt$ cat out.en.1.txt| No | Cause | Quote | Next rule ||---|---|---|---|| 1 | concept gap | 3x=10−5 | Move −5 across the equals sign as +5, so write 3x=10+5 and then divide by 3. || 2 | concept gap | (−3)²=−9 | Square a negative number as (−3)×(−3)=+9 before you do the subtraction. || 3 | concept gap | 8×5=40 | Perimeter is the distance around the shape, so add all four sides: 8+5+8+5. || 4 | calculation slip | 2×(−2)=4 | Check the sign of a positive times a negative: 2×(−2)=−4. || 5 | missed condition | 51÷3=17 | After finding the middle number 17, reread the question and give the largest, which is 17+1. || 6 | concept gap | shared 2×3=6 | For the LCM, take every prime at its highest power (2×2×3×3), not only the shared ones. || 7 | calculation slip | 2x+6−x−4=x+2 | Change the sign of both terms inside the bracket: −(x−4) becomes −x+4. || 8 | missed condition | 4×30=120 | Convert 30 minutes to 0.5 hours before you multiply by 4 km/h. | | Cause | Count ||---|---|| calculation slip | 2 || concept gap | 4 || missed condition | 2 |

A second run differed on row 1 (calculation slip instead of concept gap, so the counts were 3/3/2); the other seven rows matched on cause. Most next rules differed only in wording, but row 1 in that run said "Put 15 back into 3x − 5 to check", which is wrong: the answer is x=5, and putting in 15 gives 40, not 10. The model itself gave a wrong check instruction, so read each next rule as maths too, not only the cause column.

[4] Write a checker. It tests three things: each quote appears, character for character, in that row's working; no "Next rule" is empty; the counts in table 2 add up to the number of input rows (8) and match the causes in table 1. The quote lookup works like the one in summary with quotes.

#!/usr/bin/env python3"""usage: check.py data.txt out.txt"""import syswork = {l.split(" | ")[0]: l for l in open(sys.argv[1], encoding="utf-8").read().splitlines()[1:]}tabs, cur = [], []for l in open(sys.argv[2], encoding="utf-8").read().splitlines() + [""]:    if l.startswith("|"): cur.append([c.strip() for c in l.strip("|").split("|")])    elif cur: tabs.append([r for r in cur[1:] if not set("".join(r)) <= set("-: ")]); cur = []t1, t2 = tabs[:2]bad = 0for n, cause, quote, rule in t1:    ok = quote in work.get(n, "").split(" | ")[-1] and rule != ""    bad += not ok    print(("OK  " if ok else "BAD ") + f"{n}: {cause} / {quote}")cnt = {c: sum(r[1] == c for r in t1) for c, _ in t2}total = sum(int(k) for _, k in t2)print(f"rows {len(t1)}/{len(work)}, table2 total {total}, matches table1: {cnt == {c: int(k) for c, k in t2}}, bad {bad}")sys.exit(0 if len(t1) == len(work) == total and cnt == {c: int(k) for c, k in t2} and not bad else 1)

[5] Run it. Both runs exit 0.

$ python3 -I check.py data.en.txt out.en.1.txt | tail -1; echo "exit ${PIPESTATUS[0]}"rows 8/8, table2 total 8, matches table1: True, bad 0exit 0

For a failure, the quote in row 4 of the Korean-run output was edited by hand (the model did not make this error; the check itself is language-independent):

$ python3 -I check.py data.ko.txt out-bad.ko.txt | grep -v '^OK'; echo "exit ${PIPESTATUS[0]}"BAD 4: 계산 실수 / 2×(−2)=−4rows 8/8, table2 total 8, matches table1: True, bad 1exit 1

[6] A passing script does not mean the labels are right. Look at row 3: the question asks for the perimeter and the student computed the area, so "missed condition" fits well. In the Korean-language runs the model called it a concept gap both times, because the two definitions overlap. In the English runs it did the same (concept gap, twice).

Add one rule:

- If the student computed a different quantity from the one asked (for example area instead of perimeter), choose missed condition, not concept gap.

This fix was run on the Korean prompt only (two runs). Row 3 became "missed condition" both times, but other rows moved too:

The count table was 3/2/3 after run 1 and 1/4/3 after run 2 (calculation slip / concept gap / missed condition); the first two Korean runs gave 3/3/2 and 2/4/2. Every one still totals 8. The total matched while the contents of the cells changed, and row 1 split between runs even before the rule was added.

So two jobs stay with you. Run the same prompt twice and read the rows where the cause differs first: check the working line against the correct answer. Rows where both runs agree still need a look at the student's working, the cause label and the next rule, because the same model agreeing with itself is not proof (row 3 got the same label twice and still needed a new rule). Then rewrite each "Next rule" into something the student can actually do. "Be careful" is not a rule.