Classify product reviews by theme with an LLM, then recount
Get one line of tags per review from the model, count in a script, check for missing IDs, and fix the tag definition behind the mismatches.
Paste 200 reviews and ask for "the top complaints", and you get a ranking with rough percentages that nobody can recount. Ask the model for one line of tags per review instead, and let a script do the counting.
Data
200 fictional wireless earbud reviews in a three-column TSV, id, stars, text (for example R003 1 Tips are too big for my ears.). They were generated by a script, so the correct tags for every review are known.
Prompt
Instructions, tag list, examples and reviews each sit in their own XML tag. Anthropic's prompting best practices call a few examples the most reliable way to fix output format and structure. Example IDs start with X so any that leak into the output stand out.
<instructions>Tag every {product} review inside <reviews>, one line per review, skipping none.- Use only tags from <tags>. List every tag that applies (up to 3), comma-separated.- Sentiment is one of positive, negative, mixed. Praise and complaint together means mixed.- Output in input order, one line per review: ID|tags|sentiment- No totals, summary, explanation or header line.</instructions> <tags>sound: anything about audio, bass, highs, volumebattery: listening time, charging, case chargingfit: ear pain, falling out, ear tip sizeshipping: delivery time, packaging conditionprice: cost, discounts, value for moneyother: anything not covered above (color, app, manual, etc.)</tags> <examples>X01|fit|negative <- "My ears throb after wearing them a while."X02|battery,price|mixed <- "Battery lasts long but they are a bit pricey."X03|other|positive <- "Love the case color."</examples> <reviews>{review TSV}</reviews>$ (cat head.en.txt reviews.en.tsv; echo "</reviews>") > prompt.en.txt$ claude -p --model claude-sonnet-5-5 --max-turns 1 --tools "" --setting-sources "" --strict-mcp-config < prompt.en.txt > out.en.1.txt$ head -3 out.en.1.txtR001|shipping,sound|mixedR002|battery|positiveR003|fit|negativeThree runs gave the same 200 tag sets; the second run only listed price and sound in the other order on seven lines.
Recount
check_tags.py uses only the standard library. It compares input and output IDs as sets, counts tags and sentiment, and, given a third file with known answers, reports agreement on 20 random IDs and on all of them. With real reviews, read 20 by hand and save them in the same format.
#!/usr/bin/env python3"""usage: check_tags.py reviews.tsv out.txt [truth.tsv]1) ID set: missing / extra / duplicated 2) count per tag and sentiment 3) agreement on 20 random IDs"""import sys, csv, randomfrom collections import Counterids = [r['id'] for r in csv.DictReader(open(sys.argv[1], encoding='utf-8'), delimiter='\t')]lines = [l.strip() for l in open(sys.argv[2], encoding='utf-8') if l.count('|') == 2]got = {}dup = []for l in lines: i, tags, sent = [x.strip() for x in l.split('|')] if i in got: dup.append(i) got[i] = (set(t.strip() for t in tags.split(',')), sent)print(f'input {len(ids)} / output lines {len(lines)}')print('missing :', sorted(set(ids) - set(got)) or '-')print('extra :', sorted(set(got) - set(ids)) or '-')print('dup :', dup or '-')tc = Counter(t for tags, _ in got.values() for t in tags)sc = Counter(s for _, s in got.values())print('tags :', ', '.join(f'{k} {v}' for k, v in tc.most_common()))print('sentiment:', ', '.join(f'{k} {v}' for k, v in sc.most_common()))if len(sys.argv) > 3: truth = {r['id']: (set(r['tags'].split(',')), r['sentiment']) for r in csv.DictReader(open(sys.argv[3], encoding='utf-8'), delimiter='\t')} sample = random.Random(1).sample(ids, 20) tag_ok = sum(got.get(i, (set(), ''))[0] == truth[i][0] for i in sample) sent_ok = sum(got.get(i, (set(), ''))[1] == truth[i][1] for i in sample) print(f'sample 20: tags match {tag_ok}/20, sentiment match {sent_ok}/20') bad = [i for i in ids if got.get(i, (set(), ''))[0] != truth[i][0]] print(f'all 200 : tag mismatch {len(bad)}', bad[:8])$ python3 -I check_tags.py reviews.en.tsv out.en.1.txt truth.en.tsvinput 200 / output lines 200missing : -extra : -dup : -tags : sound 75, battery 46, fit 43, price 43, shipping 41, other 12sentiment: negative 128, positive 51, mixed 21sample 20: tags match 19/20, sentiment match 20/20all 200 : tag mismatch 6 ['R070', 'R093', 'R110', 'R123', 'R128', 'R195']Nothing was dropped and sentiment counts matched the answers. Price came out at 43 against 37 correct. The 20-review sample caught only one of the six.
The six mismatches were one sentence
All six contain the same line, and the model added price to it.
$ grep -E '^(R093|R070)' reviews.en.tsv out.en.1.txtreviews.en.tsv:R070 3 Took over a week to arrive. For this price the sound is plenty.reviews.en.tsv:R093 5 For this price the sound is plenty.out.en.1.txt:R070|shipping,sound,price|mixedout.en.1.txt:R093|sound,price|positiveThat is praise for the sound, with price as the yardstick. The tag list never said what to do with it: the original line was price: cost, discounts, value for money, and "for this price" reads like value for money, so price got tagged. The first fix only said what to leave out:
price: judging the cost or a discount itself. Do not add it when price is only the yardstick for another point, as in "for this price the sound is plenty"…all 200 : tag mismatch 0 []$ grep -E '^(R070|R093)' out.en.v2.txtR070|shipping,sound|mixedR093|sound|positiveThat was enough in this run: price is gone from the six reviews and sound stayed. A rule that only says what to leave out can also strip the other point, though, so a second version states where the tag goes instead:
price: judging the cost or a discount itself. When price is only the yardstick for another point, as in "for this price the sound is plenty", tag that other point (sound) and not price…tags : sound 75, battery 46, fit 43, shipping 41, price 37, other 12sentiment: negative 128, positive 51, mixed 21sample 20: tags match 20/20, sentiment match 20/20all 200 : tag mismatch 0 []This second wording also gave 0 mismatches. When mismatches show up, look for a shared sentence and fix the definition rather than patching rows one by one.