Draft a complaint reply that answers every point raised
An AI draft for an angry customer email skips one demand. Extract the demands as quotes, then check each is answered and each promise is on your allowed list.
The draft apologizes politely and never mentions the customer's "if I get no reply within 3 days". List the demands first, make the reply tag every answer with the demand it belongs to, and the gap shows. The quote check works like the one in summary quote verification.
[1] Input: a complaint email to a made-up furniture shop, and the list of actions the shop can take. The demands are buried in paragraphs (context lines shortened with …).
Subject: A week late, and what arrived has a broken leg … The promised delivery date was October 2, so it is ten days late, and I posted twice on your help board without any answer. When I opened the box, the lower right leg was snapped off. I have attached photos. … I have three requests. First, send me a new, undamaged bookcase instead of this one. Second, refund half the product price for being ten days late. Third, whether it is a return or an exchange, arrange the pickup yourselves, because I will not take another day off work. Text me the visit date in advance. … If I get no reply to this email within 3 days, I will ask my card company to cancel the payment."Three requests" is really five lines once you count the text message and the 3-day deadline. The allowed list is built to disagree with the customer on purpose.
[Allowed actions - make no promise outside this list]1. Damage confirmed: ship a replacement. Dispatch within 5 business days of the claim.2. Late-delivery compensation: store credit worth 10% of the product price (not a cash refund). Capped at 10% regardless of days late.3. Partial refund: only when the product has a flaw and the customer says they will keep it. Up to 30% of the product price.4. Pickup: with a replacement, the courier delivering the new item collects the damaged one the same day. A separate pickup visit is booked as a morning or afternoon window on a weekday, and someone must be home (no exact arrival time can be promised).5. Reply deadline: the person in charge replies directly within 1 business day.6. Cancelling a card payment is between the customer and the card company; we do not block it.[2] The prompt has two stages: pull the demands out as quotes, then tag each answer in the reply with [R1] and so on. Below is the final version; the first version that was run lacked the four ★ lines.
You are a customer-support writer for an online furniture shop. Work in two stages, in this order. Stage 1 - DEMANDS. List every request or demand the customer makes in <mail>. One line each: `R1 | "<quote>"`. The quote must be copied character for character from the mail. ★Include any deadline or ultimatum the customer sets. Do not list complaints that ask for nothing. Stage 2 - REPLY. Write the reply in English to the customer. Rules:- Answer each demand Rn in at least one sentence that ends with the tag [Rn]. If the answer is "we cannot", say so plainly and offer what the list allows.- Every action, amount, percentage or deadline you promise must come from <allowed>. Do not invent anything else (no compensation, no phone call, no time slot).- ★Use an item of <allowed> only if it fits the customer's situation as written (for example, a partial refund only if the customer says they keep the product).- ★If a demand is not covered by <allowed> at all, do not say yes or no. Write that the person in charge will confirm it in the reply within the deadline given in <allowed>.- ★Speak as "we", not as an individual.- One short apology, at the top. Do not repeat it.- Tone: calm, specific. No promises about what you "will try". Output exactly two parts: a line `== DEMANDS ==` with the list, then a line `== REPLY ==` with the reply. No other text.[3] Append the allowed list and the mail, then run. Feed the prompt on stdin.
$ (cat head.txt; echo "<allowed>"; cat policy.en.txt; echo "</allowed>"; echo "<mail>"; cat mail.en.txt; echo "</mail>") > prompt.en.txt$ claude -p --model claude-sonnet-5-5 --max-turns 1 --tools "" --setting-sources "" --strict-mcp-config < prompt.en.txt > out.en.txt[4] The check script. It confirms each quote appears once in the mail, each listed demand has an [Rn] sentence, and every N% or N days in the reply also appears in the allowed list.
import re, sysnorm = lambda s: " ".join(s.split())mail, policy = (norm(open(p, encoding="utf-8").read()) for p in sys.argv[1:3])dem, _, rep = open(sys.argv[3], encoding="utf-8").read().partition("== REPLY ==")bad, ids = 0, []for m in re.finditer(r'^(R\d+)\s*\|\s*"(.+)"\s*$', dem, re.M): ids.append(m.group(1)); n = mail.count(norm(m.group(2))); bad += n != 1 print("QUOTE", m.group(1), "OK" if n == 1 else f"FAIL({n}x)")for rid in ids: n = rep.count(f"[{rid}]"); bad += n == 0 print("REPLY", rid, "OK" if n else "EMPTY")for x in sorted(set(re.findall(r"\d+\s*(?:%|days?|business days?)", rep))): ok = x.replace(" ", "") in policy.replace(" ", ""); bad += not ok print("PROMISE", x, "in policy" if ok else "NOT IN POLICY")print("problems:", bad); sys.exit(1 if bad else 0)[5] The final prompt, run twice, listed all five demands both times, and both runs printed problems: 0. Part of the first reply:
We cannot refund half the product price for the late delivery. Late-delivery compensation is store credit worth 10% of the product price, not a cash refund, and it stays at 10% however many days late the order was. We offer you that 10% store credit. [R2] We cannot say yet whether we can text you the visit date in advance. The person in charge will confirm this in their reply to you within 1 business day. [R4] The person in charge will reply to you directly within 1 business day. Cancelling a card payment is between you and your card company, and we do not block it. [R5]The script only checks inside the list the model wrote. If the model drops the 3-day line from the list, nothing is missing as far as the script can tell, so count the demands in the mail yourself first. The English version was not run without the ★ lines, so there is no English before-and-after. The Korean first version, run twice, dropped the 3-day line both times and invented a refusal ("we cannot promise a text message") that is not on the list; in one run it also offered a 30% partial refund if the customer kept the damaged bookcase, to a customer who had asked for a replacement.
[6] To see the check fail, break a copy by hand: delete an [R3] sentence, change "5 business days" to "2 business days", and edit a quote. This is not model output.
$ python3 -I check.py mail.en.txt policy.en.txt out-bad.txt | grep -E 'FAIL|EMPTY|NOT|problems'; echo "exit ${PIPESTATUS[0]}"QUOTE R1 FAIL(0x)REPLY R3 EMPTYPROMISE 2 business days NOT IN POLICYproblems: 3exit 1Strip the tags before sending: sed 's/ *\[R[0-9]*\]//g' out.en.txt.