Make Claude quote its sources, then check the quotes with grep
Ask Claude for a summary with a verbatim quote per sentence, confirm each quote exists in the source with grep -F, and re-ask only for the ones that fail.
Require a verbatim quote on every summary sentence, and check with grep -F that each quote exists in the source.
[1] Cut one section out of a public document. This uses RFC 9110 sections 9.3.1 (GET) and 9.3.2 (HEAD), 105 lines.
$ curl -s https://www.rfc-editor.org/rfc/rfc9110.txt -o rfc9110.txt$ grep -n '^9.3.1\.\|^9.3.3\.' rfc9110.txt3920:9.3.1. GET4025:9.3.3. POST$ sed -n '3920,4024p' rfc9110.txt > doc.txt$ wc -lc doc.txt 105 5854 doc.txt[2] Write the prompt. Require quotes copied character for character, 80 characters or fewer, and fix the output format so the quotes can be extracted.
Summarize the document below in 5 bullet points. Rules:- Each bullet is one sentence, followed by a quote that supports it.- The quote must be copied character for character from the document, 80 characters or fewer. Do not fix spelling, do not use "...".- Output one line per bullet, in exactly this format: <summary sentence> || <quote>- No other text. <document>[3] Append the document and run claude -p. --tools "" turns tools off so the model sees only the pasted document.
$ (cat head.txt doc.txt; echo "</document>") > prompt.txt$ claude -p --model claude-sonnet-5-5 --tools "" < prompt.txt > out.txt$ cat out.txtGET requests a current selected representation of the target resource, and a successful response reflects the "sameness" identified by the target URI. || The GET method requests transfer of a current selected representation GET is the primary mechanism of information retrieval and the focus of nearly all performance optimizations. || GET is the primary mechanism of information retrieval and the focus A resource identifier need not map to a file, since resources may be content trees, database views, or gateways to other systems. || as a tree of content objects, a programmatic view on various database Clients should not send content in a GET request because it has no defined semantics and could be rejected as a possible request smuggling attack. || A client SHOULD NOT generate content in a GET HEAD is identical to GET except the server must not send content, and it is used to obtain metadata about the selected representation. || The HEAD method is identical to GET except that the server MUST NOT[4] Write the check script. RFC text wraps sentences at line ends, so collapse whitespace and newlines in the source to single spaces, then search with grep -F (literal match, not a regex).
#!/bin/bash# usage: verify.sh doc.txt out.txttr -s ' \n' ' ' < "$1" > /tmp/flat.$$n=0; miss=0while IFS= read -r l; do [ -z "$l" ] && continue n=$((n+1)); q=${l#* || } grep -qF -- "$q" /tmp/flat.$$ || { miss=$((miss+1)); echo "MISS $n: $q"; }done < "$2"rm -f /tmp/flat.$$echo "$n quotes, $miss missing"; [ "$miss" -eq 0 ][5] Run the check. All five quotes were in the source. A failure prints MISS and exits 1. The failing run uses out-bad.txt: the fourth quote of out.txt edited by hand from SHOULD NOT to MUST NOT (the model did not invent it).
$ chmod +x verify.sh$ ./verify.sh doc.txt out.txt; echo "exit $?"5 quotes, 0 missingexit 0$ ./verify.sh doc.txt out-bad.txt; echo "exit $?"MISS 4: A client MUST NOT generate content in a GET5 quotes, 1 missingexit 1[6] Re-ask only for the bullet whose quote is missing. Send the failing line and the document, and allow NO SUPPORT as an answer.
Your earlier answer contained a quote that does not appear in the document. grep -F could not find it: Clients should not send content in a GET request because it has no defined semantics and could be rejected as a possible request smuggling attack. || A client MUST NOT generate content in a GET Fix only that bullet. Find the sentence in the document below that supports it. If none does, reply with exactly: <summary sentence> || NO SUPPORTOtherwise reply with one line, same format as before:<summary sentence> || <quote copied character for character, 80 characters or fewer>No other text. <document>(the 105 lines of doc.txt)</document>[7] Run the reply through the same check.
$ claude -p --model claude-sonnet-5-5 --tools "" < prompt-fix.txt > fix.txt$ cat fix.txtClients should not send content in a GET request unless the origin server has indicated it will support it. || A client SHOULD NOT generate content in a GET$ ./verify.sh doc.txt fix.txt; echo "exit $?"1 quotes, 0 missingexit 0