Count AI Crawler Requests in an Nginx Access Log

2026-10-19·GNU awk 5.2.1 · Nginx combined-format sample log · jq 1.7 · vendor docs as of 2026-10·6 steps

Count GPTBot, ClaudeBot and PerplexityBot hits by User-Agent, check robots.txt status, then verify the source IPs against the vendors' published lists.

The last quoted field of a combined log line is the User-Agent, and AI bots put their name there. The output below comes from a hand-made combined log for 2026-10-17 (197 lines, clients in the documentation ranges 203.0.113.x, 198.51.100.x and 192.0.2.x). The path is shortened to access.log; on a real server it is usually /var/log/nginx/access.log. For the basics of counting status codes and paths with awk, see this post.

[1] Pick the names to look for. These are the User-Agent names from the vendors' documentation.

Vendor	Name	Purpose (per the docs)
OpenAI	`GPTBot`	Crawls content that may be used to train generative AI models
OpenAI	`OAI-SearchBot`	Surfaces sites in ChatGPT search
OpenAI	`ChatGPT-User`	User-initiated actions such as opening a page; not an automatic crawler
Anthropic	`ClaudeBot`	Collects web content for model development
Anthropic	`Claude-User`	Fetches pages when a user asks Claude a question
Anthropic	`Claude-SearchBot`	Improves search result quality
Perplexity	`PerplexityBot` · `Perplexity-User`	Search indexing · user requests

Google-Extended is not in the list. Google's documentation says it "doesn't have a separate HTTP request user agent string": it only works as a robots.txt token and never appears in your log.

[2] Pull the bot name out of the User-Agent field and count it. Splitting on " makes field 6 the User-Agent.

console

$ awk -F'"' '{print $6}' access.log | grep -oE '(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User)' | sort | uniq -c | sort -rn     79 GPTBot     18 ClaudeBot     11 OAI-SearchBot      9 PerplexityBot      7 Claude-SearchBot      5 ChatGPT-User      4 Perplexity-User      4 Claude-User

Add new names inside the grep -oE parentheses. To discover bots you don't know yet: awk -F'"' '$6 ~ /[Bb]ot/ {print $6}' access.log | sort | uniq -c | sort -rn | head.

[3] See what they fetched, by status code and path. The request line is field 2; the status is the first word of field 3.

console

$ awk -F'"' -v bots='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' '$6 ~ bots {split($2,r," "); split($3,s," "); print s[1], r[2]}' access.log | sort | uniq -c | sort -rn | head     82 200 /posts/a/     30 200 /posts/b/     11 200 /robots.txt      9 200 /posts/c/      3 404 /posts/b/      2 404 /llms.txt

Two 404s on /llms.txt only mean you have no such file. They don't show that a bot follows it, and this post makes no claim about its effect.

[4] Count the /robots.txt requests separately and check that they return 200.

console

$ awk -F'"' -v bots='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' '$6 ~ bots {split($2,r," "); if (r[2]=="/robots.txt") {split($3,s," "); print s[1]}}' access.log | sort | uniq -c     11 200

Some bots fetch pages without requesting robots.txt. A missing line is not proof of a violation; it only means the read isn't in this log. OpenAI's docs say robots.txt rules may not apply to ChatGPT-User.

[5] Count the source IPs behind the same names.

console

$ awk -F'"' -v bots='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' '$6 ~ bots {split($1,a," "); print a[1]}' access.log | sort | uniq -c | sort -rn     54 203.0.113.10     25 198.51.100.77     18 203.0.113.13     11 203.0.113.11      9 203.0.113.14      7 203.0.113.16      5 203.0.113.12      4 203.0.113.17      4 203.0.113.15

The 79 GPTBot hits come from two IPs (54 + 25). Anyone can write any User-Agent string, so the name alone doesn't prove the bot is real.

[6] Compare against the published IP lists. OpenAI publishes one JSON per bot (https://openai.com/gptbot.json, searchbot.json, chatgpt-user.json). Anthropic publishes https://claude.com/crawling/bots.json. Perplexity publishes https://www.perplexity.com/perplexitybot.json and perplexity-user.json. The downloaded gptbot.json has a prefixes array of ipv4Prefix entries.

console

$ curl -sL https://openai.com/gptbot.json -o gptbot.json$ jq -r '.prefixes[]|.ipv4Prefix // .ipv6Prefix' gptbot.json > gptbot.cidr$ wc -l gptbot.cidr18 gptbot.cidr

Testing CIDR membership is awkward in awk, so use Python's standard library. chk.py takes the CIDR file as its first argument and reads uniq -c output (a count and an IP per line) on stdin.

chk.py

import sys, ipaddressnets=[ipaddress.ip_network(l.strip()) for l in open(sys.argv[1]) if l.strip()]for line in sys.stdin:    count, ip = line.split()    ok=any(ipaddress.ip_address(ip) in n for n in nets)    print(count, ip, 'listed' if ok else 'NOT listed')

console

$ awk -F'"' '$6 ~ /GPTBot/ {split($1,a," "); print a[1]}' access.log | sort | uniq -c | sort -rn | python3 chk.py gptbot.cidr54 203.0.113.10 NOT listed25 198.51.100.77 NOT listed1 20.171.206.5 listed

The last line is one entry I appended to the log from a listed range (20.171.206.0/24). Every client in the sample log uses a documentation address, so the first two can never be on the list. On a real server, count the requests from NOT listed IPs separately to spot impostors. The lists change (the downloaded file's creationTime was 2026-09-22), so download a fresh copy each time.

Scope of testing: the log is hand-made combined format, not real bot traffic. Bot names, purposes and IP-list URLs were checked in the vendors' documentation (OpenAI, Anthropic, Perplexity, Google); the Perplexity JSON was not downloaded. In the sample log, the Claude-User and Claude-SearchBot lines match the documented names only; their full User-Agent strings were not verified. Nginx was not installed, and the awk field positions assume the combined format. If you changed log_format, the field numbers will differ.

Analyze your Nginx log in the browser ↗ — nothing is uploaded.

#nginx#awk#ai-crawler#robots-txt

Count AI Crawler Requests in an Nginx Access Log

Related manuals