Check Whether robots.txt Blocks AI Crawlers with Python
Blocking GPTBot doesn't block every AI bot. Parse robots.txt with Python and print an allow/block table for training, search and user-request crawlers.
Adding Disallow: / under User-agent: GPTBot does not shut out every AI service. Each company splits its bots by purpose, so check which bot gets which rule.
[1] Read what each bot is for. OpenAI's crawler docs say:
| Bot | Documentation |
|---|---|
| GPTBot | "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." |
| OAI-SearchBot | "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." |
| ChatGPT-User | "ChatGPT-User is not used for crawling the web in an automatic fashion." |
ChatGPT-User actions are started by a user, and the docs add "robots.txt rules may not apply." A BLOCK for ChatGPT-User below is not a guarantee.
[2] Anthropic describes three bots in its support article. ClaudeBot collects web content that could contribute to training. Claude-User fetches pages when a person asks Claude a question. Claude-SearchBot improves search result quality. Block each one with its own User-agent line and Disallow: /. Apply the rules to every subdomain, and don't block by IP range: the article warns that it can stop the bots from reading robots.txt.
[3] Create a sample robots.txt. It blocks the training bots (GPTBot, ClaudeBot) and allows the two search bots. Two adjacent User-agent lines share one rule group.
User-agent: *Disallow: /admin/ User-agent: GPTBotDisallow: / User-agent: ClaudeBotDisallow: / User-agent: OAI-SearchBotUser-agent: Claude-SearchBotAllow: /[4] Write a script around urllib.robotparser. It parses a local file, so it makes no network requests.
#!/usr/bin/env python3"""usage: check_ai_robots.py robots.txt URL_PATH"""import sysfrom urllib.robotparser import RobotFileParserBOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot", "Claude-User"]rp = RobotFileParser()rp.parse(open(sys.argv[1], encoding="utf-8").read().splitlines())path = sys.argv[2]for bot in BOTS: print(f"{bot:<18} {'allow' if rp.can_fetch(bot, 'https://example.com' + path) else 'BLOCK'}")[5] Check an ordinary article path.
$ python3 -I check_ai_robots.py robots.txt /blog/post/GPTBot BLOCKOAI-SearchBot allowChatGPT-User allowClaudeBot BLOCKClaude-SearchBot allowClaude-User allowOnly the two training bots are blocked.
[6] Check /admin/ too. The * group applies only to bots with no group of their own.
$ python3 -I check_ai_robots.py robots.txt /admin/xGPTBot BLOCKOAI-SearchBot allowChatGPT-User BLOCKClaudeBot BLOCKClaude-SearchBot allowClaude-User BLOCKOAI-SearchBot and Claude-SearchBot have their own Allow: / group, so /admin/ is open to them. Add Disallow: /admin/ to that group to close it.
[7] Know how names are matched. In this test Python's robotparser applied a group when its User-agent value was contained in the bot name.
$ python3 - <<'PY'from urllib.robotparser import RobotFileParserrp = RobotFileParser(); rp.parse("User-agent: Claude\nDisallow: /".splitlines())for b in ["ClaudeBot", "Claude-SearchBot", "Claude-User", "GPTBot"]: print(b, rp.can_fetch(b, "https://example.com/"))PYClaudeBot FalseClaude-SearchBot FalseClaude-User FalseGPTBot TrueOne loose Claude line blocked all three bots. Pass only the product name (GPTBot) to can_fetch(): the full string Mozilla/5.0 (compatible; GPTBot/1.4; …) returned True against a GPTBot block. Extract the product name from log user agents first.
[8] After you publish a change, wait. OpenAI's docs say it can take about 24 hours after a robots.txt update for its search systems to adjust.