What this checker tests
Enter any website address and the checker reads its robots.txt file, the same file AI crawlers read before they visit. Bot by bot, it reports whether each crawler is allowed to read the site, blocked from all of it, or blocked from parts of it.
The bots it tests:
- GPTBot: OpenAI’s crawler, gathering pages ChatGPT can learn from and cite
- ChatGPT-User: visits pages live when a ChatGPT user asks about them
- ClaudeBot and anthropic-ai: Anthropic’s crawlers, for Claude
- PerplexityBot: Perplexity’s crawler
- Google-Extended: Google’s AI crawler, for Gemini
- CCBot: Common Crawl, whose open dataset feeds many AI models
- Bytespider: ByteDance’s crawler
It also checks for an llms.txt file. llms.txt is a plain text file at your site’s root, written for AI systems rather than people. It tells them what your business does, which pages matter most, and how to describe you accurately. Think of it as a one-page briefing you hand to every AI engine that comes looking. A site without one is asking the engines to guess, and engines that have to guess often guess a competitor instead.
What a result looks like
Sample result for a typical small business site (sample only, not a real scan):
- GPTBot: Allowed
- ClaudeBot: Allowed
- PerplexityBot: Blocked by robots.txt
- llms.txt: Not found
In this sample, the business is invisible to Perplexity without knowing it, and no llms.txt briefing exists for any engine. Both are quick fixes once you can see them.
Why this matters
AI search engines can only cite and recommend what they are allowed to read. A robots.txt rule added years ago to block aggressive scrapers may also be blocking the crawlers that feed ChatGPT, Claude and Perplexity today. Two minutes with this checker shows you exactly where your site stands, in plain language, before a customer asks an AI engine about you and hears about someone else.
Go deeper: AI Search Trust Signals Explained Simply and What Makes Content AI-Citable: A Practical Guide.
AI Crawler Check: questions
What does the AI Crawler Check actually read?
Your site’s public robots.txt file, the same file AI crawlers read before visiting. It matches the rules against eight major AI crawlers and also checks whether an /llms.txt file exists at your site root.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI’s crawler, gathering pages ChatGPT can learn from and cite. ChatGPT-User visits pages live when a ChatGPT user asks about them. Blocking one does not block the other; each needs its own robots.txt rule.
If a crawler shows as Not specified, is it allowed?
Yes, by default. Crawlers are allowed unless a robots.txt rule blocks them. Not specified means your file has no rule naming that crawler, so it can read the site unless a blanket rule covers it.
What is llms.txt and do I need one?
llms.txt is a plain text file at your site’s root, written for AI systems: what your business does, which pages matter most, and how to describe you. The checker reports whether one exists. Without one, the engines piece the picture together themselves.
How do I unblock a crawler?
Add an Allow rule for its user-agent in robots.txt, above any blanket Disallow for it. Your shareable report includes the exact snippet for the crawlers your site blocks.
Benchmark state of play
Scores move fast, so this section reads from live data rather than a paragraph that goes stale: the current leader on five headline benchmarks from Epoch AI’s independent evaluation hub, the Epoch Capabilities Index top 5, and the top of the official SWE-bench Verified leaderboard.
- Epoch Capabilities Index167.3 ECI
- GPQA Diamond95.8%
- SWE-bench Verified (Epoch runs)83.5%
- FrontierMath (Tiers 1-3)93.7%
- MATH Level 598.1%
- SimpleQA Verified75.6%
- SWE-bench Verified (official)79.2 % resolved
Epoch Capabilities Index: top 5
- 1Claude Opus 5.5167.3
- 2GPT-6 Astra166.4
- 3GPT-6.1 Sol166.1
- 4Claude Sonnet 5.5165
- 5Claude Fable 5.1164.7
SWE-bench Verified (official leaderboard): top 5
- 1Claude 4.5 Opus79.2% resolved
- 2Claude 4.5 Opus79.2% resolved
- 3Doubao-Seed-Code78.8% resolved
- 4Gemini 3 Pro Preview77.4% resolved
- 5Claude 4 Sonnet76.8% resolved