AI Crawler Report: srchengine.com
Checked 2026-11-09 13:28:14 · stored report, re-run the checker for a fresh one.
https://srchengine.com
All major AI crawlers can read your site.
| Crawler | Used for | Status |
|---|
| Answer crawlers: these fetch and cite your pages in AI answers |
ChatGPT-User
OpenAI | Fetches pages for ChatGPT answers | ✓ Allowed |
OAI-SearchBot
OpenAI | ChatGPT search index | ✓ Allowed |
PerplexityBot
Perplexity | Perplexity search index | ✓ Allowed |
Perplexity-User
Perplexity | Fetches pages for Perplexity answers | ✓ Allowed |
Claude-User
Anthropic | Fetches pages for Claude answers | ✓ Allowed |
Claude-SearchBot
Anthropic | Claude search index | ✓ Allowed |
| Training crawlers: blocking these only opts out of model training |
GPTBot
OpenAI | Model training | ✓ Allowed |
ClaudeBot
Anthropic | Model training | ✓ Allowed |
anthropic-ai
Anthropic | Model training (legacy token) | ✓ Allowed |
Google-Extended
Google | Gemini training (token, sends no requests) | ✓ Allowed |
CCBot
Common Crawl | Open training data | ✓ Allowed |
Bytespider
ByteDance | Model training | ✓ Allowed |
/llms.txt
AI-readable site summary | 4 KB | ✓ Found |
Checked 2026-11-09 13:28:14 · Free tool by SrchEngine
Get the full report
Enter your email to unlock the full report for srchengine.com: what each blocked crawler means, the exact robots.txt fix, and a starter llms.txt for the site.
Unlocks on this device for 30 days. We add you to the SrchEngine list for the report and occasional AI search notes. Unsubscribe anytime.