What the checker tests
The scan runs a focused slice of the CanAgentUse assessment suit for crawler discovery and AI-readable access.
- robots.txt availability
- AI bot rule interpretation
- sitemap discovery
- canonical and link hints
AI crawler compatibility tool
AI Crawler Compatibility Checker
Check whether AI crawlers like GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI user agents can discover, crawl, and understand your website.
Live crawler access diagnostic
Bot access map
The scan follows the public discovery path an AI retrieval system would depend on: robots.txt, bot-specific policy, sitemap hints, canonical links, and AI-readable guidance.
GPTBot
ClaudeBot
PerplexityBot
Google-Extended
- Crawler rules
- AI bots
- Discovery
- Sitemap
- Guidance
- llms.txt
Robots.txt AI crawler rules
The report checks whether important AI user agents are allowed, blocked, or caught by broad wildcard rules.
- GPTBot and OAI-SearchBot
- ClaudeBot
- PerplexityBot
- Google-Extended
WAF and bot-blocking issues
Crawler access can fail even when robots.txt looks open. The scanner records fetch evidence and highlights blocking symptoms when checks detect them.
Sitemap and canonical checks
AI crawlers need stable discovery paths. The diagnostic validates sitemap candidates and link-header signals that help agents choose canonical resources.
llms.txt and AI-readable guidance
A present and useful llms.txt file can give AI systems a concise map of your important pages, policies, and source material.
How to fix issues
Every failing or warning check includes evidence and remediation guidance from the same versioned checks used in full reports.
FAQ
What people ask before scanning
What is AI crawler compatibility?
AI crawler compatibility means your site can be fetched, discovered, and interpreted by AI search and answer systems without accidental robots.txt, sitemap, header, or AI guidance blockers.
Does this check GPTBot specifically?
Yes. The focused scan checks AI bot access rules including OpenAI crawlers such as GPTBot and OAI-SearchBot, alongside other major AI crawler families.
Is this a full AI agent readiness scan?
No. This diagnostic focuses on crawler access and discovery. The full AI Agent Readiness Scanner adds structured data, APIs, OAuth, MCP, skill discovery, commerce, performance, and accessibility checks.