AI crawler checker: can ChatGPT, Claude and Perplexity read your site?

Which AI assistants may read your site, from your robots.txt, your page's robots tags and what your firewall sends each crawler.

Updated 30 October 2026

What it checks

robots.txt read the way crawlers read it (RFC 9309): the group that names each crawler, else the catch-all group, then the longest matching rule
19 crawlers and control tokens from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo, Mistral, Common Crawl and ByteDance, each labelled search, user, training or control
The page's robots meta tag and X-Robots-Tag header: noindex, nosnippet and max-snippet:0 keep a page out of AI answers even when robots.txt lets crawlers in
What your firewall sends three answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) compared with a normal request, to catch bot rules robots.txt cannot show

Four kinds of AI crawler

search: Indexes pages for AI answers. Block it and that assistant cannot cite you.
user: Fetches a page when someone asks the assistant about it. Block it and the assistant cannot open your page.
training: Collects text to train models. Blocking it does not stop you appearing in AI answers.
control: A robots.txt token with no crawler of its own. It decides whether pages already crawled may be used for AI.

Use it from code or an AI agent

The same check is a free JSON API and an MCP tool. No key, no sign-up.

curl "https://aeolookup.com/api/aeo/ai-crawlers?url=example.co.uk"

MCP: add https://aeolookup.com/mcp as a remote server and call check_ai_crawler_access. API and MCP documentation.

Questions people ask

Should I block GPTBot?

Block GPTBot only if you object to OpenAI training models on your pages. It does not decide whether ChatGPT can cite you: OAI-SearchBot and ChatGPT-User do, and OpenAI documents the three separately. Most firms that want to be found should leave OAI-SearchBot and ChatGPT-User open.

Why does robots.txt allow a crawler but the checker says it is refused?

Something in front of your site is refusing it. Cloudflare has blocked AI crawlers by default on new domains since 1 July 2025, and many hosts and security plugins do the same. Look for an AI crawler or bot setting in your CDN or firewall. Some rules refuse only requests that pretend to be a crawler, so treat a refusal as a strong sign rather than proof.

Does blocking Google-Extended take me out of AI Overviews?

No. Google says AI Overviews and AI Mode use the normal Googlebot index, and Google-Extended only controls whether Gemini models may train on or ground answers in your pages. The way to keep a page out of AI Overviews is the nosnippet rule, which also removes its ordinary search snippet.

How current is the crawler list?

Each crawler is checked against its operator's own documentation, linked in the results. The list was last checked on 2026-10-30. New crawlers appear every few months, so check again after a site change.

The other free AEO tools

Schema checkerReads every JSON-LD block on a page, checks each against the properties Google and answer engines use, and scores how clearly it says who you are.llms.txt validatorChecks a site's llms.txt against the llmstxt.org format and tests its links, or drafts one from your sitemap.Answer-readiness checkerScores a page out of 100 on the things that make an answer engine able to quote it correctly, with the fixes worth most points first.Full AEO auditRuns the crawler, schema, llms.txt and answer-readiness checks on one address at once.