AI & GEO
AI Crawler
What is AI Crawler?
An AI crawler is a bot operated by an AI company that fetches web pages either to collect training data for language models or to retrieve pages in real time when an AI assistant answers a question. Examples include OpenAI's GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (fetches on a user's behalf), Anthropic's ClaudeBot, PerplexityBot and Common Crawl's CCBot.
Some AI controls are robots.txt tokens rather than separate crawlers. Google-Extended, for example, lets a site opt out of its content being used to train and ground Gemini models without affecting Google Search crawling by Googlebot. Most reputable AI crawlers publish their user-agent strings and honour robots.txt rules.
Why it matters for SEO
Blocking or allowing AI crawlers is now a strategic choice. Blocking training bots can protect content, but blocking the retrieval and search bots of AI assistants can stop a site from being cited in their answers. Checking server logs for these user agents also shows how often AI systems read your pages.
Example
A news publisher wants its articles cited in ChatGPT search results but not used for model training. Its robots.txt contains "User-agent: GPTBot" followed by "Disallow: /", while leaving OAI-SearchBot allowed. It then filters its server logs by user agent to confirm OAI-SearchBot is still fetching new articles.
Free tools for AI Crawler
Put the theory to work
LazySEO researches keywords, writes SEO articles and publishes them to your site on a schedule.