AI crawlers and agents: who they are, what robots.txt does, and how to tell the real ones
A directory for website owners. Every entry comes from the company's own documentation, linked, as read on November 15, 2026. Updated November 15, 2026.
AI companies send three kinds of bots to websites: crawlers that collect pages to train AI models, search bots that find pages for AI search answers, and fetchers and agents that visit a page because a person asked an AI to. They're treated differently, by the companies and by robots.txt.
One thing holds for all of them: a user agent is just text. Anything can say it's GPTBot or ChatGPT. To know a request really came from the company, check its IP address against the list the company publishes, or, for agents that sign their requests, check the signature.
The directory
OpenAI
Source: OpenAI's page on its bots.
- GPTBot: collects content that may be used to train OpenAI's AI models. Disallowing it in robots.txt tells OpenAI not to use your content for training. IP addresses: openai.com/gptbot.json.
- OAI-SearchBot: finds websites to show in ChatGPT's search features. A site that blocks it can still appear as a plain link. IP addresses: openai.com/searchbot.json.
- ChatGPT-User: visits a page when someone using ChatGPT asks it to. OpenAI says it isn't used to crawl the web automatically, and that robots.txt rules may not apply to it. IP addresses: openai.com/chatgpt-user.json.
- ChatGPT's agent, which can browse and shop for a person, signs its requests with Web Bot Auth, from
chatgpt.com. How that works: AI shopping agents.
Anthropic
Source: Anthropic's page on its crawlers. Anthropic says its bots honor robots.txt, including Crawl-delay. IP addresses for its bots: claude.com/crawling/bots.json.
- ClaudeBot: collects web content that could be used to train Anthropic's models.
- Claude-SearchBot: browses to improve the quality of search results for Claude's users.
- Claude-User: visits a page when someone asks Claude a question.
- Claude-Agent: a browser that a Claude agent drives to do tasks for a person. Anthropic's page on it warns that anyone can send the same user agent, and points to a list it publishes at
claude.com/.well-known/sail, in a format that is still a draft.
Perplexity
Source: Perplexity's page on its bots.
- PerplexityBot: finds and links websites in Perplexity's search results. Perplexity says it isn't used to collect content for AI models. IP addresses: perplexity.ai/perplexitybot.json.
- Perplexity-User: visits a page when someone asks Perplexity a question, and isn't used for crawling. Perplexity says it generally ignores robots.txt. IP addresses: perplexity.ai/perplexity-user.json.
Source: Google's page on its common crawlers.
- Google-Extended isn't a separate bot: it's a robots.txt name. Google crawls with its usual crawlers, and Google-Extended decides whether what they collect may be used to train future Gemini models and for grounding. It doesn't affect Google Search. It sends no user agent of its own, so you'll never see it in your logs.
- Google publishes the addresses of its crawlers in common-crawlers.json. It says its fetchers that run when a person asks ignore robots.txt.
Apple
Source: Apple's page on Applebot.
- Applebot: crawls for Apple's search, Siri and Spotlight. Apple says what it collects may also help train Apple's AI models, and that it follows robots.txt in its general search crawls. IP addresses: search.developer.apple.com/applebot.json.
- Applebot-Extended doesn't crawl: it's a robots.txt name that decides whether what Applebot collects may be used to train Apple's AI models.
Amazon
Source: Amazon's page on its bots.
- Amazonbot: used to improve Amazon's products and services, and may be used to train Amazon's AI models. Amazon says its automated crawling follows robots.txt. IP addresses: Amazon's list.
- Amzn-SearchBot: improves search in Amazon's products, and isn't used for AI training. IP addresses: Amazon's list.
- Amzn-User: fetches live information when a person asks, such as an Alexa question. Amazon says it may not follow all robots.txt rules. IP addresses: Amazon's list.
Meta
Source: Meta's page on its crawlers. Meta doesn't publish a list of addresses on that page.
- meta-externalagent: crawls for uses such as training AI models or indexing content for Meta's products.
- meta-externalfetcher: fetches a link when a person asks, including for AI agents. Meta says it may bypass robots.txt.
Common Crawl
Source: Common Crawl's page on CCBot.
- CCBot: builds Common Crawl's open archive of web pages. To keep it out, Common Crawl gives a robots.txt rule for CCBot. IP addresses: index.commoncrawl.org/ccbot.json.
ByteDance
- Bytespider: we couldn't find a page from ByteDance that describes it, so we won't say what it does or whether it follows robots.txt.
What robots.txt can and can't do
robots.txt is a request, not a lock. Every company above whose crawling can feed AI training, ByteDance aside, documents a robots.txt name for it, and that's how you opt out. Anthropic, Apple, Amazon and Google say their crawlers follow it. But several fetchers that act for a person say plainly that they may not follow it: ChatGPT-User, Perplexity-User, Amzn-User, meta-externalfetcher, and Google's user-triggered fetchers. And a bot that lies about its name ignores robots.txt entirely.
To opt out of training by the crawlers above while staying in AI search answers, a robots.txt along these lines names each one:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /
Amazonbot and meta-externalagent aren't only for training: blocking them also keeps your site out of other uses in Amazon's and Meta's products. Check each company's page before you copy a rule.
How to tell a real one from a fake
- By IP address. Most of the companies above publish the addresses their bots use. A request that says GPTBot from an address that isn't on
openai.com/gptbot.jsonisn't GPTBot. - By signature. Agents that shop and act for people are starting to sign every request with Web Bot Auth, so a site can check it really came from the company. That's the reliable way at a checkout, where addresses are shared and lists change. More in AI shopping agents.
- Never by user agent alone, above all at checkout. A card-testing script can say it's ChatGPT as easily as anything else.
Where Ropeline fits
Ropeline checks AI agents' signatures at the door that matters to a store: the checkout. A request that claims to be a verified agent but whose signature doesn't check out is treated as the bot it is. A real one shops normally.
- The free AI agent check shows in seconds whether real AI shoppers can buy from your store, and whether a fake gets in too.
- Ropeline Pro for WordPress and WooCommerce checks signatures at checkout, and stops card testing whatever the bot calls itself.
- Ropeline Proof does the same for any store's own server, signs a receipt for each verified agent's purchase, and keeps a decision log your security team can check.
Store plan: a free week, no card, then $15 a month for stores with up to 500 orders a month ($39 up to 3,000, $99 for any size).
Install in a few minutes: download, upload to WordPress, paste your key. Cancel anytime and your site keeps the free plugin.
Start a free week, no card Check my store's AI agents freeQuestions
Should I block all AI bots?
Only if you don't want to appear in AI answers either. Blocking the training crawlers keeps your content out of training; blocking the search bots and fetchers also keeps you out of AI search results and stops people's AI assistants from reading your pages. A store that wants AI shoppers to buy from it shouldn't block the agents that act for them.
Why doesn't my robots.txt stop a bot that says it's ChatGPT?
Either it's ChatGPT-User acting for a person, which OpenAI says robots.txt may not apply to, or it isn't OpenAI at all. Check its address against OpenAI's list.
Is this list complete?
No. It covers the companies whose bots most websites see, and only what each documents. New bots appear often.