Founding offerThe first 10 online stores pay $9 a month instead of $15, for as long as they stay.Founding offer: $9 a month instead of $15 for the first 10 stores.Claim yours
Ropeline Try Pro free

AI crawlers and agents: who they are, what robots.txt does, and how to tell the real ones

A directory for website owners. Every entry comes from the company's own documentation, linked, as read on November 15, 2026. Updated November 15, 2026.

AI companies send three kinds of bots to websites: crawlers that collect pages to train AI models, search bots that find pages for AI search answers, and fetchers and agents that visit a page because a person asked an AI to. They're treated differently, by the companies and by robots.txt.

One thing holds for all of them: a user agent is just text. Anything can say it's GPTBot or ChatGPT. To know a request really came from the company, check its IP address against the list the company publishes, or, for agents that sign their requests, check the signature.

The directory

OpenAI

Source: OpenAI's page on its bots.

Anthropic

Source: Anthropic's page on its crawlers. Anthropic says its bots honor robots.txt, including Crawl-delay. IP addresses for its bots: claude.com/crawling/bots.json.

Perplexity

Source: Perplexity's page on its bots.

Google

Source: Google's page on its common crawlers.

Apple

Source: Apple's page on Applebot.

Amazon

Source: Amazon's page on its bots.

Meta

Source: Meta's page on its crawlers. Meta doesn't publish a list of addresses on that page.

Common Crawl

Source: Common Crawl's page on CCBot.

ByteDance

What robots.txt can and can't do

robots.txt is a request, not a lock. Every company above whose crawling can feed AI training, ByteDance aside, documents a robots.txt name for it, and that's how you opt out. Anthropic, Apple, Amazon and Google say their crawlers follow it. But several fetchers that act for a person say plainly that they may not follow it: ChatGPT-User, Perplexity-User, Amzn-User, meta-externalfetcher, and Google's user-triggered fetchers. And a bot that lies about its name ignores robots.txt entirely.

To opt out of training by the crawlers above while staying in AI search answers, a robots.txt along these lines names each one:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

Amazonbot and meta-externalagent aren't only for training: blocking them also keeps your site out of other uses in Amazon's and Meta's products. Check each company's page before you copy a rule.

How to tell a real one from a fake

  1. By IP address. Most of the companies above publish the addresses their bots use. A request that says GPTBot from an address that isn't on openai.com/gptbot.json isn't GPTBot.
  2. By signature. Agents that shop and act for people are starting to sign every request with Web Bot Auth, so a site can check it really came from the company. That's the reliable way at a checkout, where addresses are shared and lists change. More in AI shopping agents.
  3. Never by user agent alone, above all at checkout. A card-testing script can say it's ChatGPT as easily as anything else.

Where Ropeline fits

Ropeline checks AI agents' signatures at the door that matters to a store: the checkout. A request that claims to be a verified agent but whose signature doesn't check out is treated as the bot it is. A real one shops normally.

Store plan: a free week, no card, then $15 a month for stores with up to 500 orders a month ($39 up to 3,000, $99 for any size).

Install in a few minutes: download, upload to WordPress, paste your key. Cancel anytime and your site keeps the free plugin.

Start a free week, no card Check my store's AI agents free

Questions

Should I block all AI bots?

Only if you don't want to appear in AI answers either. Blocking the training crawlers keeps your content out of training; blocking the search bots and fetchers also keeps you out of AI search results and stops people's AI assistants from reading your pages. A store that wants AI shoppers to buy from it shouldn't block the agents that act for them.

Why doesn't my robots.txt stop a bot that says it's ChatGPT?

Either it's ChatGPT-User acting for a person, which OpenAI says robots.txt may not apply to, or it isn't OpenAI at all. Check its address against OpenAI's list.

Is this list complete?

No. It covers the companies whose bots most websites see, and only what each documents. New bots appear often.