What is robots.txt and why does it matter for AI?

It is a text file at the site root that declares which bots can access which paths. AI assistants follow this file: if your domain blocks OAI-SearchBot, ChatGPT cannot open your page to cite it in a response, even if it is the best source available on the subject.

What's the difference between a search crawler and a training crawler?

A search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) reads your page at the moment someone asks, to build the answer and credit the source with a link. A training crawler (GPTBot, Google-Extended, CCBot) collects content to train future models, with no link back. Blocking the training one protects your content; blocking the search one takes you out of the answer.

Does blocking GPTBot remove my brand from ChatGPT?

Not directly. GPTBot is OpenAI's training crawler. OAI-SearchBot and ChatGPT-User are the ones that search in real time and cite sources. You can block training and still be cited, several news outlets do exactly that.

My site has no robots.txt. Is that bad?

Under the protocol, the absence of the file means everything is allowed, so no AI bot is blocked. It is not a visibility problem. It is a problem if you thought you had control and did not.

My site blocks everything. How do I allow only the right crawlers?

Declare a group named after each agent you want to allow. The rule of the protocol is that the named group beats the wildcard User-agent, whether it comes before or after. A group with an empty Disallow allows that agent entirely.

How often should I check this?

Whenever you change CMS, CDN, or e-commerce platform, many generate robots.txt on their own and overwrite yours. It is also worth checking when a new agent enters circulation: the list of AI bots has changed several times since 2023.

Tools

See whether your website blocks AI crawlers.

One line in robots.txt decides whether ChatGPT, Claude, and Perplexity can read your website and cite your brand. Enter the domain and see the 10 agents that matter, one by one.

Module: AEO/GEO
Input: A domain
Runs on: Rovemark server
Cost: Free, no signup

What the tool reads

The file that decides whether AI can cite you

robots.txt sits at the domain root and is the first thing any crawler requests before opening a page. It is where the site declares who gets in and who does not. Until 2023, this list had only one audience: search engines. Today it has two, and most websites have never separated one from the other.

The tool fetches your robots.txt, applies the protocol rules, the named agent group overrides the wildcard, Allow overrides Disallow on the same path, and returns the verdict for the 10 AI agents most active today, grouped by what each one does.

How it works

You enter the domain

Just the domain, no signup. The tool fetches robots.txt straight from the origin, the way a crawler would.

The rules are applied per agent

It's not a text search: the file is parsed the way the protocol requires, respecting precedence between a named group and the wildcard.

The verdict comes split by role

Search crawlers and training crawlers appear in separate blocks, because blocking one is a marketing decision and blocking the other is an intellectual property decision.

Frequently asked questions

01 What is robots.txt and why does it matter for AI?: It is a text file at the site root that declares which bots can access which paths. AI assistants follow this file: if your domain blocks OAI-SearchBot, ChatGPT cannot open your page to cite it in a response, even if it is the best source available on the subject.
02 What's the difference between a search crawler and a training crawler?: A search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) reads your page at the moment someone asks, to build the answer and credit the source with a link. A training crawler (GPTBot, Google-Extended, CCBot) collects content to train future models, with no link back. Blocking the training one protects your content; blocking the search one takes you out of the answer.
03 Does blocking GPTBot remove my brand from ChatGPT?: Not directly. GPTBot is OpenAI's training crawler. OAI-SearchBot and ChatGPT-User are the ones that search in real time and cite sources. You can block training and still be cited, several news outlets do exactly that.
04 My site has no robots.txt. Is that bad?: Under the protocol, the absence of the file means everything is allowed, so no AI bot is blocked. It is not a visibility problem. It is a problem if you thought you had control and did not.
05 My site blocks everything. How do I allow only the right crawlers?: Declare a group named after each agent you want to allow. The rule of the protocol is that the named group beats the wildcard User-agent, whether it comes before or after. A group with an empty Disallow allows that agent entirely.
06 How often should I check this?: Whenever you change CMS, CDN, or e-commerce platform, many generate robots.txt on their own and overwrite yours. It is also worth checking when a new agent enters circulation: the list of AI bots has changed several times since 2023.