How do I control which AI crawlers can use my site?

What is the difference between GPTBot and ChatGPT-User?

GPTBot collects content to train models. ChatGPT-User fetches a page because a person just asked a question and the assistant is going to cite what it finds. Same company, opposite commercial consequences for you.

The pattern is straightforward once you see it: deny the training crawlers if you object to training, and allow every citation crawler unconditionally. Blocking a citation crawler removes you from answers your customers are actively asking for.

Am I already blocking AI crawlers without knowing?

Quite possibly. Around 5.9% of sites block major AI crawlers and most never made that decision.

The three usual causes: a CMS or theme template that shipped with a restrictive robots.txt, a hosting provider that enables AI bot management by default on new zones, and a well-meaning agency that read one article about AI scraping and blocked everything.

Cloudflare in particular has shipped default AI-blocking behaviour to new zones. If you created a zone recently and never reviewed it, check before assuming.

$ curl -s https://yourdomain.com/robots.txt | grep -iA2 "gptbot\|chatgpt\|perplexity\|claude"

If that returns a Disallow you did not write, you have been invisible to AI assistants for as long as it has been there.

What is a Content-Signal directive?

A line in robots.txt that declares how your content may be used, separately from whether it may be fetched. It distinguishes three purposes: training a model, appearing in search, and being used as input to an answer.

Content-Signal: ai-train=no, search=yes, ai-input=yes

That single line says: do not train on this, do index it, and do use it when answering someone’s question. Roughly 4% of sites had declared these as of April 2026, which makes it one of the cheapest differentiators available. Note it is a declaration of preference, not an enforcement mechanism.

Copy-paste robots.txt template for AI crawlers

This blocks training while allowing every citation and search agent. Adjust the training block to your own position — the structure is the point.

# Training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

# Citation and search agents
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Content-Signal: ai-train=no, search=yes, ai-input=yes

Sitemap: https://yourdomain.com/sitemap.xml

Order matters less than specificity: a named user-agent block overrides the wildcard. Test the result rather than trusting the file — our free scan checks 13 named agents and reports what each one actually resolves to.

Will blocking training crawlers hurt my visibility?

Directly, no — citation agents are separate and unaffected. Indirectly, it is genuinely unknown. If a model has never trained on your content it may have weaker background knowledge of your brand when it does cite you, and nobody has published credible evidence either way.

What is certain is the reverse: blocking citation agents removes you from live answers immediately and measurably. If you are going to be cautious about one thing, be cautious about that.

Where this is contested

Two things. First, robots.txt is a request, not a control — well-behaved crawlers honour it and badly-behaved ones ignore it. If you need enforcement you need bot management at the edge, and cryptographic bot authentication is the emerging answer rather than IP allow-lists.

Second, the training question is a business judgement, not a technical one, and we do not think there is a universally correct answer. A publisher licensing content and a SaaS company wanting mindshare should reach opposite conclusions. We implement your position; we do not have one for you.