The bots AI companies run to read the web for training data and live answers; GPTBot, ClaudeBot and PerplexityBot are the best-known examples.
What is an AI crawler?
AI crawlers are automated programs that visit and read web pages much like search engine bots do. The difference is purpose: what they collect is used to train models or to answer a question on the spot, not to rank a list of results. GPTBot, ClaudeBot and PerplexityBot are the best-known examples.
One technical detail matters most: most of these bots do not execute JavaScript. However rich your page looks in a browser, anything absent from the raw HTML the server returns does not exist for them. An FAQ that only opens on click, a chart drawn purely in script, a price baked into an image — all of it reads as an empty page.
Managing access: block or allow?
Access is governed through robots.txt and, if you run a layer like Cloudflare, through bot rules. The decision is not binary: blocking out of concern for how your content is used and allowing in order to appear in AI answers are both legitimate positions. Blocking costs you visibility; allowing means your content can be used without attribution.
- Blocking can be selective: you can refuse a training crawler while allowing one that fetches sources at answer time.
- Accidental blocking is common; an aggressive bot rule can shut out the engine you actually wanted to reach.
- Because what these bots see is raw HTML, keeping important content in the server-rendered markup is the foundation of GEO work.
For the model producing the answer see LLM; for classic crawling see search engine. The practical application is in our GEO guide.
Frequently asked questions
Should I block AI crawlers?
It depends. If you do not want your content used for model training you can block them, at the cost of reducing the chance your brand is mentioned in that engine's answers. If visibility is the priority, keeping them open makes sense. The choice is not binary either — different bots can be given different policies.
My site allows AI crawlers but I still don't appear in answers. Why?
The most common reason is that the content is not in the raw HTML. Since most of these bots do not run JavaScript, anything that appears only after interaction is invisible to them. The second most common reason is scattered, inconsistent information: clear definitions, FAQ blocks and repeated consistent brand facts are far easier to quote.
Related concepts
Cloudflare
The infrastructure provider that sits between your domain and your visitors, bundling DNS, CDN, caching, SSL and bot protection into a single dashboard.
GEO (Generative Engine Optimization)
The discipline of making sure a brand is mentioned accurately, currently and favorably inside answers generated by ChatGPT, Perplexity, Google's AI summaries and other generative engines.
LLM (Large Language Model)
An AI model trained on very large volumes of text that generates language by predicting what comes next; the technology behind products like ChatGPT, Claude and Gemini.
Search Engine
A system that crawls the web, indexes what it finds and ranks the best matches for a query; the ground on which a brand is found at the moment of intent.