The AI crawler landscape
The number of AI training crawlers has exploded. Each major AI company operates one or more crawlers that traverse the web to collect training data. Understanding who they are and how to verify them is the first step toward effective management.
By 2026 the useful question is no longer "is it an AI crawler" but "what is it doing here". The same operator runs different agents for different purposes, and your policy for each should differ.
| Crawler | Operator | Purpose | How to verify |
|---|---|---|---|
| GPTBot | OpenAI | Training | Published ranges |
| OAI-SearchBot | OpenAI | Search indexing (ChatGPT search) | Published ranges |
| ChatGPT-User | OpenAI | On-demand fetch when a user asks | Published ranges |
| ClaudeBot | Anthropic | Training | Published ranges |
| Claude-SearchBot / Claude-User | Anthropic | Search indexing / on-demand fetch | Published ranges |
| PerplexityBot / Perplexity-User | Perplexity | Answer-engine indexing / on-demand fetch | Published ranges |
| Google-Extended | Opt-out token for Gemini training (not a separate crawler) | Google ranges | |
| Googlebot | Search indexing, also feeds AI Overviews | Published ranges | |
| Applebot / Applebot-Extended | Apple | Siri and Spotlight / Apple Intelligence training opt-out | Published ranges |
| Amazonbot | Amazon | Alexa and Amazon AI | Published ranges |
| Meta-ExternalAgent | Meta | Training | Published ranges |
| Bytespider | ByteDance | Training | Reverse DNS / network consistency |
| CCBot | Common Crawl | Open web archive used for training | Reverse DNS |
Purpose beats name
Blocking GPTBot stops your content feeding the next model. It does not stop ChatGPT fetching a page on demand when a user asks about you, and it does not stop OAI-SearchBot indexing you for ChatGPT search. Decide per purpose: training, indexing and answering are three policies, not one.
Blocking Googlebot is not the same as blocking Google-Extended. Googlebot indexes your site for Google Search, and blocking it removes you from search results and from AI Overviews. Google-Extended is a robots.txt token that opts you out of Gemini training; it does not appear as its own crawler in your logs. You can set one without affecting the other.
Typical bot traffic breakdown
Understanding the normal composition of bot traffic helps you identify anomalies and prioritise management efforts. Here is a typical breakdown for a mid-traffic website.
| Bot category | Typical share |
|---|---|
| Search engines (Googlebot, Bingbot, etc.) | 30–40% |
| AI training crawlers | 15–25% |
| SEO tools (Ahrefs, SEMrush, Moz, etc.) | 10–15% |
| Social media (Facebook, Twitter, LinkedIn) | 5–10% |
| Monitoring services (UptimeRobot, Pingdom, etc.) | 3–5% |
| Feed readers (Feedly, NewsBlur, etc.) | 2–5% |
| Unknown / unverified scrapers | 10–20% |
Over 30% of traffic claiming a bot identity fails IP verification. This means nearly a third of requests that say they are Googlebot, Bingbot, or other known crawlers are actually coming from IPs that do not belong to those operators. Without IP verification, your bot analytics are significantly misleading.
Measuring crawl impact
Bots consume real infrastructure resources. Before making access decisions, measure the actual impact each crawler has on your servers.
Key metrics to track per bot
- Requests per day, the total volume of requests from each crawler
- Bandwidth consumed, total bytes transferred, which directly affects your CDN and hosting costs
- Response time impact, whether bot traffic is slowing down response times for real users
- Error rates, a bot generating high error rates may be hitting stale URLs or causing origin load
- Peak request rate, the maximum burst rate (requests per minute) during peak crawling periods
If a single bot accounts for more than 20% of your total bot traffic, investigate immediately. This level of crawling is excessive for almost any site and may indicate aggressive scraping, misconfigured crawl rates, or bot impersonation.
Robots.txt strategies by site type
There is no one-size-fits-all robots.txt policy. The right strategy depends on your business model, content type, and values.
| Site type | Recommended approach | Rationale |
|---|---|---|
| News / media | Block AI training crawlers | Original reporting has direct commercial value; allowing AI training risks displacing the source content in AI-generated answers |
| E-commerce | Selective approach | Product pages may benefit from AI visibility, but catalogue scraping is a concern; allow selectively and monitor closely |
| SaaS documentation | Allow most AI crawlers | Having your docs referenced by AI assistants drives awareness and helps users find your product |
| Blog / publisher | Values-based decision | Weigh the trade-off between AI visibility (potential traffic from AI citations) and content protection (training on your work without compensation) |
Your robots.txt decisions should be data-driven. Use Salience to measure how much each AI crawler actually consumes before deciding whether to block it. A crawler making 10 requests per day is very different from one making 10,000.
Compliance monitoring
Robots.txt is a voluntary standard. Declaring a Disallow rule does not technically prevent a crawler from accessing your pages, it only asks them not to. Monitoring compliance is essential.
Verify that blocked crawlers actually stop
After updating your robots.txt, check your server logs over the following days and weeks to confirm that the blocked crawlers have stopped visiting the disallowed paths. Salience makes this easy by showing per-bot request history with path breakdowns.
Watch for non-compliant crawlers
Some crawlers, particularly less reputable ones, ignore robots.txt entirely. If you see continued crawling from a bot you have blocked in robots.txt, you will need to escalate to IP-level blocking at your CDN or firewall.
Review quarterly
The AI crawler landscape changes rapidly. New crawlers appear regularly, and existing ones may change their behaviour. Review your robots.txt policy and actual crawl data at least once per quarter to ensure your rules still reflect your intentions.
Robots.txt rules are public. Anyone can read your robots.txt to see which paths you consider sensitive or valuable. Avoid using Disallow rules as a security mechanism, they are a content access policy, not an access control system.
What changed in 2026
Verification is a ladder, not a list
Published IP ranges cover the big operators. For everyone else, verify by reverse DNS with a forward-confirm, then by network consistency (the crawler has always come from the same autonomous system), then by edge signals. A verification status should say which rung it reached. Pending and unverified are not the same as fake, and neither should be counted as real crawl.
The give-and-take ratio
For each AI operator, count the pages it fetched and the human visits it referred back (the referrer is chatgpt.com, perplexity.ai, claude.ai, gemini.google.com or copilot.microsoft.com, or the link carries a utm_source naming one). Fetches per referred visit is the number to put in front of whoever decides your robots.txt. An operator fetching ten thousand pages to send you three people is a different negotiation from one sending you three hundred.
Check the block actually landed
Robots.txt is a request. A WAF rule is an action. Your logs show whether either worked: after a change, look at the crawler's requests split into served and rejected, and split the rejected share by verification. If the verified crawler is still being served on paths you disallowed, the operator ignored you. If your rule is rejecting the verified crawler while the impersonators sail through, the rule is keyed on the wrong thing.
New agents appear weekly
A static list of user agents is out of date within a month. The workable approach is a registry: every named agent seen in your logs, with its operator, purpose and how it was verified, and a nightly pass that surfaces new candidates for review. Salience keeps that registry across every site it monitors and promotes documented, verifiable crawlers automatically, so a new agent is classified the week it appears rather than the quarter you notice it.
Take the whole guide with you
Free PDF
The Complete Guide to Log File Analysis
SEO, security, bot management, AI crawlers and traffic intelligence in one 2026 edition. Everything in these guides, plus the checklists, in a PDF you can hand to your team.