Observations

OpenAI shifted from live reads to crawling

Across around 1.5 billion server requests to the sites Salience monitors, verified ChatGPT-User fetches fell by about 80% between mid-August and mid-October, and verified OAI-SearchBot crawling more than doubled over the same weeks.

Carl Hendy

Carl Hendy, Co-founderPublished

~80%
fall in verified ChatGPT-User requestsmid-August to mid-October, weekly snapshots
~2x
rise in verified OAI-SearchBot requestsover the same weeks
~26x
growth in distinct sources running Claude CodeJune to September, mostly on technical documentation
~1.5bn
server requests reviewedacross the sites Salience monitors

There has been a run of studies on how much traffic AI assistants send to websites, from Marie Haynes, Kevin Indig, The SEO Works, Rand Fishkin and Michael King, on different datasets and with different methods. On referrals, the log data broadly agrees with them: AI referral traffic is small next to Google organic. Server logs also record a layer of activity that analytics tools never see, because it is bots fetching pages, and that layer changed shape over the last two months.

Where the data comes from

The figures are from server and CDN logs for the sites Salience monitors, around 1.5 billion requests between mid-August and mid-October 2026. Each OpenAI request is counted only when its source IP address falls inside the ranges OpenAI publishes for that bot, so a user agent string on its own is not enough. The weekly values are approximate snapshots, indexed so that the mid-August week equals 100 for each bot.

OpenAI user agentWhat it doesAppears in analytics
ChatGPT-UserFetches a page during a conversationNo
OAI-SearchBotCrawls pages in advance for the ChatGPT search indexNo
GPTBotCrawls for model training; not part of this comparisonNo

Live reads fell while crawling doubled

50100150200
20210ChatGPT-User live reads: 100, 87, 73, 60, 51, 45, 35, 27, 20. OAI-SearchBot crawling: 100, 116, 126, 136, 146, 160, 176, 196, 210.
Mid AugLate AugMid SepLate SepMid Oct
Verified weekly requests from each OpenAI bot across the sites Salience monitors, indexed to the mid-August week = 100. Approximate weekly snapshots, mid-August to mid-October 2026. Hover a point for the value.

ChatGPT-User requests fell in every one of the nine weeks and ended at about a fifth of their mid-August level. OAI-SearchBot requests rose in every week and ended at a little over twice their mid-August level. The two lines crossed in the second half of September.

The pattern is consistent with ChatGPT crawling and indexing content in advance and answering from that index, in place of fetching pages during individual conversations. The logs show the request mix, not the reason for it.

Claude Code reached far more sites

Over a longer window, June to September, the number of distinct sources making requests identified as Claude Code grew about 26 times. Most of those requests were for technical documentation: API references, install guides and changelogs. That traffic is a coding agent reading pages on a developer's behalf, and it reaches the server without a browser or an analytics tag.

OperatorAgentWindowChange
OpenAIChatGPT-UserMid-August to mid-OctoberVerified requests down about 80%
OpenAIOAI-SearchBotMid-August to mid-OctoberVerified requests up about 2x
AnthropicClaude CodeJune to SeptemberDistinct sources up about 26x

Referrals, agent activity and indexing are three different measures

On referrals, Marie Haynes' study of 100 sites over 12 months of GA4 data put ChatGPT sessions at 0.62% of Google organic volume, and 0.73% with every other assistant included. The SEO Works benchmark found 3.2 AI-referred conversions for every 100 from organic search. The log data across the monitored sites agrees that referrals from assistants are small next to Google.

An AI agent fetching a page does not mean a person saw a recommendation, and it does not mean a purchase. It does mean the page was read and could be quoted, indexed or acted on. A crawl by OAI-SearchBot means the page can appear in ChatGPT search results later. A referral means a person clicked through. Each one is reported from a different source and answers a different question, so they belong in separate columns.

What to do with this

  1. 01Report AI referrals, agent fetches and crawler indexing as three separate figures, each from the source that records it: analytics for referrals, server logs for the other two.
  2. 02Count bot requests only when the source IP is inside the ranges the operator publishes. OpenAI, Anthropic, Google and others publish them. Salience applies this check to every request it classifies.
  3. 03Watch which pages OAI-SearchBot crawls and how often. If ChatGPT is answering from its index, the index is what decides whether a page can be cited.
  4. 04Decide per bot what to allow. ChatGPT-User, OAI-SearchBot and GPTBot have different jobs, so one robots rule for all of them is rarely the right answer. The bot management guide sets out the choices.

Sources

Your logs already show what Google and the AI crawlers are doing.

Free plan, no card, about ten minutes to your first request.