- +5,271%
- ExaSearchBot, week of 12 Octoberagainst its own average week, 10 to 30 August
- +782%
- OAI-SearchBot, week of 12 Octoberagainst its own August average
- +339%
- PerplexityBot, week of 12 Octoberagainst its own August average
- 3 of 3
- crawlers fell, 19 to 24 Octoberby 15% to 24% against the matching weekdays, 12 to 17 October
Exa, OpenAI and Perplexity each run a named crawler that fetches pages in advance for an AI search index. This piece compares how much each one's activity changed over ten weeks across the same set of sites. Each crawler is measured against its own August level, so the figures describe growth, not size. Exa started from the smallest base and grew by the largest multiple.
Where the data comes from
The figures come from the Salience production log archive, queried on 27 October 2026. The sites are a fixed cohort chosen for sustained total traffic in every full week of the study, without reference to which crawlers appeared on them. Internal Salience sites, a test domain and historical site IDs that could not be matched to the current registry were excluded. The same cohort, date windows and matching method were used for every crawler.
Each crawler's baseline is its own average weekly activity over 10 to 30 August 2026. Percentage change is weekly activity divided by that baseline, minus one. Weeks start on the Monday date shown, in UTC. Zero means equal to the baseline, not no traffic, and individual August weeks can sit above or below zero because the baseline is an average.
Requests were matched on the raw user-agent name, case-insensitively and on product-name boundaries. Unlike the OpenAI observation, this comparison does not check source IP addresses against the ranges each operator publishes, so a request that claims to be one of these crawlers is counted as one. User-agent strings can be spoofed.
| Crawler | Operator | What it does | In this comparison |
|---|---|---|---|
| ExaSearchBot | Exa | Crawls pages for the Exa search index | Yes |
| OAI-SearchBot | OpenAI | Crawls pages for the ChatGPT search index | Yes |
| PerplexityBot | Perplexity | Crawls pages for the Perplexity search index | Yes |
| ExaUser | Exa | Fetches a page when a person's query needs it | No, user-triggered |
| Exabot | Not Exa | A different crawler with a similar name | No |
Training crawlers and user-triggered fetchers from all three operators are left out, so each series is one named search crawler and not all traffic from its operator.
Growth against each crawler's own baseline
All three crawlers sat close to their baseline through August. ExaSearchBot moved first, rising to around 15 times its August level in the week of 14 September, dipping in the week of 28 September, then climbing in each of the last two weeks to 53.7 times. OAI-SearchBot rose in most weeks from early September and ended at 8.8 times. PerplexityBot peaked in the week of 14 September, fell back, and ended at 4.4 times.
The chart shares one linear scale, so the Exa line dominates it. Its growth is the largest multiple because its August baseline was the smallest of the three. The series say nothing about which crawler made the most requests, reached the most pages or produced the most citations.
The most recent weeks
| Comparison | ExaSearchBot | OAI-SearchBot | PerplexityBot |
|---|---|---|---|
| 12 to 18 October against 5 to 11 October | +58% | +67% | +108% |
| 19 to 24 October against 12 to 17 October | −15% | −24% | −19% |
In the latest full week, PerplexityBot grew fastest of the three, not Exa. In the six matched weekdays after it, all three crawlers fell. The second row compares partial weeks on matching days and is not plotted on the chart. Archived data after 24 October was incomplete when the analysis ran, so the series stops there and this is not a live traffic report.
What the figures do not show
- Traffic volume, market share, citations or the number of unique pages fetched. A faster percentage rise from a small base is not more requests.
- The median site. Aggregated activity is concentrated on some sites in the cohort, so the growth of a typical site is not the growth shown here.
- The web as a whole. The cohort is an observational sample of sites that use Salience, not a representative survey.
- Verified identity. Matches are on the user-agent name, and the original log events were not independently deduplicated.
- Uninterrupted capture. Fixing the cohort controls which sites are counted, not whether every request on them was recorded, and it does not adjust for changes in each site's ordinary traffic.
- Request counts, fleet sizes or site identities. Only percentage changes are published.
What to do with this
- 01Look for ExaSearchBot, OAI-SearchBot and PerplexityBot in your own server logs. None of them appears in script-based analytics, so the logs are the only record of what they fetched.
- 02Decide per crawler whether to allow it. Each operator documents its crawler and how to control it in robots.txt, linked under Sources. The bot management guide sets out the choices.
- 03Where an operator publishes IP ranges, count a request as that crawler only when its source address is inside them. Salience applies this check to every request it classifies, including the ones in this comparison once they reach the product.
- 04Track each crawler as its own series against its own baseline. A single AI crawler total hides a 54 times rise in one crawler behind steadier figures from the others.
Sources
- Exa crawler documentation, the published behaviour of ExaSearchBot and ExaUser.
- OpenAI crawler and user agent documentation, the published behaviour and IP ranges for OAI-SearchBot, ChatGPT-User and GPTBot.
- Perplexity crawler documentation, the published behaviour of PerplexityBot and Perplexity-User.