Free AI crawler impact report
Upload an Apache, Nginx or CloudFront access log. The report lists the AI crawlers that read your site, the requests and bandwidth each one took, the pages they read most and when, and writes a robots.txt block for every AI crawler it found.
Your file is processed entirely in your browser. No data is uploaded to any server.
Drop your log file here
Apache, Nginx or CloudFront access logs as .log, .txt, .gz or .zip
Recognised AI crawlers
- GPTBot
- ClaudeBot
- PerplexityBot
- Bytespider
- CCBot
- Amazonbot
- Applebot
- Meta AI
How the report works
What it reports
How many requests came from AI crawlers and what share of your traffic that is, how much bandwidth you served them, which bots came and how often, the pages they read most, when they crawled, and a robots.txt reflecting what you have just seen.
Where the data comes from
Your own server log, read in this browser tab. Nothing is uploaded, nothing is stored, and no account is needed. Close the tab and the report is gone.
What an access log records
Every request your site answered, including the ones from crawlers, which fetch pages without running the JavaScript that analytics depends on. AI crawlers now read sites to train models and to answer questions about them, and the log is where that traffic shows up.
AI crawler breakdown
| Bot name | Category | Requests | % of total | Bandwidth | Unique IPs | Top paths |
|---|
Crawl timeline
Paths AI crawlers requested most
Generated robots.txt
Edit this before you use it. You may want to allow some AI crawlers for discovery and disallow the ones that consume the most bandwidth.
Want continuous AI crawler monitoring?
Salience monitors your traffic in real time, verifying bot claims against official IP ranges and alerting you to changes.
Common questions
Does my log file leave my machine?
The file is parsed in your browser and stays on your machine. That is why the report appears with no waiting and no sign-up, and why nothing about your log reaches us.
Which log formats work?
Apache and Nginx access logs, and CloudFront standard and real-time logs. The format is detected automatically. Plain .log and .txt files work, and so do .gz and .zip archives, which are unpacked in the browser.
How large a file can I use?
Large ones are fine. Files over 50MB are read in chunks so the tab stays responsive, and compressed archives are handled the same way. A month of logs from a busy site is a normal input.
Which crawlers does it recognise?
52 signatures in total, 13 of them AI crawlers: GPTBot, ChatGPT User, OpenAI SearchBot, ClaudeBot, Claude Web, Anthropic, PerplexityBot, Bytespider, CCBot, Amazonbot, Applebot, Meta AI and Cohere AI. The rest cover search engines, social preview bots, monitoring services and scrapers, reported separately so you can see AI traffic in context.
Does it check the bots are genuine?
The report groups requests by the user-agent each one claims, which is also how robots.txt rules address them. To confirm that an address claiming to be Googlebot really belongs to Google, paste it into the fake Googlebot checker, which verifies it against Google's published ranges. Salience does that verification continuously, against official IP ranges.
Should I block the crawlers it finds?
That is a decision per crawler. Some AI crawlers bring referral traffic and citations, others only consume bandwidth. The generated robots.txt lists every AI crawler seen in your log with a Disallow rule, as a starting point to edit. robots.txt is a request that well-behaved crawlers honour; a crawler that ignores it is one to handle at the edge.
Why does my analytics show none of this?
Analytics runs on JavaScript in a browser, and crawlers fetch the page without running it, so they leave no trace there. A server log records the request itself, which is why it is the complete account of who is reading your site.
How do I monitor this continuously?
This report is a snapshot of the log you hand it. Salience does the same job continuously against live traffic, verifies bot claims against official IP ranges, and alerts you when something changes.
Monitor this traffic continuously with Salience
Salience reads your CDN or server access logs as they arrive, verifies crawler claims against official IP ranges, and alerts you when something changes.