See what every bot and crawler does on your Cloudflare site
Salience reads the request data Cloudflare already collects for you and turns it into verified bot identities, crawl and error analysis, and alerts that name the IPs, paths and user agents involved, without Logpush, an Enterprise plan or a single change to your website.
Any Cloudflare plan, including Free. Live in about ten minutes with no change to your website, and the visitor's response is never held.
The Log Explorer page for demo-site.example once Cloudflare is connected through Cloudflare Worker: the Website setup strip shows Logs Receiving, and the table fills with Googlebot requesting /products/wireless-headphones, status 200, Verified; GPTBot requesting /blog/pricing-guide, status 200, Verified; Googlebot (claimed) requesting /wp-login.php, status 403, Unverified; Bingbot requesting /old-product-url, status 404, 404 spike. First hour on the zone: 38,412 requests.
Cloudflare records every request, and most of that data goes unread
Your zone answers millions of requests a month and records what happened to each one. Getting an answer out of that record is the part nobody has time for, so most teams end up doing some version of this.
Reading the analytics charts, seeing that bot traffic is up, and having no way to find out which bots, on which paths, or whether they were even genuine.
Taking a user agent at its word. Anything can call itself Googlebot, and the cheapest scrapers all do, but checking a claim against published IP ranges by hand, at scale, is not a job anyone volunteers for.
Finding out about a broken deploy from Search Console two days later, when the crawl errors have already been collected and the ranking damage is done.
Asking an engineer to pull and grep the logs whenever a question comes up, which turns a five-minute question into a two-day ticket and means most questions never get asked.
See every request to your site, sorted by who sent it
Connect the zone and the stream starts filling immediately: humans, verified crawlers, AI agents and suspicious automation, separated and counted.
The Salience dashboard for demo-site.example, last 24 hours. Website setup 5 of 6 completed. Traffic over time in fifteen-minute buckets, human and bot requests stacked. Bot identity evidence: 12.1K verified requests, 762 unverified claims, 2.7K with verification unknown. Total requests 34.2K, up 7.0% on the previous period. Unique IPs 15.3K, up 2.0%. Bot traffic 45.5%, down 1.9 points. AI crawlers 18.5%, down 8.0 points.
Requests reach Salience within seconds of Cloudflare serving them and show in reports within about fifteen minutes. Anomaly detection begins once about 24 hours of baseline has built.
Check that a crawler is who it says it is
Every claimed identity is checked against the provider's own published IP ranges. Genuine Googlebot, Bingbot, GPTBot and ClaudeBot are separated from the scrapers borrowing their names from datacentre addresses, and the impostors are listed with the IPs responsible, so you can act on them rather than argue about them.
Bots and Crawlers on demo-site.example, last 24 hours: 15,541 bot requests, 12,233 verified, 655 failed verification, 2,653 other status. Served versus rejected: Googlebot 357 served, 34 rejected, Impersonators only, 0 verified · 34 unverified; Amazonbot 3,934 served, 16 rejected, The real bot, 16 verified · 0 unverified; Attack Path Probe 413 served, 104 rejected, Not verifiable, 0 verified · 104 unverified.
AmazonbotSee a broken deploy within the hour
Status codes are analysed per path and per site section against your own baseline. When a template change starts feeding Googlebot 404s on your product URLs, the spike is detected against that baseline and the alert arrives with the exact URLs attached, while a rollback is still cheap.
Status code detail for 4xx Client Error on demo-site.example, last 24 hours: Total Requests 3,272, % of All Traffic 9.58%, Unique Codes 4. Individual status codes: 403 2,337, 71.4% of 4xx; 404 896, 27.4% of 4xx; 405 29, 0.9% of 4xx; 499 10, 0.3% of 4xx. Top paths: /old-product-url 155, /blog/2019-relaunch 142, /guides/pricing-2025 119, /products/legacy 98, /sitemap-old.xml 90, /blog/page/48 42.
Each alert says what to do about it
Every alert names the affected IPs, user agents and paths, and carries a written triage note: the likely issue and a recommended next step. Not "something happened, check your logs". Salience reports and recommends; enforcement stays in your Cloudflare WAF, where you control it.
The Alerts page for demo-site.example: 2 open · 8 cleared · 10 alerts. Error spike · critical: 15,000 404s on /products/* from Googlebot since the 14:00 deploy. Next step: Roll back or fix the URL structure. Crawler blocked · warning: ClaudeBot refused with 403 on 32 URLs under /collections since the WAF rule change. Next step: Review the rule if you want those pages in AI answers.
How it works
Connect
Add your domain in Salience, deploy the generated Worker on the Cloudflare zone that serves it and set the route's failure mode to Fail open. Guided instructions and a copy-ready script are in the app; your site's code is untouched.
We analyse
Each request is classified against the crawler registry, verified against published provider ranges, grouped by URL section and compared with your own history, sitemap, robots.txt and Search Console data.
Get answers
Dashboards, alerts by email, Slack or webhook, plain-English querying, an API and a CLI. The connection confirms within seconds, and the first reports fill within about fifteen minutes.
Specification
- Collection method
- A Salience Worker on your Cloudflare zone
- Data freshness
- Within seconds
- Typical setup time
- About 5–10 minutes
- Plan requirement
- Any Cloudflare plan, including Free. Logpush and Enterprise are not needed
- Permissions needed
- Access to deploy a Worker and add a route on the zone that serves your domain
- Code or DNS changes
- None to your website. No DNS change if your records are already proxied
- Historical import
- Not from Cloudflare. Apache, Nginx and CloudFront logs and other log-analyser exports import alongside
- Key limitation
- The Worker only observes requests that reach the Worker stage on proxied (orange-cloud) records. Traffic Cloudflare rejects before Workers execute, and any grey-clouded DNS-only records, are not captured.
Which of these is you?
SEO teams
I deployed template changes yesterday and I have no idea whether Googlebot has seen them, or what it got when it tried.
Crawl activity by section, crawl waste, and error spikes caught against your own baseline: the same afternoon, not after the next Search Console refresh.
Search intelligence
AI and content teams
Which AI systems are actually reading our content, is it training or retrieval, and are they even genuine?
GPTBot, ClaudeBot, PerplexityBot and the rest, separated into training crawls, AI-search indexing and user-triggered retrieval, with impostors flagged.
AI crawler and agent intelligence
Security teams
Everyone claims to be Googlebot. Half of them aren't. Meanwhile the real one is being rate-limited by my WAF.
Probe patterns, exposed-secret fetches and impersonation surges surfaced with named source IPs, escalated when a probe returned a 200.
Threat intelligence
Engineering and site reliability
Attack traffic took the site down and our infrastructure alerts reported high CPU, not an attack in progress.
Error-rate, latency and traffic-blackout detection against section-level baselines, with the clients and paths responsible named in the alert.
Site reliability
Doesn't Cloudflare already do this?
In part, and the parts it does it does very well. Cloudflare gives you zone analytics, Instant Logs for live tailing, Log Explorer and Logpush on the higher plans, Bot Management with its own scoring, a verified-bots list and AI Crawl Control. If you are on Enterprise and you have engineers who enjoy building on top of Logpush, you can construct a great deal of this yourself. The difference is not that Cloudflare's data is lacking (it is Cloudflare's data that Salience reads). It is what happens to that data after it is collected, and on which plans.
Per-request data on any plan
- Cloudflare's native tools
- Log Explorer and Logpush are higher-plan features
- Salience
- Any plan, including Free
Bot identity verified against published IP ranges
- Cloudflare's native tools
- Verified bots and bot scoring, strongest on Bot Management plans
- Salience
- Included, across 200+ crawler identities
Search and AI crawler interpretation
- Cloudflare's native tools
- Bots are classified, not read as search or AI behaviour
- Salience
- Crawl budget, index context, training versus retrieval
Joined to your sitemap, robots.txt and Search Console
- Cloudflare's native tools
- Out of scope: Cloudflare observes Cloudflare
- Salience
- Correlated, so a drop links back to the change that caused it
Anomaly detection with written triage
- Cloudflare's native tools
- Alerting is available, tuned to security and availability events
- Salience
- 16 detectors on a 5-minute cycle, each naming IPs, paths and a next step
Enforcement: blocking, rate limiting, WAF rules
- Cloudflare's native tools
- This is Cloudflare's job, and it does it properly
- Salience
- Not attempted. Salience evidences; you enforce
When the native tooling is all you need
If your question is "is this request malicious, and should I block it right now?", Cloudflare's own tooling is the right answer and Salience does not replace it: we deliberately do not block, rate-limit or change your configuration. Salience is for the questions that come after: who was that, what were they doing across the whole site, is this new, and what should we do about it.
Facts about the integration
No customer logos and no five-star quotes on this page. These are properties of the product you can check on the free tier in an afternoon.
Illustrative, not a customer incident
A template change at 14:00 rewrites your product URLs. By 14:03 the first Googlebot 404s on the old paths are in the stream; at 14:20 the crawler error-spike alert fires with the exact URLs listed and a recommendation to roll back or ship redirects. The alternative (a nightly log export, a report at nine the next morning) is the same problem, twenty-one hours later.
Everything included
One connection, one request stream. No per-feature setup and no second pipeline to maintain.
Verified bot and AI crawler identities
Claimed identities checked against official IP ranges where providers publish them; anything unverifiable is marked unverified rather than guessed at.
Crawl budget by section
Where Googlebot's time actually goes (real content, admin paths, redirect loops, 404 ghosts), broken down by directory and URL pattern.
Which AI systems read your pages
Training crawls, AI-search indexing and user-triggered retrieval separated, so robots.txt and licensing decisions rest on verified data.
Error, latency and status analysis
2xx through 5xx per path and section, with spike detection against your own baseline rather than a fixed threshold.
Threat and probe detection
sqlmap-style requests, /.env and config fetches, credential-stuffing patterns and impersonation surges, escalated if anything returned a 200.
Alert detectors on a five-minute cycle
Traffic-drop alerts check the site is reachable before they fire, probe alerts check whether the flagged path really serves anything, and every alert is enriched first, so you never get "your site might be down" when it is not.
Sitemap, robots.txt and Search Console alongside
Sitemap, robots.txt and Search Console joined to the request stream, so a ranking change links back to the deploy or policy edit behind it.
Alerts by email, Slack and webhook
Dashboards, email and Slack alerts, webhooks, a public API, a CLI, an MCP server, CSV exports and shared read-only dashboards.
Import older logs
Apache, Nginx and CloudFront logs, and exports from other log analysers, upload, deduplicate against streamed data, and never count against your request quota.
Recommendations with a rule for your Cloudflare WAF
Addresses to block and fake crawler user agents, scored from the last 30 days of alerts, with a WAF custom-rule expression generated from the list and ready to paste.
Saved segments
Page groups you define by prefix, pattern or query string, with a one-click library of common groups, applied to all history and used in every report, alert and export.
Site checks every night
Twenty-one checks on crawler access, security hygiene and serving quality, each pass, warn or fail with the evidence, 30 days of history and an alert when a verdict changes.
Weekly report and daily digest
A weekly email report per site, a daily digest of lower-severity alerts, and email for anything above the severity you set.
Trust & data protection
Privacy and data protectionYou are the controller
We process only on your instructions. GDPR Art. 28 DPA on every account, nothing to sign.
UK data residency
AWS eu-west-2 (London). Encrypted in transit (TLS 1.2+) and at rest (AES-256).
Server-side collection
No browser tracking script and no client-side pixel.
No sale, no pooling
Your logs are never sold, never used for advertising, never shared between customers. DPA, sub-processor list and security overview available.
What is collected
- Timestamp, method, host and path
- Status code and response size
- Client address and user agent
- Referrer, country and cache status
What is never collected
- Request and response bodies
- Cookie contents
- Anything Cloudflare turned away before the Worker ran
Priced on requests, not seats, so connecting a Cloudflare zone never costs you per colleague. Start on the free tier with 500,000 requests a month and upgrade when your traffic does.
Crawler intelligence from $19/mo.
Priced on requests, not seats. Unlimited users on every plan except Solo. Start on the free tier and upgrade when your traffic does. Free trial, no card needed.
- 500K requests/mo
- 1 site
- 30 days history
- Unlimited users
- Real-time analytics, bot and AI detection
- 20M requests/mo
- 5 sites (+5)
- 1 year history
- Unlimited users
- Real-time analytics with bot and AI-crawler verification
- 100M requests/mo
- 15 sites (+15)
- 2 years history
- Unlimited users
- AI allowance: ~600 answers or ~60 reports a month
- 500M requests/mo
- 50 sites, no ceiling
- 4 years history
- Unlimited users
- AI allowance: ~3,000 answers or ~300 reports a month
- 1B requests/mo
- Unlimited sites
- Custom history
- Unlimited users
- SSO / SAML and audit log
Common questions
We already have Log Explorer and Bot Management. What does this add?
Interpretation, and plan reach. Cloudflare tells you a request happened and scores how bot-like it was; Salience tells you which named crawler it was, whether the claim verified against published IP ranges, what it did across your whole site, whether that is new against your baseline, and what to do next, joined to your sitemap and Search Console, and available on any Cloudflare plan including Free.
Do I need Cloudflare Logpush or an Enterprise plan?
No. The Worker-based integration works on any Cloudflare plan, including Free. Logpush is a separate Cloudflare product for bulk delivery to storage and SIEM tools, and Salience does not depend on it. The one limit to know about is the Workers Free plan's allowance of 100,000 Worker requests a day across your whole Cloudflare account; Workers Paid, $5 a month, includes 10 million requests and removes the daily cap.
Will the Worker slow my website down?
Your visitor's response is never held while data is sent to Salience. The Worker itself adds a small amount of edge compute per request, as any Cloudflare Worker does, and those requests count against your Workers plan's allowance.
What does the Cloudflare free plan allow?
Cloudflare's Workers Free plan allows 100,000 Worker requests a day across the whole account, resetting at 00:00 UTC. The Worker runs once for every request to the hostnames it is routed on, including images, scripts and bot traffic, so a busy site can use the allowance in a few hours. When it runs out, Cloudflare stops running the Worker until midnight and Salience stops receiving logs for the rest of the day; the site itself stays up as long as the route is set to Fail open. Workers Paid, $5 a month, includes 10 million requests and removes the daily cap (developers.cloudflare.com/workers/platform/pricing/).
Can this take my site down?
Not if every route the Worker is on has its failure mode set to Fail open, which is why the setup asks you to check it. Fail open means that if the Worker cannot run, because the daily allowance is used up or it errors, Cloudflare serves your site directly, so your site can never go down because of this Worker. Fail closed would return errors instead. The setting is in Cloudflare under Workers & Pages, then the Worker, then Settings, then Domains & Routes: edit each route and set Failure mode.
Do we need a developer, and how long does setup take?
About five to ten minutes, and someone with permission to deploy a Worker and add a route on the zone. There are no website code changes and no DNS changes if your records are already proxied. If setup needs to go to an engineer, you can delegate it from inside Salience without sharing your account.
Is this website request data or my Worker application logs?
Website request data: the requests Cloudflare proxies for your domain. Salience does not collect console.log output or exceptions from Worker scripts you have written yourself.
Can Salience tell genuine Googlebot and AI crawlers from fakes?
Where providers publish verification data, yes. Claimed identities are checked against official IP ranges, so verified Googlebot, Bingbot, GPTBot and ClaudeBot traffic is separated from scrapers borrowing their user agents. Bots without published ranges are classified but explicitly marked unverified rather than guessed at.
Does Salience block suspicious bots?
No, by design. Salience identifies and evidences them (IPs, user agents, paths, verification status) and recommends action. Blocking happens in your Cloudflare WAF or firewall rules, where you stay in control and keep one place to audit.
What happens to our data, and where is it stored?
You are the controller; Salience processes only on your instructions under a GDPR Art. 28 DPA that applies to every account. Data is stored encrypted in AWS eu-west-2 (London), never sold, never used for advertising and never pooled between customers.
What if we move off Cloudflare later?
Remove the Worker route and collection stops immediately. Your history stays in Salience, and you can connect a different source (Vercel, CloudFront, Apache, Nginx or a log import) into the same site without starting over, because every source normalises to the same request model.
Can we see data from before we connected?
Not from Cloudflare itself, which does not expose past requests to a Worker. You can import historical Apache, Nginx or CloudFront logs, or exports from other log analysers, alongside; they deduplicate against streamed data and do not count against your quota.
Your logs already show what Google and the AI crawlers are doing.
Any Cloudflare plan, including Free. Live in about ten minutes with no change to your website, and the visitor's response is never held.