Cloudflare server log monitoring
See every crawler, bot and error on your Cloudflare site.
Cloudflare's dashboard shows traffic totals and what your rules blocked. Salience shows each Googlebot request, AI crawler, scraper and error response that reaches your site, with crawler identities checked.
- Any Cloudflare plan, including Free
- No change to your site's code
- Set up in about 10 minutes
Log Explorer
docs.example · requests in real time
GooglebotVerified/guides/webhooks200
OAI-SearchBotVerifiedAI search/api/reference/payments200- GPTBotUnverified/guides/rate-limits200
BingbotVerified/v1/guides/auth301
ClaudeBotVerifiedTraining/changelog200- GooglebotVerified/search?q=retry200
Requests to /guides/* and /api/* from 198.51.100.61, an address outside OpenAI's published GPTBot ranges.
Recommended: review the address in Recommendations and add the generated WAF rule, which challenges the fake bot.
Cloudflare Analytics and Salience answer different questions
Cloudflare's dashboard reports the zone's traffic as totals and lists the requests your security rules acted on. Salience records each request that passes those rules and reaches the Worker, and reads it for crawler identity, status code and request pattern.
Cloudflare Analytics and Security Events
Zone traffic totals and the actions of your security rules
- How many requests did the zone serve, and how many came from cache?
- Which requests did a WAF, rate limiting or bot rule block or challenge?
- Which countries, paths and status codes make up the traffic?
- Is an attack under way, and is a rule stopping it?
Blocking, challenging and rate limiting stay in Cloudflare.
Salience
Every request that reaches the Salience Worker on your zone
- Which crawlers requested which URLs, and were they who they said they were?
- What status code and cache status did each request get back?
- Which AI systems request your pages, and which pages do AI assistants send people to?
- Which automated sources got past your rules, and what are they requesting?
Salience suggests WAF rules for you to paste and applies none itself.
Questions your Cloudflare dashboard can't answer
Four examples from the four teams that use Salience on a Cloudflare site. The site, addresses and figures in the panels are examples.
01Search and SEO
Is Googlebot reaching the pages you want indexed?
Internal search results, parameter URLs and old versions of pages can take a large share of Googlebot's requests while new pages wait. Cloudflare's analytics report the zone's traffic by path, country and status, with no comparison against your sitemap.
Salience records every verified Googlebot request with the URL, the status code, the cache status and the time, groups the requests by section, and compares the URLs Googlebot fetched with the URLs in your sitemap.
Find sitemap URLs Googlebot has not fetched, see how much crawling goes to search and old-version URLs, and see error responses to Googlebot on the day they start.
Verified Googlebot by section
docs.example · last 7 days
| Section | Requests | URLs | Unfetched |
|---|---|---|---|
| Guides/guides/* | 3,904 | URLs fetched812 | Sitemap URLs not fetched27 |
| API reference/api/* | 2,177 | URLs fetched1,460 | Sitemap URLs not fetched0 |
| Old versions/v1/* | 1,688 | URLs fetched1,215 | Sitemap URLs not fetchedNot in sitemap |
| Search results/search?q= | 1,021 | URLs fetched977 | Sitemap URLs not fetchedNot in sitemap |
Automated sources that reached the Worker
docs.example · last 24 hours
Requests per hour from unverified automation
peak 1,962
Unverified automated requests ran at about 65 an hour, rose to between 1,874 and 1,962 an hour from 13:00 to 18:59, then returned to about 65.
- 203.0.113.0/24Not a crawler9,612
Hosting provider · 3 rotating browser user agents · Every /api/reference/* page in order, no images or scripts
- 198.51.100.61Unverified2,248
Hosting provider · GPTBot (claimed) · /guides/* and /api/*, about 40 a minute
- GooglebotVerified1,431
Google · Googlebot · Guides and API reference, follows robots.txt
- 192.0.2.140Not a crawler873
Hosting provider · Go-http-client/1.1 · /search?q= with sequential terms
02Security and bot threats
Which automated traffic is getting past your Cloudflare rules?
Scrapers rotate browser user agents or borrow a crawler's name from a datacentre address. A request that matches none of your WAF or bot rules is served like any other.
Salience groups the requests that reach the Worker by source address and user agent, checks every claimed crawler name against the operator's published addresses, and shows each source's paths, request rate and timing.
Recommendations list the addresses and fake crawler user agents and generate a Cloudflare WAF custom-rule expression for you to paste: a block for the addresses and a challenge for the fake bots.
03AI search and discovery
Which AI systems are requesting your pages?
AI companies run separate crawlers for collecting training data, for building an AI search index and for fetching a page when someone asks an assistant a question. Each one is a separate decision in robots.txt or in your Cloudflare rules.
Salience puts every AI crawler in one of those three classes, verifies it where the operator publishes its addresses, and lists the pages each one requested.
A recorded request shows that a page was fetched. It does not show whether an answer used or cited the page.
The AI referral traffic report counts the human visits that arrive from ChatGPT, Perplexity, Claude, Gemini, Copilot and other assistants, by landing page.
AI crawlers by purpose
docs.example · last 7 days · verified requests
- GPTBot1,846
- ClaudeBot1,163
Most requested /guides/webhooks
- OAI-SearchBot534
PerplexityBot377
Most requested /api/reference/payments
- ChatGPT-User318
- Claude-User64
Most requested /guides/rate-limits
04Engineering and operations
Did a Cloudflare rule change start turning crawlers away?
A new WAF custom rule, rate limiting rule or bot setting can refuse Googlebot or AI crawlers while the site loads normally in a browser. Requests a Cloudflare rule refuses before Workers run never reach the Worker.
Salience tracks each verified crawler's daily requests and status codes against your own history, and Site Events mark your releases and rule changes on every chart. The Crawler Rate Dropped alert fires when a crawler that normally visits stops or sharply reduces its visits, and the Edge Policy Change alert fires when 401, 403 or 429 responses to crawlers rise.
Each alert email explains what changed, lists the top URLs or addresses involved and links to the dashboard at the time the alert fired.
Verified crawler requests reaching the Worker
docs.example · daily, % of 14-day average
- Verified search crawlers
- 103% 3.1%
- Verified AI crawlers
- 100% 46.8%
- Human visitors
- 100% 99.2%
- Verified search crawlers
- Verified AI crawlers
- Human visitors
- Site event · WAF custom rule deployed
Googlebot and Bingbot requests fell to 3.1% of their usual daily level, and verified AI crawler requests to 46.8%. Human visitors are at their usual level.
Recommended: check Cloudflare Security Events for the rules that acted on verified bots today, and exclude verified bots from them.
Example investigation
Googlebot started receiving 503s on 10 guide pages
An example of how an alert becomes a fix, step by step. The site, URLs and figures are illustrative.
- 1
The alert fires
At 15:10 a Crawler Error Spike alert reports that verified Googlebot requests to /guides/ URLs are returning 503 far above the site's usual rate.
- 2
The crawler is Google's
Every request came from an address inside Google's published Googlebot ranges, so these are Google's own fetches receiving a server error.
- 3
The affected URLs
87 requests across 10 guide URLs. Every response carries the cache status BYPASS, so the origin answered each one.
- 4
When it started
The first 503 arrived at 14:01, 11 minutes after a Cloudflare cache rule change your team recorded as a Site Event at 13:50. The errors continued into the evening.
- 5
The next step
Restore caching for /guides/* in the Cloudflare cache rule, then check that Googlebot receives 200 responses on its next visits to the 10 guides.
Crawler Error Spike
docs.example · opened 15:10 · Googlebot · 503
GooglebotVerified66.249.79.0/24 · in Google's published ranges- /guides/authentication50315
- /guides/pagination50313
- /guides/idempotency-keys50311
- /guides/webhooks/signature-verification50310
- /guides/errors-and-status-codes5039
- /guides/sandbox-testing5038
- 4 more URLs · 21 requests
13:50 Site event · cache rule changed
No 503s in the 13:00 hour, when the cache rule changed at 13:50; then 4, 12, 17, 15, 13, 10, 8, 5 and 3 per hour from 14:00 to 22:00, 87 in total.
Recommended next step
Restore caching for /guides/* in the Cloudflare cache rule changed at 13:50, then confirm Googlebot receives 200 on the 10 guide URLs.
What Salience shows for a Cloudflare site
Every report comes from the same request data. Search Console and Google Analytics 4 add search and session figures to the same URLs if you connect them.
URL detail
docs.example/guides/webhooks · last 28 days
/guides/webhooks
- LogsVerified Googlebot fetchesLast 200 at 09:4718
- LogsAI page fetchesChatGPT-User, Claude-User24
- LogsAI-referred visitschatgpt.com, claude.ai19
- Search ConsoleImpressions, 28 daysIndexed, last inspected 8 Oct4,260
- Google AnalyticsSessions, 28 daysOrganic search 64%905
Who is requesting
Verified crawler identities
200+ crawler identities, checked against published address ranges where the operator publishes them.
AI crawlers and AI referrals
Training, AI search and user-triggered fetches separated, with AI-referred visits by landing page.
Scraper and automation patterns
Systematic requests grouped by address, network and user agent.
What they request
Every request, URL by URL
Log Explorer and a URL page for each path, with status, cache status, client, timing and source.
Sections, segments and parameters
Page groups you define by prefix, pattern or query string, or add from a library of common groups, applied to all history.
Sitemap and robots.txt
Checked automatically when the site is added, with crawl activity set against sitemap coverage.
What happens next
Status codes and errors
Status codes per path, soft 404s and redirects against your own baseline.
Alerts and recommendations
Alerts by email, Slack or webhook with a recommended next step, and a Cloudflare WAF rule generated from the recommendations.
Reports
A weekly email report, a daily digest and a written technical SEO report.
Salience also runs its own site crawler and crawl audit, and has a public API, a CLI and an MCP server.
How to set up server log monitoring on Cloudflare
Setup takes about 10 minutes. The app installs the Worker for you or walks you through each step, and checks the connection at the end.
- 1
Enter your domain and create your account
Use the domain your visitors see, such as yoursite.com. The Free plan needs no card.
- 2
Deploy the Salience Worker on your zone
Choose automatic setup with a Cloudflare API token limited to the site's zone, or follow the steps in the app: create the Worker, store your ingest key as a secret, and add routes for yoursite.com/* and www.yoursite.com/* with Failure mode set to Fail open.
- 3
Confirm requests are arriving and check your site
The connection check sends three test requests to your site and confirms they reach Salience. Then open your site in a browser and check its pages still work. From then on, requests appear in Log Explorer within seconds.
What you need
- A site on a Cloudflare zone, on any plan including Free
- The site's DNS records set to Proxied (orange cloud)
- Permission to deploy a Worker and add routes on the zone
What stays the same
- No change to your site's code
- No DNS change when the records are already proxied
- Your security, cache and bot rules run as before
Before you start
- Setup checks whether another Worker already handles the site, and stops without changing anything if one does
- Set Fail open on every Worker route. It lets requests through to your origin if the Worker's daily allowance runs out
- Collection covers requests on proxied records that reach the Worker, after your security rules have run
- History starts when collection is connected
Someone else manages Cloudflare? Send them a delegated setup link from the app. It is limited to this site for 30 days, and they complete the Worker steps without access to your Salience account. Your AI agent can also do the setup from a prompt the app gives you.
How your request data is collected and controlled
What is collected
- Timestamp, method, host and path
- Status code and response size
- Client address and user agent
- Referrer, country and cache status
What is not collected
- Requests a Cloudflare rule refuses before the Worker runs
- Requests to DNS-only (grey cloud) records
- Request and response bodies
- Cookie contents
Where it is processed
- AWS eu-west-2 (London), with encrypted S3, DynamoDB and Aurora storage
- A GDPR Article 28 DPA applies to every account
- Never sold, used for advertising or shared between customers
Who controls it
- The Worker runs in your own Cloudflare account
- Removing its routes stops collection straight away
- WAF rules from Salience are applied only when you paste them
The Cloudflare Worker allowance
Cloudflare's Workers Free plan allows 100,000 Worker requests a day across your Cloudflare account, resetting at 00:00 UTC. The Worker runs on every request to the routed hostnames, including images, scripts and bots, so a busy site can use the allowance within hours; after that, Salience receives no requests until midnight UTC and Fail open keeps the site serving. Workers Paid costs $5 a month and includes 10 million requests. Cloudflare Logpush sends the zone's request logs with no Worker where Cloudflare offers it for the zone, and Cloudflare bills it by usage.
Cloudflare questions
Do I need Cloudflare Enterprise or Logpush?
No. The Salience Worker works on any Cloudflare plan, including Free. Logpush is an alternative that needs no Worker: Cloudflare sends the zone's own request logs to Salience, where Cloudflare offers the HTTP requests dataset for the zone, and bills Logpush by usage on your Cloudflare account.
Will the Worker slow down or break my site?
The Worker passes each request straight through to your origin and sends a compact log record to Salience in the background. Automatic setup fetches your pages before and after it adds the routes, and removes its routes if a page answers differently. Fail open on every route lets requests through when the Worker's daily allowance is used up; it does not cover an error in the Worker or at your origin. Removing the Worker's routes stops collection.
What if another Worker already runs on my site?
Setup checks the zone's Worker routes and Workers Custom Domains first. If one already covers your hostname, it stops, lists what it found and changes nothing; an existing Worker is never replaced or removed. You can send the findings to a developer with a delegated setup link, or use Cloudflare Logpush, which needs no Worker.
Does Salience see requests my Cloudflare rules block?
Requests a Cloudflare rule refuses before Workers run never reach the Worker, so they are not collected. Salience shows the effect as a drop in that crawler's requests, and Cloudflare's Security Events list what each rule acted on. Refusals sent by your origin (401, 403 and 429 responses) are collected. With Logpush as the log source, each request also carries Cloudflare's security action.
Can I see Googlebot and AI crawlers?
Yes. Every request from Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and 200+ other crawlers that reaches the Worker is recorded with the URL and the status code it received, and checked against the operator's published addresses where they publish them. AI crawlers are split into training, AI search and user-triggered fetches. A recorded fetch shows that a page was requested. It does not show whether an answer used it.
Is this my site's request data or my Worker's application logs?
Request data for your site: the requests Cloudflare proxies for your domain. Salience does not collect console output or exceptions from Workers you have written yourself.
Can I see data from before I connected?
Collection starts when the Worker is connected; the Worker cannot send requests from before it was deployed. Older Apache, Nginx or CloudFront log files can be uploaded with Import Logs to backfill earlier dates.
What does monitoring cost?
The Free plan covers one website, 500,000 requests a month and 30 days of history, with no card. Paid plans start at $19 a month, each with a 14-day trial. On the Cloudflare side, Workers Free allows 100,000 Worker requests a day and Workers Paid is $5 a month for 10 million.
Start monitoring your Cloudflare site for free.
The Free plan needs no card. Setup needs permission to deploy a Worker on your Cloudflare zone.