CloudFront server log monitoring
See every crawler, bot and error on your CloudFront distribution.
CloudFront reports and CloudWatch show request totals, cache results and error rates. Salience shows which crawler, AI bot or scraper sent each request, the URL it asked for and the response it got.
- Any AWS account
- No website or DNS changes
- Standard log files import as history
Log Explorer
news.example · requests in real time
GooglebotVerified/news/budget-vote200
ChatGPT-UserVerifiedUser-triggered/news/rail-strike-dates200
BingbotUnverified/news/flood-defences200- GooglebotVerified/tag/harbour-festival-2019404
ClaudeBotVerifiedTraining/section/business200
PerplexityBotVerifiedAI search/news/budget-vote200
Requests to /news/ article URLs from 198.51.100.61, an address outside Microsoft's published Bingbot ranges.
Recommended: review the address in Recommendations and deploy the generated CloudFront Function.
CloudFront reports and Salience answer different questions
The CloudFront console reports and CloudWatch metrics add up requests, bytes, cache results and error rates for a distribution. Salience reads each request in your real-time logs and identifies the client that sent it.
CloudFront reports and CloudWatch
Aggregated reports in the CloudFront console and CloudWatch metrics
- How many requests and bytes did the distribution serve?
- What share of requests were cache hits?
- Which objects and referrers were requested most?
- Did the 4xx or 5xx error rate go up?
Athena can query the standard log files in S3 with SQL, billed per query.
Salience
Every request in your CloudFront real-time logs, delivered by your Firehose stream
- Which crawlers requested which URLs, and were they who they said they were?
- What status code and edge result did each request get?
- Which AI systems request your pages, and which pages do AI assistants send people to?
- Which addresses are copying your content page by page?
Salience receives log records only and holds no AWS credentials.
Questions your CloudFront reports can't answer
Four examples from the four teams that use Salience on a CloudFront distribution. The news site, addresses and figures in the panels are examples.
01Search and SEO
Is Googlebot reaching your new articles?
A publishing site generates tag pages, date archives and section fronts around every article. Googlebot can spend much of its time on those and reach new articles late, and the CloudFront reports do not show which URLs Googlebot requested.
Salience records every verified Googlebot request with the URL, the status code, the edge result and the time, groups the requests by section, and compares the URLs Googlebot fetched with the URLs in your sitemap.
Find sitemap articles Googlebot has not fetched, see how much crawling goes to tag pages and date archives, and see error responses to Googlebot on the day they start.
Verified Googlebot by section
news.example · last 7 days
| Section | Requests | URLs | Unfetched |
|---|---|---|---|
| Articles/news/* | 6,204 | URLs fetched2,318 | Sitemap URLs not fetched37 |
| Section fronts/section/* | 1,418 | URLs fetched46 | Sitemap URLs not fetched0 |
| Tag pages/tag/* | 3,377 | URLs fetched2,904 | Sitemap URLs not fetchedNot in sitemap |
| Date archives/archive/* | 1,861 | URLs fetched1,240 | Sitemap URLs not fetchedNot in sitemap |
Sources requesting the articles
news.example · last 24 hours
Requests per hour from unverified automation
peak 3,057
Unverified automated requests ran at about 100 an hour, rose to between 2,788 and 3,057 an hour from 06:00 to 13:59, then returned to about 100.
- 203.0.113.0/24Not a crawler21,734
Hosting provider · 6 rotating browser user agents · Each new /news/* URL within minutes of publication, no images or scripts
- 198.51.100.61Unverified2,688
Hosting provider · Bingbot (claimed) · /news/* articles, oldest first
- BingbotVerified2,215
Microsoft · Bingbot · Articles and section fronts, follows robots.txt
- 192.0.2.140Not a crawler946
Hosting provider · Scrapy/2.11 · Every /section/* front, every 5 minutes
02Security and content protection
Who is copying your articles?
Scrapers request new articles within minutes of publication, often from datacentre addresses, rotating browser user agents or borrowing a search crawler's name. CloudWatch shows the request count rising without saying who sent the requests.
Salience groups requests by source address and user agent, checks every claimed crawler name against the operator's published addresses, and shows each source's paths, request rate and timing. The evidence is the address, the network, the user agent and the pattern of requests.
Block the sources in your own AWS account. Recommendations list the addresses and fake crawler user agents and generate a CloudFront Function that returns 403, for you to paste.
03AI search and discovery
Which AI systems are requesting your articles?
AI companies run separate crawlers for collecting training data, for building an AI search index and for fetching a page when someone asks an assistant a question. Each one is a separate robots.txt decision.
Salience puts every AI crawler in one of those three classes, verifies it where the operator publishes its addresses, and lists the articles and sections each one requested.
A recorded request shows that a page was fetched. It does not show whether an answer used or cited the page.
The AI referral traffic report counts the human visits that arrive from ChatGPT, Perplexity, Claude, Gemini, Copilot and other assistants, by landing page.
AI crawlers by purpose
news.example · last 7 days · verified requests
- GPTBot2,874
- ClaudeBot1,652
Most requested /news/hospital-waiting-times
- OAI-SearchBot938
- PerplexityBot721
Most requested /section/politics
- ChatGPT-User412
- Claude-User88
Most requested /news/rail-strike-dates
04Engineering and operations
What broke after your last distribution change?
A new AWS WAF rule, a changed cache behavior or an origin release can start refusing crawlers or returning errors while the site still loads in a browser.
Salience tracks status codes per path and per type of client against your own history. Site Events mark your releases and changes on every chart, and the Edge Policy Change alert fires when verified crawlers start receiving 401, 403 or 429 responses.
The alert names the date, the crawlers affected and how their refused share compares with the usual level, with a recommended next step.
Requests refused (401, 403, 429) by client
news.example · daily share · 14 days
- Verified search crawlers
- 0.2% 31.6%
- Verified AI crawlers
- 0.4% 57.9%
- Human visitors
- 0.2%
- Verified search crawlers
- Verified AI crawlers
- Human visitors
- Site event · AWS WAF rate-based rule added
Googlebot, Bingbot, GPTBot and ClaudeBot are being refused with 403 at the edge. Human visitors are unaffected.
Recommended: check the AWS WAF rules changed today on the distribution's web ACL, and allow verified bots.
Example investigation
Googlebot started receiving 502s on 10 explainer URLs
An example of how an alert becomes a fix, step by step. The site, URLs and figures are illustrative.
- 1
The alert fires
At 11:20 a Crawler Error Spike alert reports that verified Googlebot requests to /explainers/ URLs are returning 502 far above the site's usual rate.
- 2
The crawler is Google's
Every request came from an address inside Google's published Googlebot ranges, so these are Google's own fetches failing at the edge.
- 3
The affected URLs
71 requests across 10 URLs, all under /explainers/, the path pattern of the cache behavior your team changed that morning to use a new origin.
- 4
When it started
The first 502 arrived at 10:41, 11 minutes after the behavior change your team recorded as a Site Event at 10:30. The requests continued through the day.
- 5
The next step
CloudFront returns 502 when it cannot connect to the origin, for example over a certificate that does not match the origin's domain. Fix the new origin or point the behavior back at the previous one, then check that Googlebot receives 200 on its next visits.
Crawler Error Spike
news.example · opened 11:20 · Googlebot · 502
GooglebotVerified66.249.70.0/24 · in Google's published ranges- /explainers/council-tax50212
- /explainers/rail-strike-ballots50210
- /explainers/flood-warning-levels5029
- /explainers/hospital-waiting-lists5028
- /explainers/election-results5027
- /explainers/budget-vote-guide5027
- 4 more URLs · 18 requests
10:30 Site event · /explainers/* behavior changed
The /explainers/* cache behavior changed at 10:30. Googlebot received 3 502s from 10:41 in the 10:00 hour, then 12, 14, 11, 10, 9, 7 and 5 per hour from 11:00 to 17:00, 71 in total.
Recommended next step
Fix the origin the /explainers/* behavior now points at, or point the behavior back at the previous origin, then confirm Googlebot receives 200 on these 10 URLs.
What Salience shows for a CloudFront distribution
Every report comes from the same request data. Search Console and Google Analytics 4 add search and session figures to the same URLs if you connect them.
URL detail
news.example/news/rail-strike-dates · last 28 days
/news/rail-strike-dates
- LogsVerified Googlebot fetchesLast 200 at 13:589
- LogsAI page fetchesChatGPT-User, Claude-User23
- LogsAI-referred visitschatgpt.com, perplexity.ai17
- Search ConsoleImpressions, 28 daysIndexed, last inspected 18 Oct48,600
- Google AnalyticsSessions, 28 daysOrganic search 64%5,214
Who is requesting
Verified crawler identities
200+ crawler identities, checked against published address ranges where the operator publishes them.
AI crawlers and AI referrals
Training, AI search and user-triggered fetches separated, with AI-referred visits by landing page.
Scraper and automation patterns
Systematic requests for your pages grouped by address, network and user agent.
What they request
Every request, URL by URL
Log Explorer and a URL page for each path, with status, client, timing and source.
Segments for your site sections
Define sections once by prefix or pattern, or add common ones such as blog, guides and pagination in one click, applied to all history.
Standard log import
CloudFront standard log files from S3 upload with format detection and deduplication against the real-time stream.
What happens next
Status codes and errors
Status codes per path, soft 404s, redirects and response times against your own baseline.
Alerts and recommendations
Alerts by email, Slack or webhook with a recommended next step, and a CloudFront Function generated from the recommendations.
Reports
A weekly email report, a daily digest and a written technical SEO report.
Salience also runs its own site crawler and crawl audit, and has a public API, a CLI and an MCP server.
How to set up server log monitoring on CloudFront
Setup takes about 10 minutes in the AWS console. The app walks you through each step and checks the connection at the end.
- 1
Enter your domain and create your account
Use the domain your CloudFront distribution serves. The Free plan needs no card.
- 2
Create the Kinesis stream, Firehose stream and real-time log configuration
Create an on-demand Kinesis data stream, a Firehose stream that delivers to your Salience endpoint with your API key, and a real-time log configuration with the 23 fields the app lists, in that order. Then select the configuration on your distribution's cache behaviors.
- 3
Confirm requests are arriving and start investigating
The connection check sends three test requests to your site and looks for them in what Firehose delivers. With Firehose's default 60-second buffer, requests appear within a few minutes.
What you need
- The AWS account that runs the CloudFront distribution for your site
- IAM permissions to configure CloudFront real-time logs, Kinesis and Firehose
- An S3 bucket where Firehose keeps any failed deliveries
What stays the same
- No changes to your website or its code
- No DNS changes
- CloudFront keeps serving every request
Before you start
- Select only the 23 listed fields, in the listed order. The connection check reports a field list it cannot read
- Set Firehose content encoding to Disabled
- A sampling rate of 100% records every request; a lower rate delivers that share of requests
- Monitoring starts at connection. Standard log files already in S3 can be imported for earlier dates
Someone else runs your AWS account? Send them a delegated setup link from the app. It lasts 30 days and is limited to this site, and they complete the AWS steps without access to the rest of your Salience account. The setup screen can also hand the steps to an AI coding agent.
How your request data is collected and controlled
What is collected
- Timestamp, method, host and path
- Status code and bytes served
- Client address, user agent and referrer
- Country, edge location, cache result and timing
What is not collected
- Request and response bodies
- Your origin's application logs
- Requests outside your sampling rate
- Anything else in your AWS account
Where it is processed
- AWS eu-west-2 (London), with encrypted S3, DynamoDB and Aurora storage
- A GDPR Article 28 DPA applies to every account
- Never sold, used for advertising or shared between customers
Who controls it
- The Kinesis stream, Firehose stream and log configuration run in your own AWS account
- Removing the real-time log configuration from the distribution stops collection
- Salience holds no AWS credentials and cannot change your distribution
AWS charges for real-time logs
CloudFront real-time logs, Kinesis Data Streams and Amazon Data Firehose carry AWS usage charges, billed by AWS on your account and separate from your Salience plan. The charges rise with the number of requests logged, so estimate them for a high-traffic distribution before you enable 100% sampling. A lower sampling rate reduces them and delivers that share of requests.
AWS CloudFront questions
Do I need Athena or a log pipeline of my own?
No. Salience receives the real-time log records from your Firehose stream and classifies, verifies and groups them as they arrive, so the reports need no SQL and no Athena query. Athena over your S3 log bucket still answers one-off questions from the same requests.
How long does setup take, and do I need a developer?
About 10 minutes in the AWS console, for someone with IAM permissions to configure CloudFront real-time logs, Kinesis and Firehose. AWS offers to create the two IAM roles if they do not exist. If someone else runs your AWS account, send them a delegated setup link from the app.
Can I import the CloudFront logs already in my S3 bucket?
Yes. Upload standard CloudFront log files on Import Logs; the format is detected, records that overlap the real-time stream are deduplicated, and imports do not count against your monthly request allowance. Real-time logs then cover monitoring and alerts from the day you connect.
Does the sampling rate matter?
Yes. At 100%, the rate the setup guide recommends, Salience receives every request CloudFront logs. At a lower rate it receives that share of requests, which covers trends and proportions; a handful of rare requests, such as a few probes against one path, may not appear in the sample.
Why does Salience say my field list is wrong?
CloudFront sends real-time log fields by position, without labels, and Salience reads the 23 fields in the order the setup guide lists. A missing, extra or reordered field makes every record unreadable. The dashboard, the connection check and the status badge say what is wrong and give the field list to copy. Correct the configuration and data flows again within a minute; requests rejected before the fix cannot be recovered.
Does Salience get access to our AWS account?
No. Your Firehose stream sends log records to the Salience endpoint with an API key. Salience holds no AWS credentials and cannot read your buckets, inspect other services or change your distribution. The setup checks read Salience's own records, never your AWS account.
Can I see Googlebot and AI crawlers?
Yes. Every request from Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and 200+ other crawlers is recorded with the URL and the status code it received, and checked against the operator's published addresses where they publish them. AI crawlers are split into training, AI search and user-triggered fetches. A recorded fetch shows that a page was requested; it does not show that an answer used it.
What does monitoring cost?
The Free plan covers one website, 500,000 requests a month and 30 days of history, with no card. Paid plans start at $19 a month, each with a 14-day trial. CloudFront real-time logs, Kinesis and Firehose are billed separately by AWS on your account.
Start monitoring your CloudFront distribution for free.
The Free plan needs no card. Setup needs permission to configure CloudFront real-time logs, Kinesis and Firehose in your AWS account.