See what every bot and crawler does on your CloudFront distribution
Salience reads your CloudFront real-time logs and turns them into verified bot identities, crawl and error analysis, and alerts that name the addresses, paths and user agents involved, and imports your existing standard log files alongside, deduplicated.
Any AWS account. Salience holds no credentials and touches nothing: your pipeline delivers records to an endpoint. No website or DNS changes, and your existing standard log files import alongside.
The Log Explorer page for demo-site.example once CloudFront is connected through Kinesis + Firehose real-time logs: the Website setup strip shows Logs Receiving, and the table fills with Googlebot requesting /catalog/summer-collection, status 200, Verified; PerplexityBot requesting /guides/returns, status 200, Verified; Googlebot (claimed) requesting /.env, status 403, Unverified; Applebot requesting /assets/old-bundle.js, status 404, 404 spike. First hour on the distribution: 54,118 requests.
CloudFront logs sit in an S3 bucket and rarely get read
CloudFront records every request it serves and drops the files in S3, faithfully, forever. What happens next is the same story at almost every organisation.
Standing up Athena over the log bucket, writing a query when a question comes up, and paying for a scan every time somebody wants an answer.
Building a pipeline into your data warehouse, which works, and then discovering that nobody outside the data team can get an answer out of it without asking first.
Reading CloudWatch metrics that show error rates are up without saying which paths, which clients, or whether the clients were even genuine.
Taking user agents at face value, because verifying claimed crawlers against published address ranges is another engineering project nobody has budgeted for.
Salience classifies every request CloudFront serves
Real-time logs stream in as your distribution serves them: humans, verified crawlers, AI agents and suspicious automation, separated and counted.
The Salience dashboard for demo-site.example, last 24 hours. Website setup 5 of 6 completed. Traffic over time in fifteen-minute buckets, human and bot requests stacked. Bot identity evidence: 12.1K verified requests, 762 unverified claims, 2.7K with verification unknown. Total requests 34.2K, up 7.0% on the previous period. Unique IPs 15.3K, up 2.0%. Bot traffic 45.5%, down 1.9 points. AI crawlers 18.5%, down 8.0 points.
Delivery is typically seconds to a couple of minutes depending on your buffer settings. Standard log files import alongside and deduplicate automatically.
Check that a crawler is who it says it is
Every claimed identity is checked against the provider's own published address ranges. Genuine Googlebot, Bingbot, GPTBot and ClaudeBot are separated from the scrapers borrowing their names from datacentre addresses, and the impostors are listed with the addresses responsible.
Bots and Crawlers on demo-site.example, last 24 hours: 15,541 bot requests, 12,233 verified, 655 failed verification, 2,653 other status. Served versus rejected: Googlebot 357 served, 34 rejected, Impersonators only, 0 verified · 34 unverified; Amazonbot 3,934 served, 16 rejected, The real bot, 16 verified · 0 unverified; Attack Path Probe 413 served, 104 rejected, Not verifiable, 0 verified · 104 unverified.
AmazonbotGet answers without writing a query
No Athena scan to pay for, no warehouse job to schedule, no analyst in the loop. Crawl behaviour, error patterns and automated traffic are already computed per path and per section, and anyone on the team can read them or ask in plain English.
Path Analysis on demo-site.example, last 24 hours: Total Requests 34,163, Unique Paths 17,700, 4xx Errors 3,272, 5xx Errors 28. Top paths: / 1,704 requests, 5.0% of total, importance 7; /page-8a5da5 1,245 requests, 3.6% of total; /robots.txt 376 requests, 1.1% of total; /page-29d247/page-08a441 202 requests, 0.6% of total, importance 5; /favicon.ico 183 requests, 0.5% of total; /page-29d247/page-900bc8 161 requests, 0.5% of total, importance 5; /apple-touch-icon.png 120 requests, 0.4% of total.
Import the logs already in your bucket
Standard CloudFront access-log files upload directly, with format detection and deduplication against anything already streaming. The history sitting in your S3 bucket becomes a baseline instead of a storage cost, and imports never count against your monthly request quota.
Import Logs on demo-site.example. Import jobs and current progress. Each job retains its own source dates. Import Historical Logs: Upload CloudFront or Apache/Nginx log files, or a log-analyser Events CSV, to analyse historical traffic data. Drag & drop log files, or click to browse (multiple files supported). Supported formats: CloudFront, Apache/Nginx, log-analyser Events CSV (.log, .txt, .csv, .gz, .zip). Supported formats: CloudFront standard/real-time logs, Apache/Nginx Combined Log Format, log-analyser Events CSV Auto-detection: We'll automatically detect the log format and parse the data Compressed files: .gz and .zip files are supported and will be decompressed automatically
How it works
Connect
Configure CloudFront real-time logs to deliver through Kinesis and Firehose to the endpoint Salience generates, selecting the 23 log fields in the order the app lists them. Salience checks the field order on the first batch and rejects a misaligned configuration loudly rather than storing bad data. Guided instructions are in the app; your distribution's behaviour is unchanged.
We analyse
Each record is classified against the crawler registry, verified against published provider ranges, grouped by URL section and compared with your own history, sitemap, robots.txt and Search Console data.
Get answers
Dashboards, alerts by email, Slack or webhook, plain-English querying, an API and a CLI. Upload your standard log files at any point to backfill history.
Specification
- Collection method
- CloudFront real-time logs delivered through Kinesis and Firehose
- Data freshness
- Seconds to minutes
- Typical setup time
- Roughly 30–60 minutes in the AWS console
- Plan requirement
- Any AWS account. Real-time logs, Kinesis and Firehose carry AWS usage charges
- Permissions needed
- IAM permissions to configure CloudFront real-time logs, Kinesis and Firehose
- Code or DNS changes
- None to your website and no DNS changes
- Historical import
- Yes. Standard CloudFront access-log files import with deduplication
- Key limitation
- Real-time logs deliver the sampling rate you configure. At 100% Salience sees every request CloudFront serves; below that it sees a representative sample, and Salience will not pretend otherwise. The AWS delivery components are billed by AWS on your account.
Which of these is you?
Platform and cloud engineering
We have the logs. What we do not have is anyone with time to build and maintain the layer that makes them useful.
The interpretation layer, already built: classification, verification, baselining and alerting, with no pipeline of your own to run.
Site reliability
SEO teams
I need to know what Googlebot did across the distribution, and I cannot get an Athena query prioritised to find out.
Crawl activity by section and URL pattern, crawl waste and error spikes against your baseline, without asking anyone for a query.
Search intelligence
Security teams
CloudWatch tells me error rates moved. It does not tell me who, or from where, or whether anything succeeded.
Probe patterns, exposed-secret fetches and impersonation surges surfaced with named source addresses, escalated if anything returned a 200.
Threat intelligence
AI and content teams
Which AI systems are pulling our content through the CDN, and is it training or live retrieval?
GPTBot, ClaudeBot, PerplexityBot and the rest, separated into training, AI-search indexing and user-triggered retrieval.
AI access intelligence
Why not just query the logs in Athena?
You certainly can, and for some questions you should. CloudFront's standard logs, CloudWatch metrics, the built-in reports and Athena over the log bucket are a capable combination, and if you have a data team that enjoys this work you can build most of what follows yourself. Plenty of organisations have. The honest comparison is not capability, it is cost and standing: what you pay per question, who is able to ask one, and whether anything is watching when nobody is asking.
Query the raw records
- Athena, CloudWatch, CloudFront reports
- Yes, and flexibly
- Salience
- Yes, plus plain-English querying
Cost per question
- Athena, CloudWatch, CloudFront reports
- An Athena scan each time
- Salience
- Included; no per-query charge
Who can get an answer
- Athena, CloudWatch, CloudFront reports
- Someone who writes SQL
- Salience
- Anyone on the team
Bot identity verified against published ranges
- Athena, CloudWatch, CloudFront reports
- You would build and maintain this
- Salience
- Included, across 200+ crawler identities
Search and AI crawler interpretation
- Athena, CloudWatch, CloudFront reports
- You would build and maintain this
- Salience
- Crawl budget, index context, training versus retrieval
Watching when nobody is asking
- Athena, CloudWatch, CloudFront reports
- Alarms on metrics you define
- Salience
- 16 detectors on a 5-minute cycle, each naming a next step
Joined to sitemap, robots.txt and Search Console
- Athena, CloudWatch, CloudFront reports
- Out of scope
- Salience
- Correlated, so a change links back to its cause
When the native tooling is all you need
If you need arbitrary analysis over years of raw records, or your questions are genuinely bespoke each time, Athena over your log bucket is the right tool and Salience does not replace it. Keep both, they read the same data. Salience is for the standing questions: who is crawling us, is that genuine, what changed, and who should be told without anyone writing a query first.
Facts about the integration
No customer logos and no five-star quotes on this page. These are properties of the product you can check on the free tier in an afternoon.
Illustrative, not a customer incident
A new behaviour on the distribution routes a whole path prefix to the wrong origin, and every request under it starts returning 404. The CloudWatch graphs show the error rate climbing without saying which paths or who is hitting them. In the request stream the 404s appear per path with Googlebot's share separated out, the error-spike alert names the prefix and the verified crawlers affected, and the behaviour is corrected before the crawler returns.
Everything included
One connection, one request stream. No per-feature setup and no second pipeline to maintain.
Verified bot and AI crawler identities
Claimed identities checked against official IP ranges where providers publish them; anything unverifiable is marked unverified rather than guessed at.
The field order, checked
Real-time logs must carry 23 fields in a fixed order. Salience verifies the order on the first batch and tells you exactly which field is out of place, instead of quietly misreading every record.
Crawl budget by section
Where Googlebot's time actually goes across the distribution, which URLs it never reaches, and where the budget is being wasted.
Error, latency and status analysis
2xx through 5xx per path and section, with spike detection against your own baseline rather than a fixed threshold.
Threat and probe detection
sqlmap-style requests, /.env and config fetches, credential-stuffing patterns and impersonation surges, escalated if anything returned a 200.
Alert detectors on a five-minute cycle
Traffic-drop alerts check the site is reachable before they fire, probe alerts check whether the flagged path really serves anything, and every alert is enriched first, so you never get "your site might be down" when it is not.
Standard log import with deduplication
The files already in your S3 bucket upload with format detection, merge cleanly against streamed records, and never count against your quota.
Ask questions in plain English
Ask a question in words rather than SQL, and get an answer without an Athena scan or a ticket to the data team.
Alerts by email, Slack and webhook
Dashboards, email and Slack alerts, webhooks, a public API, a CLI, an MCP server, CSV exports and shared read-only dashboards.
Recommendations with a rule for CloudFront
Addresses to block and fake crawler user agents, scored from the last 30 days of alerts, with a CloudFront Function generated from the list and ready to paste.
Saved segments
Page groups you define by prefix, pattern or query string, with a one-click library of common groups, applied to all history and used in every report, alert and export.
Site checks every night
Twenty-one checks on crawler access, security hygiene and serving quality, each pass, warn or fail with the evidence, 30 days of history and an alert when a verdict changes.
Weekly report and daily digest
A weekly email report per site, a daily digest of lower-severity alerts, and email for anything above the severity you set.
Trust & data protection
Privacy and data protectionYou are the controller
We process only on your instructions. GDPR Art. 28 DPA on every account, nothing to sign.
UK data residency
AWS eu-west-2 (London). Encrypted in transit (TLS 1.2+) and at rest (AES-256).
Server-side collection
No browser tracking script and no client-side pixel.
No sale, no pooling
Your logs are never sold, never used for advertising, never shared between customers. DPA, sub-processor list and security overview available.
What is collected
- Timestamp, method, host and path
- Status code and bytes served
- Client address and user agent
- Edge location, cache result and edge timing
What is never collected
- Request and response bodies
- Origin application logs
- Requests excluded by your sampling rate
Priced on requests, not seats, and separate from your AWS bill: the Kinesis and Firehose charges stay with AWS. Start on the free tier with 500,000 requests a month and upgrade when your traffic does.
Crawler intelligence from $19/mo.
Priced on requests, not seats. Unlimited users on every plan except Solo. Start on the free tier and upgrade when your traffic does. Free trial, no card needed.
- 500K requests/mo
- 1 site
- 30 days history
- Unlimited users
- Real-time analytics, bot and AI detection
- 20M requests/mo
- 5 sites (+5)
- 1 year history
- Unlimited users
- Real-time analytics with bot and AI-crawler verification
- 100M requests/mo
- 15 sites (+15)
- 2 years history
- Unlimited users
- AI allowance: ~600 answers or ~60 reports a month
- 500M requests/mo
- 50 sites, no ceiling
- 4 years history
- Unlimited users
- AI allowance: ~3,000 answers or ~300 reports a month
- 1B requests/mo
- Unlimited sites
- Custom history
- Unlimited users
- SSO / SAML and audit log
Common questions
Real-time logs or standard logs: which do I need?
Both, ideally, and they do different jobs. Real-time logs stream through Kinesis and Firehose and are what makes monitoring and alerting possible. Standard log files land in S3 on a delay and are what you import for history. Most teams connect real-time for monitoring and upload their existing standard files to build a baseline immediately.
How long does setup take, and do I need a developer?
Roughly 30 to 60 minutes in the AWS console for someone with the IAM permissions to configure real-time logs, Kinesis and Firehose. It is the most involved of our integrations, purely because AWS has more moving parts. The app walks through each step, and the work can be delegated from inside Salience without sharing your account.
What does this cost on the AWS side?
Real-time logs, Kinesis Data Streams and Firehose all carry AWS usage charges billed on your own account, and they scale with your traffic. We flag this rather than burying it: it is a real cost of the integration and worth estimating before you enable it at 100% sampling on a high-traffic distribution.
Why does the app say my field order is wrong?
CloudFront lets you pick any subset of fields in any order, and a real-time log with the wrong order still looks valid. Salience expects the 23 canonical fields in the standard order (the setup page lists them) and checks the first batch it receives. If they do not line up it refuses the batch and names the field, so nothing is stored misaligned. Re-select the fields in the listed order and delivery resumes.
Does sampling matter?
Yes, and we will not paper over it. At 100% Salience sees every request CloudFront serves. Below that it sees a representative sample, which is fine for trends and proportions and unreliable for questions about specific rare events: a handful of probe requests against one path can disappear entirely from a 10% sample.
Does Salience get access to our AWS account?
No. The delivery is one-directional: your pipeline sends records to an endpoint. Salience holds no AWS credentials, cannot read your buckets, cannot see other services and cannot change your distribution or any other resource.
Can I import the logs already in my S3 bucket?
Yes, and it is the fastest way to get a useful baseline. Standard CloudFront log files upload with automatic format detection, deduplicate against anything already streaming, and do not count against your monthly request quota.
How is this different from querying with Athena?
Athena answers the question you write, each time, for a scan charge, if you can write SQL. Salience has already computed the standing answers, verifies claimed bot identities against published address ranges, keeps a baseline, joins the data to your sitemap and Search Console, and alerts when something changes without anyone asking. Many teams keep both.
What happens to our data, and where is it stored?
You are the controller; Salience processes only on your instructions under a GDPR Art. 28 DPA that applies to every account. Data is stored encrypted in AWS eu-west-2 (London), never sold, never used for advertising and never pooled between customers.
Can we remove it later?
Disable the real-time log configuration on the distribution, or tear down the Firehose delivery stream, and collection stops immediately. Nothing in your AWS account depends on Salience.
Your logs already show what Google and the AI crawlers are doing.
Any AWS account. Salience holds no credentials and touches nothing: your pipeline delivers records to an endpoint. No website or DNS changes, and your existing standard log files import alongside.