Analyse your Apache access log for bots, crawlers and errors
Salience streams the Apache access log you already write and turns it into verified bot identities, crawl and error analysis, and alerts that name the addresses, paths and user agents involved, without changing your log format, your site or your server configuration.
Any Linux server you control, live in about ten minutes with no change to your Apache configuration. Archived logs import too.
The Log Explorer page for demo-site.example once Apache is connected through Vector agent: the Website setup strip shows Logs Receiving, and the table fills with Googlebot requesting /products/wireless-headphones, status 200, Verified; GPTBot requesting /docs/api/, status 200, Verified; Googlebot (claimed) requesting /phpmyadmin/, status 404, Unverified; Bingbot requesting /old-product-url, status 404, 404 spike. First hour of the tail: 18,340 requests.
- Cloudflare
- Vercel
- Shopify
- Nginx
- Netlify
Reading a raw access log by hand is slow
A busy Apache server writes millions of lines a month, each one a complete record of a request. Almost nobody reads them, because reading them properly is a job rather than a task.
Running grep and awk over a rotated log when something has already gone wrong, reconstructing an incident after the fact instead of being told about it while it happens.
Taking user agents at face value, because checking a claimed Googlebot against Google's published address ranges by hand, across millions of lines, is not realistic.
Running a log analyser that produces yesterday's totals at nine this morning: accurate, comprehensive and about twenty-one hours too late to act on.
Losing history to logrotate, so the one week you actually want to compare against has already been compressed, archived or deleted.
Salience reads the same log as it is written
Lines stream from the file as your server writes them: humans, verified crawlers, AI agents and suspicious automation, separated and counted.
The Salience dashboard for demo-site.example, last 24 hours. Website setup 5 of 6 completed. Traffic over time in fifteen-minute buckets, human and bot requests stacked. Bot identity evidence: 12.1K verified requests, 762 unverified claims, 2.7K with verification unknown. Total requests 34.2K, up 7.0% on the previous period. Unique IPs 15.3K, up 2.0%. Bot traffic 45.5%, down 1.9 points. AI crawlers 18.5%, down 8.0 points.
Lines reach Salience within seconds of Apache writing them and show in reports within about fifteen minutes. Anomaly detection begins once about 24 hours of baseline has built.
Check that a crawler is who it says it is
Your log records what each client claimed to be, and that is all it is: a claim. Salience checks every claimed identity against the provider's own published address ranges, separating genuine Googlebot, Bingbot, GPTBot and ClaudeBot from the scrapers borrowing their names, and lists the impostors with the addresses responsible.
Bots and Crawlers on demo-site.example, last 24 hours: 15,541 bot requests, 12,233 verified, 655 failed verification, 2,653 other status. Served versus rejected: Googlebot 357 served, 34 rejected, Impersonators only, 0 verified · 34 unverified; Amazonbot 3,934 served, 16 rejected, The real bot, 16 verified · 0 unverified; Attack Path Probe 413 served, 104 rejected, Not verifiable, 0 verified · 104 unverified.
AmazonbotAlerts arrive within minutes of a change
Status codes are analysed per path and per site section against your own baseline. A deploy that starts feeding crawlers 404s, a section that begins returning 5xx, a login endpoint under sustained attempt: each is detected against your own history and alerted while it is still cheap to fix.
Status code detail for 4xx Client Error on demo-site.example, last 24 hours: Total Requests 3,272, % of All Traffic 9.58%, Unique Codes 4. Individual status codes: 403 2,337, 71.4% of 4xx; 404 896, 27.4% of 4xx; 405 29, 0.9% of 4xx; 499 10, 0.3% of 4xx. Top paths: /old-product-url 155, /blog/2019-relaunch 142, /guides/pricing-2025 119, /products/legacy 98, /sitemap-old.xml 90, /blog/page/48 42.
Import the logs you already have
Rotated and archived access logs upload directly, with format detection and deduplication against anything already streamed. Your baseline starts with real history rather than building from zero, and imports never count against your monthly request quota.
Import Logs on demo-site.example. Import jobs and current progress. Each job retains its own source dates. Import Historical Logs: Upload CloudFront or Apache/Nginx log files, or a log-analyser Events CSV, to analyse historical traffic data. Drag & drop log files, or click to browse (multiple files supported). Supported formats: CloudFront, Apache/Nginx, log-analyser Events CSV (.log, .txt, .csv, .gz, .zip). Supported formats: CloudFront standard/real-time logs, Apache/Nginx Combined Log Format, log-analyser Events CSV Auto-detection: We'll automatically detect the log format and parse the data Compressed files: .gz and .zip files are supported and will be decompressed automatically
How it works
Connect
Install the open-source Vector agent on the server and point it at the access log you already write. Guided instructions and a ready-made configuration are in the app; Apache itself is untouched.
We analyse
Each line is classified against the crawler registry, verified against published provider ranges, grouped by URL section and compared with your own history, sitemap, robots.txt and Search Console data.
Get answers
Dashboards, alerts by email, Slack or webhook, plain-English querying, an API and a CLI. The connection confirms within seconds, and the first reports fill within about fifteen minutes. Seconds of the agent starting.
Specification
- Collection method
- The open-source Vector agent, tailing the access log you already write
- Data freshness
- Within seconds
- Typical setup time
- About 10 minutes
- Plan requirement
- Any Linux server you control. No Apache version or licence requirement
- Permissions needed
- Root or sudo to install Vector, and read access to the log files
- Code or DNS changes
- No Apache modules and no DNS changes. An agent is installed on the server
- Historical import
- Yes. Rotated and archived access logs upload with deduplication
- Key limitation
- Access logs only: Apache's error log is not ingested. Response-time analysis needs a timing field in your log format; the default formats do not include one, and Salience will not invent it.
Which of these is you?
Systems administrators
I find out about traffic problems when someone complains, and then I am reading log files at eleven at night.
Error rates, latency and traffic anomalies detected against section-level baselines, with the clients and paths responsible named in the alert.
Site reliability
WordPress and agency hosting
Every site on this box gets the same background of /wp-login.php and /xmlrpc.php attempts and I have no idea which are actually landing.
Section-level views per site, with probe patterns escalated when something returned a 200 rather than a rejection.
Threat intelligence
SEO teams
I want to know where Googlebot spends its time on this server, not what a crawl simulation thinks it would do.
Crawl activity by directory and URL pattern, crawl waste, and error spikes caught against your baseline, joined to your sitemap.
Search intelligence
AI and content teams
Which AI systems are reading this site, and is it training or someone's live question?
GPTBot, ClaudeBot, PerplexityBot and the rest, separated into training, AI-search indexing and user-triggered retrieval.
AI access intelligence
Why not just use GoAccess, AWStats or grep?
They are good tools and some of them are excellent. GoAccess in particular gives you a genuinely useful real-time terminal dashboard for nothing, AWStats has been producing dependable traffic reports for two decades, and there is no substitute for grep when you already know precisely what you are looking for. If you are running one server and you enjoy the command line, you can get a long way without paying anybody. The difference shows up in three places: verification, memory, and being told rather than looking.
Parse and summarise the log
- GoAccess, AWStats, grep
- Yes, and quickly
- Salience
- Yes
Bot identity verified against published ranges
- GoAccess, AWStats, grep
- User agent taken at face value
- Salience
- Checked against official ranges, 200+ identities
Search and AI crawler interpretation
- GoAccess, AWStats, grep
- Counted as user-agent strings
- Salience
- Crawl budget, index context, training versus retrieval
Alerting when something changes
- GoAccess, AWStats, grep
- Not the design: you run the tool
- Salience
- 16 detectors on a 5-minute cycle, each naming a next step
Baseline across weeks and months
- GoAccess, AWStats, grep
- Bounded by what logrotate kept
- Salience
- Retained per plan; archives import and deduplicate
Multiple servers in one view
- GoAccess, AWStats, grep
- One host at a time
- Salience
- Every server and site in one organisation
Joined to sitemap, robots.txt and Search Console
- GoAccess, AWStats, grep
- Out of scope
- Salience
- Correlated, so a change links back to its cause
When the native tooling is all you need
If you run a single server, you are comfortable at the command line, and your question is usually "what happened in the last hour" rather than "what changed this week and who should know", GoAccess and grep will serve you well and cost you nothing. Salience earns its place when the log has to answer to more than one person, across more than one machine, without anyone having to remember to look.
Facts about the integration
No customer logos and no five-star quotes on this page. These are properties of the product you can check on the free tier in an afternoon.
Illustrative, not a customer incident
One datacentre address sends 114,838 POST requests to /wp-login.php over six hours, rotating twenty browser user agents to look like separate visitors. In the access log it is 114,838 ordinary-looking lines among millions. Against a five-minute baseline it is a single alert in the first window, with the address, the path and the fact that every attempt was rejected, or, more usefully, that one was not.
Everything included
One connection, one request stream. No per-feature setup and no second pipeline to maintain.
Verified bot and AI crawler identities
Claimed identities checked against official IP ranges where providers publish them; anything unverifiable is marked unverified rather than guessed at.
Common and Combined formats supported
The default formats work as written. Custom formats are welcome, and nothing about your existing configuration needs to change.
Per-site and per-section views
Each server connects as its own site, and activity is broken down by directory and URL pattern, so a problem on a shared box lands with the site and section named.
Error, latency and status analysis
2xx through 5xx per path and section, with spike detection against your own baseline rather than a fixed threshold.
Threat and probe detection
sqlmap-style requests, /.env and config fetches, login attempt floods and impersonation surges, escalated if anything returned a 200.
Alert detectors on a five-minute cycle
Traffic-drop alerts check the server is reachable before they fire, probe alerts check whether the flagged path really serves anything, and every alert is enriched first, so you never get "your server might be down" when it is not.
Historical import with deduplication
Rotated and archived logs upload with format detection, merge cleanly against streamed data, and never count against your quota.
Many servers, one account
Every Apache host, and any Nginx, Cloudflare, Vercel or CloudFront source, normalised to one request model in one organisation.
Alerts by email, Slack and webhook
Dashboards, email and Slack alerts, webhooks, a public API, a CLI, an MCP server, CSV exports and shared read-only dashboards.
Recommendations to act on
Addresses to block, fake crawler user agents, 404 paths crawlers keep hitting and slow paths, scored from the last 30 days of alerts, with the sample user agent and paths behind each one.
Saved segments
Page groups you define by prefix, pattern or query string, with a one-click library of common groups, applied to all history and used in every report, alert and export.
Site checks every night
Twenty-one checks on crawler access, security hygiene and serving quality, each pass, warn or fail with the evidence, 30 days of history and an alert when a verdict changes.
Weekly report and daily digest
A weekly email report per site, a daily digest of lower-severity alerts, and email for anything above the severity you set.
Trust & data protection
Privacy and data protectionYou are the controller
We process only on your instructions. GDPR Art. 28 DPA on every account, nothing to sign.
UK data residency
AWS eu-west-2 (London). Encrypted in transit (TLS 1.2+) and at rest (AES-256).
Server-side collection
No browser tracking script and no client-side pixel.
No sale, no pooling
Your logs are never sold, never used for advertising, never shared between customers. DPA, sub-processor list and security overview available.
What is collected
- Timestamp, method, host and path
- Status code and response size
- Client address and user agent
- Referrer
What is never collected
- Apache's error log
- Request and response bodies
- Response timing, unless your log format records it
Priced on requests, not seats, and not per server: connect as many Apache hosts as you like on one plan. Start on the free tier with 500,000 requests a month and upgrade when your traffic does.
Crawler intelligence from $19/mo.
Priced on requests, not seats. Unlimited users on every plan except Solo. Start on the free tier and upgrade when your traffic does. Free trial, no card needed.
- 500K requests/mo
- 1 site
- 30 days history
- Unlimited users
- Real-time analytics, bot and AI detection
- 20M requests/mo
- 5 sites (+5)
- 1 year history
- Unlimited users
- Real-time analytics with bot and AI-crawler verification
- 100M requests/mo
- 15 sites (+15)
- 2 years history
- Unlimited users
- AI allowance: ~600 answers or ~60 reports a month
- 500M requests/mo
- 50 sites, no ceiling
- 4 years history
- Unlimited users
- AI allowance: ~3,000 answers or ~300 reports a month
- 1B requests/mo
- Unlimited sites
- Custom history
- Unlimited users
- SSO / SAML and audit log
Common questions
Do I have to change my Apache configuration or log format?
No. The default Common and Combined formats are supported as written, and custom formats are welcome. Nothing about Apache changes; the agent reads the file your server is already producing.
How long does setup take, and do I need a developer?
About ten minutes for someone with root or sudo on the server. The app generates a ready-made agent configuration, so the work is installing a package and pointing it at a file. If that needs to go to a systems administrator, you can delegate it from inside Salience without sharing your account.
Will the agent affect server performance?
It reads files and ships lines over HTTPS, which is light work on any modern server. It does not sit in the request path, so it cannot slow a response down or fail a request. If the agent stops, Apache carries on exactly as before and you simply stop receiving new data.
Can I see response times?
Only if your log format records them. The default Common and Combined formats carry no timing field, so unless you have added one, Salience will not report response times rather than inventing them. Everything else (status, path, client, user agent, size) is there by default.
Can I import the logs I already have?
Yes, and it is worth doing. Rotated and archived access logs upload with automatic format detection, deduplicate against anything already streaming, and do not count against your monthly request quota. Your baseline then starts with real history.
Does this cover the error log too?
No. Salience analyses the access log: the record of requests and what your server answered. Apache's error log is a different file with a different purpose and is not ingested.
How is this different from GoAccess or AWStats?
Those parse and summarise the file you point them at, on demand. Salience verifies claimed bot identities against published address ranges, keeps a baseline across weeks and months, joins the log to your sitemap and Search Console, watches every server in one place, and tells you when something changes instead of waiting for you to look.
What happens to our data, and where is it stored?
You are the controller; Salience processes only on your instructions under a GDPR Art. 28 DPA that applies to every account. Data is stored encrypted in AWS eu-west-2 (London), never sold, never used for advertising and never pooled between customers.
Can I run this on several servers?
Yes, and pricing does not penalise it: plans are priced on requests, not per host. Each server or site appears separately, and everything normalises to one request model so a report reads the same whatever produced the underlying line.
Can we remove it later?
Stop the agent and collection stops immediately. Nothing was changed in Apache, so there is nothing to unwind; uninstall the package and the server is exactly as it was.
Your logs already show what Google and the AI crawlers are doing.
Any Linux server you control, live in about ten minutes with no change to your Apache configuration. Archived logs import too.