Analyse your Nginx access log for bots, crawlers and errors
Salience streams your existing access log and turns it into verified bot identities, crawl and error analysis, and alerts that name the addresses, paths and user agents involved, without touching your Nginx configuration or adding anything to the request path.
Any server running Nginx, live in about ten minutes with no config change. Archived logs import too.
The Log Explorer page for demo-site.example once Nginx is connected through Vector agent: the Website setup strip shows Logs Receiving, and the table fills with Googlebot requesting /products/wireless-headphones, status 200, Verified; ClaudeBot requesting /guides/setup/, status 200, Verified; Googlebot (claimed) requesting /admin/, status 404, Unverified; Bingbot requesting /old-product-url, status 404, 404 spike. First hour of the tail: 22,116 requests.
Nginx records every request, and most of that log goes unread
Whether it is fronting an application, terminating TLS or serving static files, your Nginx writes a complete record of every request. That file is usually consulted only after something has gone wrong.
Tailing and grepping a log during an incident, reconstructing what happened after the fact rather than being told while there is still time to act.
Treating the user-agent field as an identity, because verifying a claimed crawler against published address ranges by hand, across millions of lines, is not realistic.
Watching request-rate graphs in your monitoring stack that show traffic is up, without any way to say which clients, on which routes, or whether they were genuine.
Losing the week you actually wanted to compare against, because logrotate already compressed and discarded it.
Salience reads the same log as it is written
Lines stream from the files your access_log directives already write: humans, verified crawlers, AI agents and suspicious automation, separated and counted.
The Salience dashboard for demo-site.example, last 24 hours. Website setup 5 of 6 completed. Traffic over time in fifteen-minute buckets, human and bot requests stacked. Bot identity evidence: 12.1K verified requests, 762 unverified claims, 2.7K with verification unknown. Total requests 34.2K, up 7.0% on the previous period. Unique IPs 15.3K, up 2.0%. Bot traffic 45.5%, down 1.9 points. AI crawlers 18.5%, down 8.0 points.
Lines reach Salience within seconds of Nginx writing them and show in reports within about fifteen minutes. Anomaly detection begins once about 24 hours of baseline has built.
Check that a crawler is who it says it is
Your log records what each client claimed to be, and that is all it is: a claim. Salience checks every claimed identity against the provider's own published address ranges, separating genuine Googlebot, Bingbot, GPTBot and ClaudeBot from the scrapers borrowing their names, and lists the impostors with the addresses responsible.
Bots and Crawlers on demo-site.example, last 24 hours: 15,541 bot requests, 12,233 verified, 655 failed verification, 2,653 other status. Served versus rejected: Googlebot 357 served, 34 rejected, Impersonators only, 0 verified · 34 unverified; Amazonbot 3,934 served, 16 rejected, The real bot, 16 verified · 0 unverified; Attack Path Probe 413 served, 104 rejected, Not verifiable, 0 verified · 104 unverified.
AmazonbotSee traffic by route and endpoint
Nginx usually sits in front of something: an application, an API, a set of upstreams. Activity is broken down by route and URL pattern, so an endpoint starting to return 5xx, or an API path suddenly carrying ten times its usual traffic, is a named finding rather than a bump on a graph.
Status code detail for 4xx Client Error on demo-site.example, last 24 hours: Total Requests 3,272, % of All Traffic 9.58%, Unique Codes 4. Individual status codes: 403 2,337, 71.4% of 4xx; 404 896, 27.4% of 4xx; 405 29, 0.9% of 4xx; 499 10, 0.3% of 4xx. Top paths: /old-product-url 155, /blog/2019-relaunch 142, /guides/pricing-2025 119, /products/legacy 98, /sitemap-old.xml 90, /blog/page/48 42.
Import the logs you already have
Rotated and archived access logs upload directly, with format detection and deduplication against anything already streamed. Your baseline starts with real history rather than building from zero, and imports never count against your monthly request quota.
Import Logs on demo-site.example. Import jobs and current progress. Each job retains its own source dates. Import Historical Logs: Upload CloudFront or Apache/Nginx log files, or a log-analyser Events CSV, to analyse historical traffic data. Drag & drop log files, or click to browse (multiple files supported). Supported formats: CloudFront, Apache/Nginx, log-analyser Events CSV (.log, .txt, .csv, .gz, .zip). Supported formats: CloudFront standard/real-time logs, Apache/Nginx Combined Log Format, log-analyser Events CSV Auto-detection: We'll automatically detect the log format and parse the data Compressed files: .gz and .zip files are supported and will be decompressed automatically
How it works
Connect
Install the open-source Vector agent on the server and point it at the files your access_log directives already write. Guided instructions and a ready-made configuration are in the app; Nginx itself is untouched.
We analyse
Each line is classified against the crawler registry, verified against published provider ranges, grouped by route and section, and compared with your own history, sitemap, robots.txt and Search Console data.
Get answers
Dashboards, alerts by email, Slack or webhook, plain-English querying, an API and a CLI. The connection confirms within seconds, and the first reports fill within about fifteen minutes of the agent starting.
Specification
- Collection method
- The open-source Vector agent, tailing the access log you already write
- Data freshness
- Within seconds
- Typical setup time
- About 10 minutes
- Plan requirement
- Any server running Nginx that you control
- Permissions needed
- Root or sudo to install Vector, and read access to the log files
- Code or DNS changes
- No Nginx configuration changes and no DNS changes. An agent is installed on the server
- Historical import
- Yes. Rotated and archived access logs upload with deduplication
- Key limitation
- Access logs only: Nginx's error log is not ingested. Behind another proxy the log records the proxy's address unless real-IP handling is configured, and Salience reports what the log says.
Which of these is you?
Site reliability and platform teams
Our metrics said CPU was high. They did not say we were being walked through every endpoint by one client.
Error rates, latency and traffic anomalies detected against route-level baselines, with the clients and paths responsible named in the alert.
Site reliability
Security teams
Everything claims to be Googlebot. Half of it is not, and I need the addresses, not a percentage.
Probe patterns, exposed-secret fetches and impersonation surges surfaced with named source addresses, escalated if anything returned a 200.
Threat intelligence
SEO teams
I need to know what Googlebot actually did on this application, not what a crawl simulation predicts.
Crawl activity by route and URL pattern, crawl waste, and error spikes caught against your baseline, joined to your sitemap.
Search intelligence
AI and content teams
Which AI systems are reading this, and is it training or somebody's live question?
GPTBot, ClaudeBot, PerplexityBot and the rest, separated into training, AI-search indexing and user-triggered retrieval.
AI access intelligence
Why not just use GoAccess, stub_status or grep?
They are good tools, and for their jobs they are the right ones. GoAccess gives you a genuinely useful real-time terminal dashboard for nothing, stub_status is the cheapest possible way to see connection state, and grep remains unbeatable when you already know exactly what you are looking for. On a single server, in the hands of someone comfortable at the command line, they cover a lot of ground. The difference shows up in three places: verification, memory, and being told rather than looking.
Parse and summarise the log
- GoAccess, stub_status, grep
- Yes, and quickly
- Salience
- Yes
Live connection and request rates
- GoAccess, stub_status, grep
- stub_status does this well
- Salience
- Request-level, with the clients named
Bot identity verified against published ranges
- GoAccess, stub_status, grep
- User agent taken at face value
- Salience
- Checked against official ranges, 200+ identities
Search and AI crawler interpretation
- GoAccess, stub_status, grep
- Counted as user-agent strings
- Salience
- Crawl budget, index context, training versus retrieval
Alerting when something changes
- GoAccess, stub_status, grep
- Not the design: you run the tool
- Salience
- 16 detectors on a 5-minute cycle, each naming a next step
Baseline across weeks and months
- GoAccess, stub_status, grep
- Bounded by what logrotate kept
- Salience
- Retained per plan; archives import and deduplicate
Every server and site in one view
- GoAccess, stub_status, grep
- One host at a time
- Salience
- One organisation, one request model
When the native tooling is all you need
If you run one server, you live in the terminal, and your question is usually "what is happening right now" rather than "what changed this week and who needs to know", GoAccess and stub_status will serve you well and cost you nothing. Salience earns its place when the log has to answer to more than one person, across more than one machine, without anyone having to remember to look.
Facts about the integration
No customer logos and no five-star quotes on this page. These are properties of the product you can check on the free tier in an afternoon.
Illustrative, not a customer incident
An API route that normally serves a few hundred requests an hour starts serving forty thousand, all from one address rotating user agents, all returning 200. Request-rate graphs show a spike and no more than that. Against a five-minute baseline it is an alert in the first window, naming the route, the address and the fact that every request succeeded, which is the part worth knowing.
Everything included
One connection, one request stream. No per-feature setup and no second pipeline to maintain.
Verified bot and AI crawler identities
Claimed identities checked against official IP ranges where providers publish them; anything unverifiable is marked unverified rather than guessed at.
Default and custom log formats
The standard combined format works as written, and custom log_format directives are welcome. Nothing about your configuration needs to change.
Route and section analysis
Activity grouped by route, directory and URL pattern, so findings arrive with the endpoint named rather than as a site-wide total.
Error, latency and status analysis
2xx through 5xx per route, with spike detection against your own baseline rather than a fixed threshold.
Threat and probe detection
sqlmap-style requests, /.env and config fetches, credential-stuffing patterns and impersonation surges, escalated if anything returned a 200.
Alert detectors on a five-minute cycle
Traffic-drop alerts check the server is reachable before they fire, probe alerts check whether the flagged path really serves anything, and every alert is enriched first, so you never get "your server might be down" when it is not.
Historical import with deduplication
Rotated and archived logs upload with format detection, merge cleanly against streamed data, and never count against your quota.
Many servers, one account
Every Nginx host, and any Apache, Cloudflare, Vercel or CloudFront source, normalised to one request model in one organisation.
Alerts by email, Slack and webhook
Dashboards, email and Slack alerts, webhooks, a public API, a CLI, an MCP server, CSV exports and shared read-only dashboards.
Recommendations with a rule for Nginx
Addresses to block and fake crawler user agents, scored from the last 30 days of alerts, with an Nginx rule generated from the list and ready to paste.
Saved segments
Page groups you define by prefix, pattern or query string, with a one-click library of common groups, applied to all history and used in every report, alert and export.
Site checks every night
Twenty-one checks on crawler access, security hygiene and serving quality, each pass, warn or fail with the evidence, 30 days of history and an alert when a verdict changes.
Weekly report and daily digest
A weekly email report per site, a daily digest of lower-severity alerts, and email for anything above the severity you set.
Trust & data protection
Privacy and data protectionYou are the controller
We process only on your instructions. GDPR Art. 28 DPA on every account, nothing to sign.
UK data residency
AWS eu-west-2 (London). Encrypted in transit (TLS 1.2+) and at rest (AES-256).
Server-side collection
No browser tracking script and no client-side pixel.
No sale, no pooling
Your logs are never sold, never used for advertising, never shared between customers. DPA, sub-processor list and security overview available.
What is collected
- Timestamp, method, host and path
- Status code and response size
- Client address and user agent
- Referrer
What is never collected
- Nginx's error log
- Request and response bodies
- Response timing, unless your log format records it
Priced on requests, not seats, and not per server: connect as many Nginx hosts as you like on one plan. Start on the free tier with 500,000 requests a month and upgrade when your traffic does.
Crawler intelligence from $19/mo.
Priced on requests, not seats. Unlimited users on every plan except Solo. Start on the free tier and upgrade when your traffic does. Free trial, no card needed.
- 500K requests/mo
- 1 site
- 30 days history
- Unlimited users
- Real-time analytics, bot and AI detection
- 20M requests/mo
- 5 sites (+5)
- 1 year history
- Unlimited users
- Real-time analytics with bot and AI-crawler verification
- 100M requests/mo
- 15 sites (+15)
- 2 years history
- Unlimited users
- AI allowance: ~600 answers or ~60 reports a month
- 500M requests/mo
- 50 sites, no ceiling
- 4 years history
- Unlimited users
- AI allowance: ~3,000 answers or ~300 reports a month
- 1B requests/mo
- Unlimited sites
- Custom history
- Unlimited users
- SSO / SAML and audit log
Common questions
Do I have to change my Nginx configuration?
No. The standard combined format is supported as written and custom log_format directives are welcome. The agent reads the files your existing access_log directives already produce; Nginx is not modified and is not restarted.
My Nginx is behind another proxy. Will the client addresses be right?
They will be whatever your log records. Behind a load balancer or CDN, Nginx logs the proxy's address unless real-IP handling is configured, and if that is the case, Salience reports what the log says rather than guessing at the original client. It is worth checking before you connect; the setup guide covers it.
How long does setup take, and do I need a developer?
About ten minutes for someone with root or sudo on the server. The app generates a ready-made agent configuration, so the work is installing a package and pointing it at a file. If that needs to go to an engineer, you can delegate it from inside Salience without sharing your account.
Will the agent affect server performance?
It reads files and ships lines over HTTPS, which is light work on any modern server. It does not sit in the request path, so it cannot slow a response down or fail a request. If the agent stops, Nginx carries on exactly as before and you simply stop receiving new data.
Can I see response times?
Only if your log_format records them. Timing fields such as $request_time exist only where you have added them, so unless they are in your format, Salience will not report response times rather than inventing them. Everything else (status, route, client, user agent, size) is there by default.
Can I import the logs I already have?
Yes, and it is worth doing. Rotated and archived access logs upload with automatic format detection, deduplicate against anything already streaming, and do not count against your monthly request quota. Your baseline then starts with real history.
Does this cover the error log too?
No. Salience analyses the access log: the record of requests and what your server answered. Nginx's error log is a different file with a different purpose and is not ingested.
How is this different from GoAccess?
GoAccess parses and summarises the file you point it at, on demand, on one host. Salience verifies claimed bot identities against published address ranges, keeps a baseline across weeks and months, joins the log to your sitemap and Search Console, watches every server in one place, and tells you when something changes instead of waiting for you to look.
What happens to our data, and where is it stored?
You are the controller; Salience processes only on your instructions under a GDPR Art. 28 DPA that applies to every account. Data is stored encrypted in AWS eu-west-2 (London), never sold, never used for advertising and never pooled between customers.
Can we remove it later?
Stop the agent and collection stops immediately. Nothing was changed in Nginx, so there is nothing to unwind; uninstall the package and the server is exactly as it was.
Your logs already show what Google and the AI crawlers are doing.
Any server running Nginx, live in about ten minutes with no config change. Archived logs import too.