Nginx server log monitoring
See every crawler, bot and error on your Nginx server.
Salience streams the access log Nginx already writes, on a web server or a reverse proxy in front of your app. Each request is sorted into verified crawlers, AI systems, scrapers and people.
- No Nginx configuration change
- Combined log format as written
- Archived logs import too
Log Explorer
app.example · requests in real time
GooglebotVerified/docs/webhooks200
GPTBotVerifiedTraining/blog/2026-roadmap200
BingbotUnverified/templates/weekly-planner200- GooglebotVerified/templates/invoice-generator502
PerplexityBotVerifiedAI search/docs/getting-started200- sqlmapNot a crawler/.env404
Requests for /.env and /.git/config from 192.0.2.140. Salience fetched both paths and each returned 404.
Recommended: review the address in Recommendations and paste the generated Nginx deny rule.
GoAccess and Salience answer different questions about the same log
GoAccess, tail and grep read the access log on the server you run them on, at the moment you run them. Salience reads the same lines as Nginx writes them, checks each claimed crawler against the operator's published addresses and compares every path with its own history.
GoAccess, tail and grep
The access log on one server, read when someone runs the tool
- How many requests did the server answer, and with which status codes?
- Which paths, referrers and user agents appear most often in this log?
- What is the server answering right now?
- Which lines match a pattern you already know to look for?
GoAccess is free and open source, with a terminal dashboard and an HTML report.
Salience
Every line Nginx writes, streamed by the Vector agent as it is written
- Which crawlers requested which URLs, and were they who they said they were?
- Which paths started returning 5xx from the upstream, and to which clients?
- Which AI systems request your pages, and which pages do AI assistants send people to?
- Which addresses are probing for config files or working through your API?
Logs from every server and site you connect are kept in one account.
Questions your access log can't answer on its own
Four examples from the four teams that use Salience on a site behind Nginx. The site, addresses and figures in the panels are examples.
01Search and SEO
Where does Googlebot spend its requests on your site?
When Nginx fronts an app, Googlebot can spend its requests on login redirects, internal search results and parameter URLs, and reach new docs and pages late. Those requests sit in the access log beside scrapers that use Googlebot's name.
Salience records every verified Googlebot request with the URL, the status code it received and the time, groups the requests by section, and compares the URLs Googlebot fetched with the URLs in your sitemap.
Find sitemap URLs Googlebot has not fetched, see how much crawling goes to redirects and internal search, and see error responses to Googlebot on the day they start.
Verified Googlebot by section
app.example · last 7 days
| Section | Requests | URLs | Unfetched |
|---|---|---|---|
| Docs/docs/* | 3,904 | URLs fetched812 | Sitemap URLs not fetched17 |
| Templates/templates/* | 2,118 | URLs fetched640 | Sitemap URLs not fetched52 |
| App login redirects/app/* → 302 | 1,846 | URLs fetched1,530 | Sitemap URLs not fetchedNot in sitemap |
| Internal search/search?q= | 1,321 | URLs fetched1,207 | Sitemap URLs not fetchedNot in sitemap |
Sources requesting the site and API
app.example · last 24 hours
Requests per hour from unverified automation
peak 2,730
Unverified automated requests ran at about 100 an hour, rose to between 2,655 and 2,730 an hour from 01:00 to 06:59, then returned to about 100.
- 203.0.113.0/24Not a crawler14,212
Hosting provider · python-httpx/0.27 · /api/v1/templates/{id} for every id in order
- 198.51.100.61Unverified2,486
Hosting provider · Bingbot (claimed) · Every /templates/* URL in sitemap order
- GooglebotVerified1,512
Google · Googlebot · Docs, templates and blog, follows robots.txt
- 192.0.2.140Not a crawler1,377
Hosting provider · sqlmap/1.8 · /.env, /.git/config and admin paths, all 404
02Security and bot threats
Who is probing your server or working through your API?
Scanners request /.env, /.git/config and admin paths on every public server, and scripts page through API endpoints from datacentre addresses, some borrowing a crawler's name. In the raw log each of them is one more user agent string.
Salience groups requests by source address and user agent, checks every claimed crawler name against the operator's published addresses, and shows each source's paths, request rate and timing. Before a Suspicious URL Patterns alert escalates, Salience fetches the flagged paths itself to check whether they serve anything.
Block or rate-limit the sources in your Nginx configuration. Recommendations list the addresses and fake crawler user agents and generate deny lines or a user agent map for you to paste.
03AI search and discovery
Which AI systems are requesting your docs and pages?
AI companies run separate crawlers for collecting training data, for building an AI search index and for fetching a page when someone asks an assistant a question. Each one is a separate robots.txt decision.
Salience puts every AI crawler in one of those three classes, verifies it where the operator publishes its addresses, and lists the URLs each one requested with the status code it received.
A recorded request shows that a page was fetched. It does not show whether an answer used or cited the page.
The AI referral traffic report counts the human requests referred by ChatGPT, Perplexity, Claude, Gemini, Copilot and other assistants, by landing page.
AI crawlers by purpose
app.example · last 7 days · verified requests
- GPTBot1,644
ClaudeBot1,207
Most requested /docs/api/authentication
- OAI-SearchBot534
- PerplexityBot389
Most requested /docs/getting-started
- ChatGPT-User276
- Perplexity-User58
Most requested /pricing
04Engineering and operations
What changed after your last deploy or config reload?
A new limit_req zone, a deny rule or an upstream release can start refusing crawlers or returning 502s while the site still loads for the person who checks it.
Salience tracks status codes per path and per type of client against your own history. Site Events mark your deploys and changes on every chart, and the Edge Policy Change alert fires when verified crawlers start receiving 401, 403 or 429 responses.
The alert names the date, the crawlers affected and how their refused share compares with the usual level, with a recommended next step.
Requests refused (401, 403, 429) by client
app.example · daily share · 14 days
- Verified search crawlers
- 26.4% 24.9%
- Verified AI crawlers
- 47.1% 45.6%
- Human visitors
- 0.2% 0.3%
- Verified search crawlers
- Verified AI crawlers
- Human visitors
- Site event · limit_req zone added
Googlebot, Bingbot, GPTBot and ClaudeBot are receiving 429 responses. Human visitors are unaffected.
Recommended: review the rate limit added today and the crawlers it now refuses.
Example investigation
Googlebot started receiving 502s on template pages
An example of how an alert becomes a fix, step by step. The site, URLs and figures are illustrative.
- 1
The alert fires
At 14:20 a Crawler Error Spike alert reports that verified Googlebot requests to /templates/ URLs are returning 502 far above the site's usual rate.
- 2
The crawler is Google's
Every request came from an address inside Google's published Googlebot ranges. Nginx returned the 502 because the app behind it failed to answer these pages.
- 3
The affected URLs
67 requests across 9 template URLs. The rest of the site returned 200 to Googlebot over the same hours.
- 4
When it started
The first 502 arrived at 14:02, 12 minutes after the app deploy your team recorded as a Site Event at 13:50. The 502s continued through 20:00.
- 5
The next step
Fix or roll back the template rendering change in that deploy, then check that Googlebot receives 200 on the 9 URLs on its next visits.
Crawler Error Spike
app.example · opened 14:20 · Googlebot · 502
GooglebotVerified66.249.68.0/24 · in Google's published ranges- /templates/invoice-generator50212
- /templates/project-roadmap50210
- /templates/okr-tracker5029
- /templates/sprint-retrospective5028
- /templates/meeting-notes5027
- /templates/content-calendar-quarterly-planning5027
- 3 more URLs · 14 requests
13:50 Site event · app deploy
No 502s in the 13:00 hour before the deploy at 13:50; then 6, 11, 15, 13, 9, 8 and 5 per hour from 14:00 to 20:00, 67 in total.
Recommended next step
Fix or roll back the template rendering change in the 13:50 deploy, then confirm Googlebot receives 200 on these 9 URLs.
What Salience shows for a site behind Nginx
Every report comes from the same request data. Search Console and Google Analytics 4 add search and session figures to the same URLs if you connect them.
URL detail
app.example/docs/webhooks · last 28 days
/docs/webhooks
- LogsVerified Googlebot fetchesLast 200 at 14:2216
- LogsAI page fetchesChatGPT-User, Perplexity-User4
- LogsAI-referred visitschatgpt.com, perplexity.ai2
- Search ConsoleImpressions, 28 daysIndexed, last inspected 6 Oct2,240
- Google AnalyticsSessions, 28 daysOrganic search 64%418
Who is requesting
Verified crawler identities
200+ crawler identities, checked against published address ranges where the operator publishes them.
AI crawlers and AI referrals
Training, AI search and user-triggered fetches separated, with AI-referred visits by landing page.
Scanners and automation
Config-file probes, scanner user agents and API enumeration grouped by address, network and user agent.
What they request
Every request, URL by URL
Log Explorer and a URL page for each path, with status, client, timing and source.
Segments for every part of the site
One click adds segments for guides and help, blog, search results, pagination, parameter URLs and account pages, applied to all history.
Sitemap and robots.txt
Checked automatically when the site is added, with crawl activity set against sitemap coverage.
What happens next
Status codes and errors
Status codes per path, including 502 and 504 from the upstream, soft 404s and redirects against your own baseline.
Alerts and recommendations
Alerts by email, Slack or webhook with a recommended next step, and Nginx deny lines, a user agent map or 410 location blocks generated from the recommendations.
Reports
A weekly email report, a daily digest and a written technical SEO report.
Salience also runs its own site crawler and crawl audit, and has a public API, a CLI and an MCP server.
How to set up server log monitoring on Nginx
Setup takes about 10 minutes on a Linux server where you have root or sudo. The app generates the agent's configuration and checks the connection at the end.
- 1
Enter your domain and create your account
Choose Apache / Nginx (Vector) as the source. The app generates a vector.yaml file with this site's ingest token and your access log path. The Free plan needs no card.
- 2
Install Vector and add the config
Install the open-source Vector agent from its apt or yum repository, move vector.yaml to /etc/vector, give Vector read access to the access log, then run systemctl enable --now vector.
- 3
Confirm requests are arriving and start investigating
The connection check sends three test requests to your homepage and confirms they reach Salience. Requests then appear in the live view within about 30 seconds of Nginx writing them.
What you need
- A Linux server running Nginx: Debian, Ubuntu, RHEL, CentOS or Amazon Linux
- Root or sudo access to install Vector
- Outbound HTTPS on port 443 to api.salience.com
- The access log path, usually /var/log/nginx/access.log
What stays the same
- No change to your Nginx configuration and no reload
- No DNS change
- Vector reads the log file and is not in the request path
Before you start
- The generated config parses the combined log format. A custom log_format needs the parse step in vector.yaml edited to match
- Behind a load balancer or CDN, set up real-IP handling so the log records each client's own address
- vector.yaml holds the ingest token, so keep it readable by root only (chmod 600)
- Rotated and archived access logs can be imported, and lines already streamed are skipped
Someone else runs the server? Send them a delegated setup link from the app. It is limited to this site for 30 days and builds a ready-made vector.yaml for the log path they enter, without access to your Salience account.
How your request data is collected and controlled
What is collected
- Timestamp, method, host and path
- Status code and response size
- Client address and user agent
- Referrer
What is not collected
- Nginx's error log
- Request and response bodies
- Response timing, unless your log format records it
Where it is processed
- AWS eu-west-2 (London), with encrypted S3, DynamoDB and Aurora storage
- A GDPR Article 28 DPA applies to every account
- Never sold, used for advertising or shared between customers
Who controls it
- Vector runs on your own server, from a config file you hold
- Stopping the Vector service stops collection straight away
- The ingest token can be regenerated from the website settings
Nginx questions
Do I need to change my Nginx configuration?
No. The generated Vector config parses the combined log format, which Nginx writes unless you have set your own log_format. With a custom log_format, edit the parse step in vector.yaml to match it. Nginx is not modified or reloaded.
Does it work when Nginx is a reverse proxy in front of my app?
Yes. Nginx writes an access log line for every request it answers, whichever upstream served it, so the log records the status code each client received, including the 502 and 504 responses Nginx returns when the upstream fails or times out.
Nginx sits behind a load balancer or CDN. Will the client addresses be right?
Salience reports the address your log records. Behind another proxy, Nginx logs the proxy's address unless real-IP handling is configured. Requests that arrive from a proxy or CDN address are marked Proxied, because the real client address cannot be checked against a crawler's published ranges.
Can I see response times?
Only if your log_format records a timing field such as $request_time. The combined format has none, so for a combined log the response time reports show no figure. Status codes, paths, clients, user agents and response sizes are all in the combined format.
Can I import the logs I already have?
Yes. Rotated and archived Nginx access logs in the default or combined format upload from Import Logs, with the format detected automatically. Lines Salience has already received from Vector are recognised and skipped, so a file that overlaps the live stream can be imported safely.
Will the agent affect the server?
Vector reads the log file and sends batches of lines over HTTPS. It is not in the request path, so no response waits for it. If Vector stops, Nginx keeps serving and Salience receives no new lines until it starts again.
Can one agent send logs for several sites on the same server?
Yes. Add a source and a sink to vector.yaml for each site's access log, each with that site's own Salience token.
What does monitoring cost?
The Free plan covers one website, 500,000 requests a month and 30 days of history, with no card. Paid plans start at $19 a month, each with a 14-day trial. Vector is open source and free.
Start monitoring your Nginx server for free.
The Free plan needs no card. Setup needs root or sudo on the server and about 10 minutes.