Vercel server log monitoring
See every crawler, bot and error on your Vercel site.
Vercel's dashboard shows how each deployment and function behaved. Salience shows which crawlers, AI bots and scrapers requested every route, and what each deploy changed for them.
- Vercel Pro or Enterprise
- No code change or redeploy
- Live within seconds
Log Explorer
docs.example · requests in real time
GooglebotVerified/docs/routing/layouts200
OAI-SearchBotVerifiedAI search/docs/getting-started200- GooglebotUnverified/api/docs/search?q=install200
- GooglebotVerified/docs/v2/configuration404
- GPTBotVerifiedTraining/blog/edge-caching200
BingbotVerified/changelog200
Requests to /api/docs/search from 198.51.100.61, an address outside Google's published ranges.
Recommended: review the address in Recommendations and paste the generated middleware.ts rule.
Vercel Observability and Salience answer different questions
Vercel's dashboard is organised around deployments: what a function did, how a route performed and how fast pages loaded for visitors. Salience is organised around the clients making requests: who each one was, whether its claimed identity checks out, and how this month compares with the months before.
Vercel Observability
Runtime logs, Web Analytics and Speed Insights in your Vercel dashboard
- Why did this function fail, and what did it log?
- Which routes are slow or returning errors on this deployment?
- How many people visited, and from which referrers and countries?
- How fast do pages load for real visitors?
Runtime logs are kept for 1 day on Pro and 30 days with Observability Plus.
Salience
Every request your deployments serve, delivered by a Vercel Drain
- Which crawlers requested which routes, and were they who they said they were?
- What status code did each crawler get back before and after a deploy?
- Which AI systems request your pages, and which pages do AI assistants send people to?
- Which addresses are working through the site page by page?
Salience keeps request history for 30 days on Free and up to 4 years on paid plans.
Questions Next.js teams ask about their traffic
Four examples from the four teams that use Salience on a Vercel site. The site, addresses and figures in the panels are examples.
01Search and SEO
Is Googlebot spending its time on your real pages?
Next.js App Router pages also answer React Server Component requests at the same path with an ?_rsc= parameter, and site search creates a URL for every query. Googlebot can spend a large share of its requests on those URLs and reach new or changed pages late.
Salience records every verified Googlebot request with the URL, the status code it received and the time, groups the requests by section, and compares the URLs Googlebot fetched with the URLs in your sitemap.
Find sitemap pages Googlebot has not fetched, see how much crawling goes to RSC payload and site search URLs, and see error responses to Googlebot on the day a deploy introduces them.
Verified Googlebot by section
docs.example · last 7 days
| Section | Requests | URLs | Unfetched |
|---|---|---|---|
| Docs/docs/* | 5,208 | URLs fetched1,412 | Sitemap URLs not fetched37 |
| Blog and changelog/blog/*, /changelog | 1,174 | URLs fetched186 | Sitemap URLs not fetched0 |
| RSC payload requests?_rsc= | 2,386 | URLs fetched1,647 | Sitemap URLs not fetchedNot in sitemap |
| Site search/docs/search?q= | 1,893 | URLs fetched1,240 | Sitemap URLs not fetchedNot in sitemap |
Sources requesting the docs
docs.example · last 24 hours
Requests per hour from unverified automation
peak 2,744
Unverified automated requests ran at about 100 an hour, rose to between 2,630 and 2,744 an hour from 01:00 to 06:59, then returned to about 100.
- 203.0.113.0/24Not a crawler14,212
Hosting provider · 3 rotating browser user agents · Every /docs/* URL in order, no /_next/static assets
- 198.51.100.61Unverified2,688
Hosting provider · Googlebot (claimed) · /api/docs/search?q= with dictionary words
- GooglebotVerified1,537
Google · Googlebot · Docs and blog pages, follows robots.txt
- 192.0.2.140Not a crawler1,071
Hosting provider · Go-http-client/1.1 · /api/og?title= once for every docs page
02Scrapers and bot threats
Who is requesting every page on your site?
Scrapers copy docs and content page by page, often from datacentre addresses, rotating browser user agents or borrowing Googlebot's name. Requests that reach a function, an API route or image optimisation also count towards your Vercel usage.
Salience groups requests by source address and user agent, checks every claimed crawler name against the operator's published addresses, and shows each source's paths, request rate and timing. The evidence is the address, the network, the user agent and the pattern of requests.
Block or rate-limit the sources in your own Vercel project. Recommendations list the addresses and fake crawler user agents and generate a middleware.ts rule for you to review and paste.
03AI search and discovery
Which AI systems are reading your docs and pages?
AI companies run separate crawlers for collecting training data, for building an AI search index and for fetching a page when someone asks an assistant a question. Each one is a separate robots.txt decision.
Salience puts every AI crawler in one of those three classes, verifies it where the operator publishes its addresses, and lists the pages each one requested.
A recorded request shows that a page was fetched. It does not show whether an answer used or cited the page.
The AI referral traffic report counts the human visits that arrive from ChatGPT, Perplexity, Claude, Gemini, Copilot and other assistants, by landing page.
AI crawlers by purpose
docs.example · last 7 days · verified requests
- GPTBot1,846
ClaudeBot1,203
Most requested /docs/api-reference
- OAI-SearchBot534
PerplexityBot417
Most requested /docs/getting-started
- ChatGPT-User268
- Claude-User59
Most requested /docs/routing/layouts
04Engineering and operations
What broke after your last deploy?
A deploy can change routes, redirects in next.config or vercel.json, middleware or data fetching. Cached pages keep working while pages rendered on demand fail, and crawlers request the long tail of pages that people rarely open, so they often receive the errors first.
Salience tracks status codes per path and per type of client against your own history. Site events mark deploys on every chart and can be added from the public API or the CLI as a step in your deploy pipeline. Server Error Spike, Crawler Error Spike and Elevated 404s alerts fire against your baseline.
The alert names the paths, the clients affected and when the errors started, with a recommended next step.
Server errors (5xx) by client
docs.example · daily share · 14 days
- Verified search crawlers
- 0.2% 14.6%
- Verified AI crawlers
- 0.3% 18.9%
- Human visitors
- 0.1% 1.2%
- Verified search crawlers
- Verified AI crawlers
- Human visitors
- Site event · production deploy
Docs pages rendered on demand are returning 500 since the 11:20 production deploy. Verified crawlers received most of the errors.
Recommended: check the latest deploy for changes to data fetching on /docs/[slug], or roll back to the previous deployment.
Example investigation
Googlebot started receiving 403s after a middleware change
An example of how an alert becomes a fix, step by step. The site, URLs and figures are illustrative.
- 1
The alert fires
At 16:05 a Crawler Error Spike alert reports that verified Googlebot requests to docs URLs are returning 403 far above the site's usual rate.
- 2
The crawler is Google's
Every request came from an address inside Google's published Googlebot ranges, so these are Google's own fetches being refused.
- 3
The affected URLs
73 requests across 14 docs URLs. Each one served a 200 to Googlebot before the afternoon's deploy.
- 4
When it started
The first 403 arrived at 15:34, two minutes after a deploy recorded as a site event at 15:32. That deploy added a user-agent rule to middleware.ts to block scrapers. The 403s continued through the evening.
- 5
The next step
Exempt verified crawlers from the user-agent rule in middleware.ts and redeploy, then check that Googlebot receives 200 on the same 14 URLs on its next visits.
Crawler Error Spike
docs.example · opened 16:05 · Googlebot · 403
GooglebotVerified66.249.66.0/24 · in Google's published ranges- /docs/routing/middleware4039
- /docs/getting-started4038
- /docs/api-reference/config4037
- /docs/caching/revalidation4037
- /docs/deployment/environment-variables4036
- /docs/data-fetching/server-actions4036
- 8 more URLs · 30 requests
15:32 Site event · middleware.ts deploy
3 Googlebot 403s in the 15:00 hour, all after the middleware deploy at 15:32; then 11, 14, 12, 10, 9, 8 and 6 per hour from 16:00 to 22:00, 73 in total.
Recommended next step
Exempt verified crawlers from the user-agent rule in middleware.ts, redeploy, then confirm Googlebot receives 200 on the 14 URLs.
What Salience shows for a Vercel site
Every report comes from the same request data. Search Console and Google Analytics 4 add search and session figures to the same URLs if you connect them.
URL detail
docs.example/docs/api-reference/cli · last 28 days
/docs/api-reference/cli
- LogsVerified Googlebot fetchesLast 200 at 07:4617
- LogsAI page fetchesChatGPT-User, Claude-User9
- LogsAI-referred visitschatgpt.com, claude.ai5
- Search ConsoleImpressions, 28 daysIndexed, last inspected 6 Oct2,740
- Google AnalyticsSessions, 28 daysOrganic search 64%488
Who is requesting
Verified crawler identities
200+ crawler identities, checked against published address ranges where the operator publishes them.
AI crawlers and AI referrals
Training, AI search and user-triggered fetches separated, with AI-referred visits by landing page.
Scraper and automation patterns
Systematic requests grouped by address, network and user agent.
What they request
Every request, URL by URL
Log Explorer and a URL page for each path, with status, client, cache status and source.
Docs, blog, search and parameter URLs
One click adds segments for blog, guides and help, pagination, search results and parameter URLs, applied to all history.
Sitemap and robots.txt
Sitemap and robots.txt checks, with crawl activity set against sitemap coverage.
What happens next
Status codes and errors
Status codes per path, soft 404s and redirects against your own baseline.
Alerts and recommendations
Alerts by email, Slack or webhook with a recommended next step, and middleware.ts or vercel.json snippets generated from the recommendations.
Reports
A weekly email report, a daily digest and a written technical SEO report.
Salience also runs its own site crawler and crawl audit, and has a public API, a CLI and an MCP server.
How to set up server log monitoring on Vercel
Setup takes about 10 minutes, with no code change, package or redeploy. The app walks you through each step and checks the connection at the end.
- 1
Enter your domain and create your account
Use the domain your production deployment serves, such as yoursite.com, and choose Vercel as the source. Salience shows an endpoint URL and a signing secret. The Free plan needs no card.
- 2
Create a Drain in Vercel
In the Vercel dashboard open Settings, Observability, Drains and create a Drain for Logs, with the projects, all sources and the environments you want. Choose HTTP delivery, paste the endpoint URL and signing secret, and set the format to NDJSON.
- 3
Confirm requests are arriving and start investigating
The connection check sends three test requests to your site and confirms they reach Salience. From then on, requests appear in Log Explorer within seconds.
What you need
- A Vercel team on the Pro or Enterprise plan
- Owner or admin access to the Vercel team
- The domain your production deployment serves
What stays the same
- No package, SDK or code change
- No redeploy and no DNS change
- Vercel keeps serving every request
Before you create the Drain
- Leave the sampling rules empty, so the Drain sends every request
- Select Preview as well as Production if you want preview deployments covered
- Requests a CDN in front of Vercel answers from its own cache never reach Vercel's logs
- History starts when the Drain is created
Someone else administers your Vercel team? Send them a delegated setup link from the app. It is limited to this site and lasts 30 days, and they create the Drain without access to your Salience account. The same step can hand the connection to an AI agent such as Claude Code, Codex or Cursor.
How your request data is collected and controlled
What is collected
- Timestamp, method, host, path and query string
- Status code and response size
- Client address, country and user agent
- Cache status and the Vercel region that served the request
What is not collected
- Request and response bodies
- Function console output and stack traces
- Requests answered by a CDN in front of Vercel
- Response times, which Drains do not carry
Where it is processed
- AWS eu-west-2 (London), with encrypted S3, DynamoDB and Aurora storage
- A GDPR Article 28 DPA applies to every account
- Never sold, used for advertising or shared between customers
Who controls it
- The Drain lives in your own Vercel team
- Each batch is signed, and Salience checks the signature against your secret
- Deleting the Drain stops collection straight away
Vercel Drains pricing
Drains are available on Vercel's Pro and Enterprise plans, and not on Hobby or a Pro trial. Vercel bills Drains at $0.50 per GB, measured on the uncompressed JSON of each record sent, with no included allowance on Pro. Sampling rules on the Drain reduce that volume, and Salience then receives only the sampled requests.
Vercel questions
Do I need a paid Vercel plan?
Yes. Vercel offers Drains on the Pro and Enterprise plans. Hobby and Pro trial teams need to upgrade to Pro before a Drain can be created. Vercel bills Drains at $0.50 per GB of log data.
Does this need a code change or a redeploy?
No. The Drain is configured in your Vercel dashboard and points at an endpoint Salience generates. There is no package to install, no change to next.config or middleware, and no redeploy. Setup takes about 10 minutes, and a delegated setup link lets someone else in your team complete it.
Which requests does the Drain include?
Static, function, edge and external rewrite requests, from the projects and environments you select, including responses served from Vercel's cache. Requests a CDN in front of Vercel answers from its own cache never reach Vercel and are not included.
Are preview deployments included?
Yes, if you select the Preview environment when you create the Drain. Production, Preview and Development can each be selected.
Can I see Googlebot and AI crawlers?
Yes. Every request from Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and 200+ other crawlers is recorded with the URL and the status code it received, and checked against the operator's published addresses where they publish them. AI crawlers are split into training, AI search and user-triggered fetches. A recorded fetch shows that a page was requested. It does not show whether an answer used it.
Can I see requests from before I connected?
Collection starts when the Drain is created; Vercel keeps no request history that a Drain can send. If you also serve traffic from Apache, Nginx or CloudFront, their historical log files can be imported.
Does Salience show response times for a Vercel site?
Vercel Drains do not include response timings, so the Response Times report and latency alerts have no data for a Vercel site. Status codes, cache status, crawlers and every other report work from the Drain.
What does monitoring cost?
The Free plan covers one website, 500,000 requests a month and 30 days of history, with no card. Paid plans start at $19 a month, each with a 14-day trial. On the Vercel side, Drains need a Pro or Enterprise team and are billed by Vercel at $0.50 per GB.
Start monitoring your Vercel site for free.
The Free plan needs no card. Setup needs a Vercel Pro or Enterprise team and about 10 minutes.