See what crawlers and bots do on your Vercel site
Salience streams your Vercel request logs and turns them into verified bot identities, crawl and error analysis, and alerts that name the addresses, paths and user agents involved, across production and preview deployments apart, and no change to your application.
Configured in your Vercel team settings and live in about ten minutes with no code changes. Production and preview stay separate.
The Log Explorer page for demo-site.example once Vercel is connected through Vercel Drain: the Website setup strip shows Logs Receiving, and the table fills with Googlebot requesting /docs/getting-started, status 200, Verified; ClaudeBot requesting /blog/changelog, status 200, Verified; Googlebot (claimed) requesting /api/auth?next=, status 401, Unverified; Bingbot requesting /pricing-old, status 404, 404 spike. First hour, production: 21,760 requests.
- Cloudflare
- Shopify
- Apache
- Nginx
- Netlify
Every deploy changes what crawlers get back from your site
Vercel makes deploying trivial, which means the gap between a change going live and anyone noticing its effect on crawlers, errors and automated traffic is where most of the risk now sits.
Opening runtime logs during an incident, filtering by hand, and finding that the window you actually needed has already rolled out of retention.
Shipping a routing or middleware change and having no idea whether Googlebot is still reaching your pages until Search Console catches up two days later.
Watching a traffic graph rise without being able to say whether that was users, an AI crawler working through your content, or something scraping you.
Discovering that preview deployments have been getting crawled, or that bot traffic against production and preview has been quietly mixed together in whatever you were looking at.
See every request to your deployment, sorted by who sent it
Static and function requests stream in as they happen: humans, verified crawlers, AI agents and suspicious automation, separated and counted.
The Salience dashboard for demo-site.example, last 24 hours. Website setup 5 of 6 completed. Traffic over time in fifteen-minute buckets, human and bot requests stacked. Bot identity evidence: 12.1K verified requests, 762 unverified claims, 2.7K with verification unknown. Total requests 34.2K, up 7.0% on the previous period. Unique IPs 15.3K, up 2.0%. Bot traffic 45.5%, down 1.9 points. AI crawlers 18.5%, down 8.0 points.
Requests reach Salience within seconds and show in reports within about fifteen minutes. History begins at connection: Vercel exposes no past logs to backfill from.
Check that a crawler is who it says it is
Every claimed identity is checked against the provider's own published address ranges. Genuine Googlebot, Bingbot, GPTBot and ClaudeBot are separated from the scrapers borrowing their names, and the impostors are listed with the addresses responsible so you can rate-limit them rather than argue about them.
Bots and Crawlers on demo-site.example, last 24 hours: 15,541 bot requests, 12,233 verified, 655 failed verification, 2,653 other status. Served versus rejected: Googlebot 357 served, 34 rejected, Impersonators only, 0 verified · 34 unverified; Amazonbot 3,934 served, 16 rejected, The real bot, 16 verified · 0 unverified; Attack Path Probe 413 served, 104 rejected, Not verifiable, 0 verified · 104 unverified.
AmazonbotStatic, edge and function requests arrive in one stream
A Vercel Drain can carry requests served statically, at the edge and by serverless functions. Salience takes all of them into one stream, so a route reads the same however Vercel happened to serve it, and a crawler's path through the site is not split across three logs.
Path Analysis on demo-site.example, last 24 hours: Total Requests 34,163, Unique Paths 17,700, 4xx Errors 3,272, 5xx Errors 28. Top paths: / 1,704 requests, 5.0% of total, importance 7; /_next/static/chunks/main.js 1,245 requests, 3.6% of total; /robots.txt 376 requests, 1.1% of total; /docs/getting-started 202 requests, 0.6% of total, importance 5; /api/auth 183 requests, 0.5% of total; /blog/changelog 161 requests, 0.5% of total, importance 5; /favicon.ico 120 requests, 0.4% of total.
See a deploy that broke crawling within the hour
Status codes are analysed per path and per section against your own baseline. When a routing change starts feeding crawlers 404s, or a function begins returning 5xx on a live route, the spike is detected against your own history and the alert arrives with the exact URLs attached.
Status code detail for 4xx Client Error on demo-site.example, last 24 hours: Total Requests 3,272, % of All Traffic 9.58%, Unique Codes 4. Individual status codes: 403 2,337, 71.4% of 4xx; 404 896, 27.4% of 4xx; 405 29, 0.9% of 4xx; 499 10, 0.3% of 4xx. Top paths: /old-product-url 155, /blog/2019-relaunch 142, /guides/pricing-2025 119, /products/legacy 98, /sitemap-old.xml 90, /blog/page/48 42.
How it works
Connect
Add your site in Salience and create a Drain in your Vercel team settings pointing at the endpoint it generates. No code change, no dependency, no redeploy.
We analyse
Each request is classified against the crawler registry, verified against published provider ranges, grouped by route and section, and compared with your own history, sitemap, robots.txt and Search Console data.
Get answers
Dashboards, alerts by email, Slack or webhook, plain-English querying, an API and a CLI. The connection confirms within seconds, and the first reports fill within about fifteen minutes. Seconds of the Drain going live.
Specification
- Collection method
- A Vercel Drain pointed at your Salience endpoint
- Data freshness
- Within seconds
- Typical setup time
- About 10 minutes
- Plan requirement
- Vercel Pro or Enterprise. Drains are not available on the Hobby plan
- Permissions needed
- Vercel team access sufficient to create a Drain (owner or admin)
- Code or DNS changes
- None. No code changes and no DNS changes
- Historical import
- No. Data starts the moment the Drain is connected
- Key limitation
- There is no backfill: a Drain only delivers events from the moment it is created, so history builds from day one of the connection. Requests answered entirely outside your Vercel deployment, for example by a CDN in front of it, never reach Vercel's logs.
Which of these is you?
SEO teams
We ship several times a day. I need to know within the hour if a deploy broke crawling, not two days later.
Crawl activity by route, crawl waste and error spikes caught against your baseline, joined to your sitemap and Search Console.
Search intelligence
Engineering and platform teams
Runtime logs are great for debugging one deploy and useless for answering what changed this week.
Error rates, latency and traffic anomalies against route-level baselines that persist across deployments, with clients and paths named.
Site reliability
AI and content teams
Which AI systems are reading our content, is it training or retrieval, and are they genuine?
GPTBot, ClaudeBot, PerplexityBot and the rest, separated into training, AI-search indexing and user-triggered retrieval, with impostors flagged.
AI access intelligence
Security teams
Something is probing our API routes and I have graphs rather than addresses.
Probe patterns, exposed-secret fetches and impersonation surges surfaced with named source addresses, escalated if anything returned a 200.
Threat intelligence
Doesn't Vercel Observability already do this?
For what it is built for, yes, and well. Runtime Logs, the Observability filters, Web Analytics and Speed Insights give you a genuinely strong picture of a deployment: what a function did, how fast a page was, where errors happened. If your question is "why did this request fail", that is the right place to look and Salience is not trying to replace it. The difference is what happens over time. Vercel's tooling is oriented around the deployment; Salience is oriented around the request stream: who the clients were, whether their claims verified, and how this week compares with the last three months.
Debugging a specific request or function
- Vercel Observability
- Its core purpose, and the right tool
- Salience
- Not attempted
Bot identity verified against published ranges
- Vercel Observability
- Out of scope
- Salience
- Checked against official ranges, 200+ identities
Search and AI crawler interpretation
- Vercel Observability
- Requests are requests
- Salience
- Crawl budget, index context, training versus retrieval
Baseline across weeks and months
- Vercel Observability
- Retention is oriented to recent activity
- Salience
- Retained per plan, with anomaly detection against it
Joined to sitemap, robots.txt and Search Console
- Vercel Observability
- Out of scope
- Salience
- Correlated, so a change links back to its cause
Alerting with a recommended next step
- Vercel Observability
- Available around deployment and function health
- Salience
- 16 detectors on a 5-minute cycle, each naming a next step
Human behaviour and page speed
- Vercel Observability
- Web Analytics and Speed Insights
- Salience
- Not attempted: different question entirely
When the native tooling is all you need
If your question is about a deployment (did this build work, why did that function throw, how fast is this page), Vercel's own tooling is the correct answer and adding Salience will not help. Salience is for the questions that outlive a deployment: who is crawling us, is that genuine, what changed since last month, and who needs to be told.
Facts about the integration
No customer logos and no five-star quotes on this page. These are properties of the product you can check on the free tier in an afternoon.
Illustrative, not a customer incident
A middleware change ships at 14:00 and quietly starts redirecting a whole route segment. At 14:03 the first Googlebot 404s on the old paths are in the stream; at 14:08 the crawler error-spike alert fires with the exact URLs listed. Runtime logs would show the same requests, if someone happened to be looking at the right filter at the right moment.
Everything included
One connection, one request stream. No per-feature setup and no second pipeline to maintain.
Verified bot and AI crawler identities
Claimed identities checked against official IP ranges where providers publish them; anything unverifiable is marked unverified rather than guessed at.
Production and preview together
The Drain delivers requests from every environment you select, so a preview deployment that gets crawled shows up alongside production.
Static and function requests together
Both are part of one stream, so a route's behaviour reads the same whether it was served statically or rendered on demand.
Crawl budget by route
Where Googlebot's time actually goes across your routes, which URLs it never reaches, and where the budget is being wasted.
Error, latency and status analysis
2xx through 5xx per route, with spike detection against your own baseline rather than a fixed threshold.
Alert detectors on a five-minute cycle
Each alert names the affected addresses, user agents and paths, and carries a written triage note with a recommended next step.
Sitemap, robots.txt and Search Console alongside
Sitemap, robots.txt and Search Console joined to the request stream, so a ranking change links back to the deploy behind it.
Alerts by email, Slack and webhook
Dashboards, email and Slack alerts, webhooks, a public API, a CLI, an MCP server, CSV exports and shared read-only dashboards.
Recommendations to act on
Addresses to block, fake crawler user agents, 404 paths crawlers keep hitting and slow paths, scored from the last 30 days of alerts, with the sample user agent and paths behind each one.
Saved segments
Page groups you define by prefix, pattern or query string, with a one-click library of common groups, applied to all history and used in every report, alert and export.
Site checks every night
Twenty-one checks on crawler access, security hygiene and serving quality, each pass, warn or fail with the evidence, 30 days of history and an alert when a verdict changes.
Weekly report and daily digest
A weekly email report per site, a daily digest of lower-severity alerts, and email for anything above the severity you set.
Trust & data protection
Privacy and data protectionYou are the controller
We process only on your instructions. GDPR Art. 28 DPA on every account, nothing to sign.
UK data residency
AWS eu-west-2 (London). Encrypted in transit (TLS 1.2+) and at rest (AES-256).
Server-side collection
No browser tracking script and no client-side pixel.
No sale, no pooling
Your logs are never sold, never used for advertising, never shared between customers. DPA, sub-processor list and security overview available.
What is collected
- Timestamp, method, host and path
- Status code and response size
- Client address and user agent
- Production and preview, kept separate
What is never collected
- Request and response bodies
- Application console output and stack traces
- Requests answered by a CDN sitting in front of Vercel
Priced on requests, not seats, so a whole team can read the data on every plan except Solo. Start on the free tier with 500,000 requests a month and upgrade when your traffic does.
Crawler intelligence from $19/mo.
Priced on requests, not seats. Unlimited users on every plan except Solo. Start on the free tier and upgrade when your traffic does. Free trial, no card needed.
- 500K requests/mo
- 1 site
- 30 days history
- Unlimited users
- Real-time analytics, bot and AI detection
- 20M requests/mo
- 5 sites (+5)
- 1 year history
- Unlimited users
- Real-time analytics with bot and AI-crawler verification
- 100M requests/mo
- 15 sites (+15)
- 2 years history
- Unlimited users
- AI allowance: ~600 answers or ~60 reports a month
- 500M requests/mo
- 50 sites, no ceiling
- 4 years history
- Unlimited users
- AI allowance: ~3,000 answers or ~300 reports a month
- 1B requests/mo
- Unlimited sites
- Custom history
- Unlimited users
- SSO / SAML and audit log
Common questions
Do I need a paid Vercel plan?
Yes. Drains are a Pro and Enterprise feature, and they are not available on Hobby. That is Vercel's boundary rather than ours. If you are on Hobby and want this, putting Cloudflare in front of your deployment and using the Cloudflare integration works on any plan.
We already have Runtime Logs and Observability. What does this add?
Memory and identity. Vercel tells you what happened on a deployment; Salience tells you which named crawler that was, whether the claim verified against published address ranges, how it compares with your own baseline over months, and what to do about it, joined to your sitemap and Search Console.
Does this require a code change or a redeploy?
No. A Drain is configured in your Vercel team settings and points at an endpoint Salience generates. There is no package to install, no dependency added and no redeploy needed.
How long does setup take, and do I need a developer?
About ten minutes, and someone with owner or admin access to the Vercel team. There is nothing to write; if it needs to go to an engineer, you can delegate it from inside Salience without sharing your account.
Can we see traffic from before we connected?
No. Vercel does not expose past logs, so a Drain delivers only what happens after it is created and your history in Salience starts at connection. If you also run servers of your own, their historical Apache, Nginx or CloudFront logs can be imported alongside.
Are preview deployments included?
Yes, if you select the preview environment when creating the Drain. Preview deployments do get crawled, and their requests arrive in the same stream as production, attributed to the deployment host they were served from.
Does Salience see our application logs?
No. Salience reads request metadata: what was asked for, what your deployment answered, and who asked. Your application's console output, stack traces and any data inside a request or response body are never collected.
We have a CDN in front of Vercel. Will everything be captured?
Only what actually reaches Vercel. Requests a CDN answers from its own cache never touch your deployment and so never appear in its logs. If that describes your setup, connecting at the CDN layer (Cloudflare, for instance) gives the fuller picture.
What happens to our data, and where is it stored?
You are the controller; Salience processes only on your instructions under a GDPR Art. 28 DPA that applies to every account. Data is stored encrypted in AWS eu-west-2 (London), never sold, never used for advertising and never pooled between customers.
Can we remove it later?
Delete the Drain in Vercel and collection stops immediately. Nothing was added to your application, so there is nothing to unwind.
Your logs already show what Google and the AI crawlers are doing.
Configured in your Vercel team settings and live in about ten minutes with no code changes. Production and preview stay separate.