Salience

Find the pages Googlebot misses and the crawl it wastes.

Salience joins your live server logs with your sitemap, robots.txt and Search Console data, per URL. You see which important pages search engines never request, where crawl goes to parameters and dead URLs, and what broke after your last release.

Connect a log sourceExplore the demo

The free plan needs no card and adds nothing to your pages. First requests arrive in about ten minutes on most integrations.

The Bot detail page for Googlebot on demo-site.example, last 24 hours: 389 total requests, 324 unique paths, 4 status codes. Response status codes: 200 on 333 requests, 403 on 34, 301 on 21, 404 on 1. Top paths accessed: / 13 times, /robots.txt 7 times, /page-4c9184/page-5aee9d/page-a292ec, /page-32981a/page-734a38/page-5aee9d, /page-21232f/page-734a38/page-5aee9d and /page-e9cf80 3 times each. The last three requests: /page-e9cf80 returned 200; /page-e9cf80 returned 200; /page-4c9184/page-5aee9d/page-8e3f1b.html returned 404.

The search crawlers we verify
  • Googlebot
  • Bingbot
  • DuckDuckBot
  • Applebot

Compare the crawl you planned with the crawl you got

Your sitemap lists the URLs that matter, and your logs record which of them search engines actually request. Salience fetches the sitemap and joins it with live crawler activity, so the gaps between the two are listed per URL.

  • Important sitemap URLs that search engines have never requested, or have not requested for longer than you would expect, with the last crawl date
  • Low-value URLs requested repeatedly, such as parameters, facets, internal search, admin paths and redirect chains
  • Important URLs returning errors or responding slowly when a crawler asks for them
  • Crawled URLs that are missing from the sitemap
  • With a Site Crawler run or an uploaded crawl, what the crawler found at each URL next to whether search engines requested it: indexable, noindex, canonical to another URL or a redirect

Sitemap Coverage on demo-site.example: 31,041 Total Active URLs, 22,514 Recently Crawled, 7,912 Stale, 615 Never Crawled, 73% Coverage Rate. / first seen 23/01/2026, 135 recorded crawls, last crawled 15h ago by Bingbot, Crawled; /page-c0cb5f first seen 29/04/2026, 1 recorded crawls, last crawled 22/03/2026 by Googlebot Smartphone, Stale; /page-1ab3b7 first seen 25/02/2026, 2 recorded crawls, last crawled 25/02/2026 by Bingbot, Stale; /page-63dc90 first seen 14/02/2026, 2 recorded crawls, last crawled 14/02/2026 by Googlebot, Stale.

What this looks like in practice

Scenarioin-house technical SEO

I deployed sitemap changes yesterday. No idea if Googlebot has seen them. Search Console won't tell me for 48 hours.

A Friday URL change caught before Monday

  1. Fri 16:00A URL-structure deploy goes out with the redirect rules.
  2. Sat 08:00Salience alerts: Googlebot has requested 1,247 of the old URLs and received 404s, because the redirect rules missed one pattern.
  3. SatThe pattern is added and the redirects tested before Monday's traffic.

The same 404s would have reached Search Console on the Tuesday, after Google had begun dropping the URLs.

Scenariotechnical SEO agency

A crawler seat for every client and every consultant doesn't scale.

Fifty client sites in one account

  1. One Salience account holds all 50 client sites, and every consultant on the team has access.
  2. Each client gets a shared read-only dashboard link for their own site.
  3. When something breaks on any client's site, the whole team sees it as it happens.

One invoice covers the portfolio, with no per-seat licences and no desktop log analysis to re-run each month.

One request stream, tied to your website context.

Logs, sitemap, Search Console and robots.txt: correlated, so a ranking drop links back to the deploy or policy edit that caused it.

The Salience dashboard for demo-site.example, last 24 hours. Website setup 5 of 6 completed: Logs Receiving, Google Search Console Connected, Google Analytics Connected, Sitemap Checked, Robots.txt Checked, First site crawl Optional. Active incidents to review: 3 shown, each a Single /wp-login.php probe from AU IP returned 403; verification shows endpoint is protected. Showing 3 of 4 active, unacknowledged incidents. Review all in Alerts. Site checks: Checks need attention, 22 recorded checks: 16 passing, 5 warnings, 1 failing, 0 not applicable. Recorded 15 Sept 2026, 11:38 · independent of the dashboard period.

Catch regressions after releases and migrations

Crawler traffic shows an SEO regression before it reaches aggregated search reporting. Salience watches how crawlers are answered against a baseline and alerts when that changes.

  • New 404s and 5xx responses on paths crawlers were fetching successfully
  • Redirect chains and broken URL structures after a migration, with the internal links that cause them found by Crawl Audit
  • Crawler activity dropping away from a section
  • robots.txt and sitemap changes, tracked over time
  • Status codes that alternate between healthy and broken

Status Codes on demo-site.example, last 24 hours, the Status Codes Over Time chart: 24,447 2xx Success (71.6%), 6,416 3xx Redirect (18.8%), 3,272 4xx Client Error (9.6%), 28 5xx Server Error (0.1%), in half-hour buckets, stacked by status class.

The SEO Overview page for demo-site.example, last 24 hours: Crawler Requests 1,684, 3 unique crawlers; Pages Crawled 1,501, unique URLs discovered; Daily Crawl Rate 1,684, requests/day avg; Crawl Health 91.0%, 2xx success rate. 2xx Success 1,532, 91.0%; 3xx Redirects 112, 6.7%, wasting crawl budget; 4xx Errors 40, 2.4% broken pages; 5xx Server Errors —, None detected.

Crawl budget

See where Googlebot wastes your crawl budget.

Googlebot's time, by section, live.

Crawl budget is the time Google is willing to spend on your site. A slow response, a redirect chain and a dead URL each use some of it before Google reaches a page you want indexed. Salience shows how much of Googlebot's time reached those pages, and how much went to admin paths, redirect loops and dead URLs.

Search crawler intelligence

Investigate crawl waste by directory and URL pattern

Crawl budget is the time Google is prepared to spend crawling your site, and it matters most on large sites, sites that change often and sites that generate URLs through parameters and facets. Slow responses, redirect chains and error pages all use that time before Google reaches the pages you want indexed. Salience breaks Googlebot's time down by the parts of the site you think in, with the status and response time each part answered.

  • Product, category and content areas
  • Faceted navigation, parameters and internal search pages
  • Admin paths, redirects and error pages
  • File types and site directories
  • Auto-detected URL patterns, with high-cardinality sections flagged
  • Response time by section against its own baseline, so a section that has slowed shows up as time lost
  • Saved segments for the page groups you think in, defined by prefix, pattern or query string, with a one-click library of common groups and period-on-period comparison per segment

See which pages Google keeps coming back to

Page Importance scores every page from 0 to 10 by how often verified Googlebot and Bingbot return to it over the last 90 days. The score comes from your own logs, so it reflects what Google does on your site. With Search Console connected, each URL's index status is crossed with the fetches in your logs.

  • Page Importance per URL and per section, for Google and for Bing, with the distribution across the site
  • Pages Google has indexed but is no longer revisiting
  • Pages Google fetched and did not index, with the reason Search Console gives and the impressions each still earns
  • The Search Console queries that showed each page, and its daily impressions and clicks
  • Everything known about one URL in one place: requests, crawler fetches, crawl status, sitemap entry, index status, queries and importance

Cross your internal links with what search engines fetch

A Site Crawler run maps every internal link on the site. Salience crosses that graph with your logs, so each page carries its inlinks, click depth and internal PageRank next to the verified search, AI and human requests it received.

  • Pages with many internal links that search engines never fetch
  • Pages search engines fetch often that barely get linked
  • Orphan pages: fetched by search engines and linked from nowhere in the crawl
  • Internal links to broken pages, redirects and redirect chains, ordered by how often bots hit them

Verify the crawler before trusting the report

Any request can call itself Googlebot. Salience checks supported crawler identities against the provider's published IP ranges, or another reliable method where one exists, so a scraper using Googlebot's name does not pollute your crawl analysis.

Where Salience is most useful

Salience reads the logs of any site. These are the situations it was built around.

Large e-commerce catalogues

Find the products Googlebot never requests, the facet and parameter URLs where its requests go, and the discontinued URLs still drawing 404s.

Marketplaces and classifieds

Listing, filter and search URLs are grouped by pattern and high-cardinality sections are flagged, so crawl spent on expired listings and filter combinations is visible per section.

Publishers with deep archives

See which archive sections search engines still crawl, which they never reach, and which new URLs have been requested since they entered the sitemap.

International and multi-domain estates

Several sites sit in one account, each with its own sitemap join, dashboards and alerts, so a crawl problem is located to the domain and section where it occurs.

Sites mid-migration

Redirect chains, new 404s on old URLs and crawler activity moving between the old and new structures are shown against the migration as a site event.

SEO agencies

Shared read-only dashboards for clients, a client-ready Technical SEO Report generated on demand, weekly per-site reports, CSV exports and API access, and delegated setup links for whoever owns the client's CDN.

Why not the tools you already have

Most teams already own a site crawler, Search Console and somewhere the logs are kept. Each answers a different question from Salience.

Google Search Console

What it shows
Google's own crawling, aggregated, with a sample of example URLs, and reporting that runs behind the crawl.
What Salience adds
Every request from every search crawler, per URL, as your server answered it, joined to Search Console's own data.

Site crawlers

What it shows
What a crawler could fetch from your site when you run one.
What Salience adds
A Salience crawl, or a crawl export you upload from another crawler, joined to what search engines did request, when, and what they were given.

Generic log analysers and CDN analytics

What it shows
Raw requests, filtered by hand, with no sitemap, robots.txt or Search Console context.
What Salience adds
Crawler requests joined to sitemap, robots.txt, Search Console and site events, each with a verification status.

BigQuery or a self-built pipeline

What it shows
Engineering time to build and keep running, with crawler verification, classification and alerting left to you.
What Salience adds
Managed ingestion from your CDN or server, verified crawler identity and alerts when crawler behaviour changes.

Common questions

How is this different from Google Search Console?

Search Console's crawl stats report summarises Google's own crawling: request totals, response codes, response times, host status, crawl purpose and Googlebot type, with example URLs rather than the full list, over a limited window. Salience shows every request your server answered, for every search crawler, per URL, joined with your sitemap, robots.txt and Search Console data. The two work together, with Search Console giving Google's verdict and Salience showing the requests behind it.

Does crawl budget matter for a site my size?

Google's guidance is that crawl budget mainly matters for large sites, sites that change quickly and sites that generate many URLs through parameters or facets. It is a measure of Google's time, so a smaller site that answers slowly or through long redirect chains can still run short of it. Sitemap coverage, post-release regressions, intermittent errors and crawler verification apply at any size.

Can Salience identify fake Googlebot traffic?

Yes, for supported crawlers. Requests claiming to be Googlebot are checked against Google's published IP ranges, and each crawler carries a verification status. Traffic that fails verification is reported as unverified so you can treat it as a scraper rather than a search engine.

Does Salience crawl my website?

Only when you run the Site Crawler. Log analysis reads the access logs your CDN or web server already produces and adds no load of its own. The Site Crawler runs SalienceBot against your site on request, from one fixed IP address under a named user agent, so Crawl Join and Crawl Audit can compare every page and link with what search engines fetched. An export from another crawler, or a plain list of URLs, can be uploaded in place of a crawl. A handful of nightly site-check probes also fetch robots.txt and sitemap.xml, and can be switched off in the site settings.

Does it require JavaScript on my pages?

No. Collection is server-side, from your CDN or web server logs. There is no tracking script or pixel, so a request is captured whether or not the requesting system executes JavaScript, and whether or not your analytics tool filters it out as a bot.

Which platforms are supported?

Streaming integrations exist for Cloudflare, Vercel, AWS CloudFront, Netlify, Kinsta and Shopify (via Cloudflare), plus Apache and Nginx through a lightweight agent. Historical Apache, Nginx and CloudFront logs, and exports from other log analysers, can be imported.

How quickly does data appear?

It depends on the integration. Streaming CDN integrations confirm within seconds of connecting and fill the first reports within about fifteen minutes, and connecting takes about ten minutes on most of them. Kinsta is pulled on a scheduled interval, and historical imports are processed in batches.

What does the free plan include?

One website, 500,000 requests a month, 30 days of history, real-time analytics, bot and AI crawler detection, all 21 site checks and two email alerts. Solo at $19 a month adds the sitemap and Search Console joins, all 16 alerts by email and Slack, log import and six months of history; webhooks and shared dashboards start on Starter at $49. Every plan except Solo has unlimited users, and no card is needed to start.

Can an agency manage several websites?

Yes. Plans include multiple sites with per-site dashboards, shared read-only dashboards for clients, alerts, weekly reports, exports and API access from one account. A Technical SEO Report can be generated for any site on demand, with the client's name on the cover, from the verified crawl data, Page Importance, site checks and findings.

Does Salience change robots.txt or block crawlers?

No. Salience audits robots.txt and reports requests observed on paths your rules disallow for that user agent, but it never edits your configuration and never blocks traffic. Enforcement stays with your CDN, WAF and team.

Trust & data protection

Privacy and data protection

You are the controller

We process only on your instructions. GDPR Art. 28 DPA on every account, nothing to sign.

UK data residency

AWS eu-west-2 (London). Encrypted in transit (TLS 1.2+) and at rest (AES-256).

Server-side collection

No browser tracking script and no client-side pixel.

No sale, no pooling

Your logs are never sold, never used for advertising, never shared between customers. DPA, sub-processor list and security overview available.

Related

Your logs already show what Google and the AI crawlers are doing.

The free plan needs no card and adds nothing to your pages. First requests arrive in about ten minutes on most integrations.

Connect a log sourceExplore the demo