Search Console and server log monitoring
See what Googlebot fetched next to what Google indexed.
Search Console reports Google's index status, queries and clicks. Salience puts them beside every Googlebot request in your server logs, URL by URL, with the response your server returned.
- Read-only Google access
- No code or DNS changes
- Synced daily
Log Explorer
www.example · requests in real time
Google-InspectionToolVerified/guides/api-keys200- GooglebotVerified/guides/rate-limits200
OAI-SearchBotVerifiedAI search/blog/edge-caching-explained200- GooglebotUnverified/search?q=pricing200
- GooglebotVerified/blog/tag/releases?page=14200
BingbotVerified/docs/sdk-go301
Verified Googlebot requests to /blog/ are well below the usual rate for this hour of the week.
Recommended: check robots.txt and the latest release for changes affecting /blog/.
Search Console and your server logs answer different questions
Search Console reports what Google decided about your pages and how they perform in search, from Google's side. Your server logs record each request Googlebot made and the response it received. Salience shows the two together for each URL.
Google Search Console
Google's own records of crawling, indexing and search results
- Which pages has Google indexed, and why were others left out?
- Which queries show your pages, with what clicks and position?
- How many crawl requests did Google make, by response and file type?
- What does a live test of one URL return?
Index status, canonical selection and search performance come from Google. Salience reads them through the official API.
Salience
Every request from your log source, joined to Search Console through its API
- Which URLs did verified Googlebot fetch, when, and what status code did each get?
- Which pages does Googlebot fetch that Search Console reports as not indexed?
- Which sitemap URLs has Googlebot not requested in 30 days?
- Which requests using Google's name came from Google's addresses?
Salience has read-only access and cannot change a setting or submit URLs.
Questions Search Console can't answer on its own
Four examples from the four teams that use Salience with Search Console. The site, addresses and figures in the panels are examples.
01Search and SEO
Is Google indexing the pages Googlebot spends its time on?
The page indexing report lists pages as crawled or discovered but not indexed. It does not show what your server returned on each fetch, or how much of Googlebot's crawling went to tag pages and internal search results.
Salience crosses each URL's index status from the URL Inspection API with verified Googlebot fetches from your logs, in four groups: indexed and fetched, indexed and not fetched, fetched and not indexed, and fetched but never inspected. Requests are also grouped by section and compared with your sitemap.
Find pages Googlebot fetches that Google does not index, sitemap pages Googlebot has not requested, and sections where crawling goes to URLs outside the sitemap.
Verified Googlebot by section
www.example · last 7 days
| Section | Requests | URLs | Unfetched |
|---|---|---|---|
| Guides/guides/* | 3,406 | URLs fetched412 | Sitemap URLs not fetched18 |
| Blog posts/blog/* | 2,431 | URLs fetched688 | Sitemap URLs not fetched0 |
| Tag pages/blog/tag/* | 2,207 | URLs fetched951 | Sitemap URLs not fetchedNot in sitemap |
| Internal search/search?q= | 1,074 | URLs fetched1,020 | Sitemap URLs not fetchedNot in sitemap |
Sources requesting the site
www.example · last 24 hours
Requests per hour from unverified automation
peak 2,356
Unverified automated requests ran at about 100 an hour, rose to between 2,214 and 2,356 an hour from 13:00 to 19:59, then returned to about 100.
- 203.0.113.0/24Not a crawler14,280
Hosting provider · 5 rotating browser user agents · Every /guides/* and /blog/* URL in order, HTML only
- 198.51.100.61Unverified2,416
Hosting provider · Googlebot (claimed) · /search?q= with sequential terms
- GooglebotVerified1,874
Google · Googlebot · Guides and blog posts, follows robots.txt
- 192.0.2.140Not a crawler968
Hosting provider · HeadlessChrome/128 · /blog/tag/* pages 1 to 40
02Security and bot threats
Is the traffic using Google's name from Google?
Scrapers borrow Googlebot's user agent because sites rarely refuse it. Search Console's Crawl Stats report counts Google's own requests, so an impersonator appears in your server logs and nowhere in Search Console.
Salience checks every request claiming a Google identity against Google's published address ranges, groups the rest by address, network and user agent, and shows each source's paths, request rate and timing.
Block the sources at your CDN or server. Recommendations list the addresses and fake crawler user agents and generate a ready-to-paste rule for Cloudflare, CloudFront, Vercel, Netlify, Nginx or Apache.
03AI search and discovery
Which AI systems fetch the pages that rank in Google?
Search Console reports Google Search only. AI companies run separate crawlers for collecting training data, for building an AI search index and for fetching a page when someone asks an assistant a question, and their requests appear only in your server logs.
Salience puts every AI crawler in one of those three classes, verifies it where the operator publishes its addresses, and lists the pages each one requested. With Search Console connected, it lists pages earning Google impressions that no AI search or user-triggered fetcher requested in the period.
A recorded request shows that a page was fetched. It does not show whether an answer used or cited the page.
The AI referral traffic report counts the human visits that arrive from ChatGPT, Perplexity, Claude, Gemini, Copilot and other assistants, by landing page.
AI crawlers by purpose
www.example · last 7 days · verified requests
- GPTBot1,742
ClaudeBot1,119
Most requested /guides/authentication
- OAI-SearchBot534
PerplexityBot387
Most requested /blog/edge-caching-explained
- ChatGPT-User206
- Perplexity-User58
Most requested /guides/rate-limits
04Engineering and operations
What did your last release change for Googlebot?
A URL move, a redirect rule or a CMS release changes what Googlebot receives on the day it ships. Search Console's performance data runs two to three days behind, and a page's index status changes only after Google recrawls it.
Salience tracks status codes per path and per type of client against your own history. Site Events mark your releases on every chart, and alerts fire when 404s, redirects or errors to crawlers rise above your baseline.
See the change on the day it ships, then follow the same URLs in Google Index as Search Console's inspections refresh.
404 and 3xx responses by client
www.example · daily share · 14 days
- 404 to verified Googlebot
- 0.8% 21.6%
- 3xx to verified Googlebot
- 3% 34.2%
- 404 to human visitors
- 0.4% 1.9%
- 404 to verified Googlebot
- 3xx to verified Googlebot
- 404 to human visitors
- Site event · /docs/ moved to /guides/
Old /docs/ URLs with no redirect to /guides/ are returning 404 to Googlebot and to visitors.
Recommended: add redirects for the /docs/ paths listed in the alert, then confirm Googlebot receives 301 followed by 200.
Example investigation
Googlebot received 503s on 14 guide pages during a release
An example of how a crawl alert and Search Console data fit together, step by step. The site, URLs and figures are illustrative.
- 1
The alert fires
At 02:35 a Crawler Error Spike alert reports that verified Googlebot requests to /guides/ are returning 503 far above the site's usual rate.
- 2
The crawler is Google's
Every request came from an address inside Google's published Googlebot ranges, so these are Google's own fetches receiving an error.
- 3
The affected URLs
87 requests across 14 guide URLs. All 14 are in the sitemap, and Search Console reported each one as indexed at its last inspection.
- 4
When it started
The first 503 arrived at 02:07, 7 minutes after the docs release your team recorded as a Site Event at 02:00. There were none in the 07:00 hour.
- 5
The next step
Fix the release and confirm Googlebot receives 200 on its next fetches. Index status for the 14 URLs updates as Salience re-inspects them, about every 14 days, and their impressions arrive two to three days behind.
Crawler Error Spike
www.example · opened 02:35 · Googlebot · 503
GooglebotVerified66.249.66.0/24 · in Google's published ranges- /guides/webhooks50312
- /guides/pagination50310
- /guides/error-codes5039
- /guides/authentication5038
- /guides/idempotency-keys5037
- /guides/v2-migration5037
- 8 more URLs · 34 requests
02:00 Site event · docs release
No 503s in the 01:00 hour before the release at 02:00; then 9, 21, 24, 19 and 14 per hour from 02:00 to 06:59, 87 in total, and none in the 07:00 hour.
Recommended next step
Fix the release and confirm Googlebot receives 200 on the 14 URLs, then follow them in Google Index as their inspections refresh.
What Salience shows with Search Console connected
Search Console data sits beside the request data from your log source, URL by URL. Google Analytics 4 adds sessions to the same URLs if you connect it.
URL detail
www.example/guides/sso-setup · last 28 days
/guides/sso-setup
- LogsVerified Googlebot fetchesLast 200 at 06:4111
- Google inspectionIndex statusGoogle canonical is this URLIndexed
- Search ConsoleImpressions, 28 daysTop query: saml sso setup4,260
- Search ConsoleClicks, 28 daysAverage position 6.4212
- LogsAI page fetchesChatGPT-User, Perplexity-User5
Crawling and indexing
Google Index
Every URL in one of four groups: crawled and indexed, crawled and not indexed, indexed but not recently crawled, and neither, with a directory breakdown.
Index and fetch reconciliation
Search Console index status crossed with verified Googlebot fetches in the period, including URLs fetched but never inspected.
Sitemap coverage
Sitemap URLs crawled in the last 30 days, stale or never crawled, and crawled URLs missing from the sitemap.
Search performance
Impressions, clicks and position
Totals against the previous period, a daily chart, and the top 200 queries over the trailing 28 days.
URL detail
One page per path with its requests, sitemap membership, Google's stored inspection and the page's top queries.
Pages ranking that AI has not fetched
Pages above an impression threshold with no recorded AI search or user-triggered fetch in the period.
Google's crawlers
Verified Googlebot
Every request claiming a Google identity checked against Google's published address ranges.
Inspection Tool fetches
Requests from Google-InspectionTool when someone uses Test live URL or Request indexing, from your logs alone.
Page Importance
A 0 to 10 score per page from how often verified Googlebot or Bingbot returns to it over 90 days.
Alerts go by email, Slack or webhook. The written SEO report includes an indexation section, and the public API, CLI and MCP server return the same search data.
How to set up Search Console with server log monitoring
Connect a log source first: it supplies the requests every report needs. Search Console then takes a Google sign-in and a property pick, and the first sync starts on its own.
- 1
Enter your domain and create your account
Start the 14-day trial. Search Console is on paid plans from $19 a month.
- 2
Connect a log source
Choose your platform: Cloudflare, Vercel, CloudFront, Kinsta, Shopify, Netlify, Apache or Nginx. The connection check confirms requests are arriving.
- 3
Connect Search Console
In setup or Website Settings → Integrations, sign in with Google, grant read-only access and choose the domain or URL-prefix property. The first sync starts automatically.
What you need
- A verified Search Console property, domain or URL-prefix
- A Google account with at least Read permission on it
- A connected log source for the same site
What stays the same
- No code, tag or DNS changes
- Search Console settings and URL submissions stay with you
- Log collection carries on unchanged
How the data refreshes
- A daily sync at 05:00 UTC, with Sync Now in Settings → Integrations
- Search analytics run two to three days behind Google
- Each sync stores the trailing 28 days, and history builds up across syncs
- Up to 2,000 URL inspections a day, highest impressions first, each re-inspected after 14 days
Someone else manages your hosting or DNS? Send them a delegated setup link from the app for the log source step. It lasts 30 days and is limited to this one site.
How your Search Console data is read and controlled
What is read
- Index status per URL, with Google's reason
- Google's chosen canonical for each URL
- Impressions, clicks and average position per URL
- Top queries for high-impression URLs
What is not read or changed
- Search Console settings
- URL submissions and indexing requests
- Queries Google withholds as anonymised
- Bing or other engines' consoles
Where it is processed
- AWS eu-west-2 (London), with encrypted S3, DynamoDB and Aurora storage
- A GDPR Article 28 DPA applies to every account
- Never sold, used for advertising or shared between customers
Who controls it
- Disconnect in Settings → Integrations at any time
- Revoking Salience's access in your Google account also stops the sync
- Cached Search Console data expires after you disconnect
Search Console API quotas
Search Console quotas are per property. A daily sync makes about 206 Search Analytics requests, and URL inspections use Google's separate quota of 2,000 a day. The last sync's request count is shown in the Search Performance header.
Google Search Console questions
Does Salience replace Search Console?
No. Search Console is Google's own report of what it indexed and how pages perform, and Salience reads it through the official API. Salience adds the requests Googlebot made to your server and the responses it received, and shows the two together for each URL.
What does Salience read from Search Console, and can it change anything?
Index status per URL with Google's reason, Google's chosen canonical, impressions, clicks and average position per URL, and the top queries, all with read-only access. Salience cannot change a setting, submit URLs or request indexing.
Do I need a log source connected as well?
Yes. Search Console supplies Google's side and a log source supplies your server's side, and every report starts from the logs. Cloudflare, Vercel, CloudFront, Netlify, Kinsta, Shopify, Apache and Nginx all work.
How often does the data refresh?
The sync runs daily at 05:00 UTC, and Sync Now in Settings → Integrations runs it on demand. Google's search analytics run two to three days behind. Each sync stores the trailing 28 days, and history builds up across syncs. Up to 2,000 URLs are inspected per daily run, highest impressions first, and each is re-inspected after 14 days; a URL that has not been inspected yet shows as pending until its first inspection.
Why do Salience and Search Console show different numbers?
They count different things. Logs count the requests your server answered, as they happen. Search Console reports Google's view, two to three days behind, with some queries withheld as anonymised, and its Crawl Stats report covers Google's crawlers only.
Do I need Search Console connected to see Google's inspection fetches?
No. When someone uses Test live URL or Request indexing, Google fetches the page as Google-InspectionTool, and that request arrives in your logs like any other. The Inspection Tool report works from your log source alone, with each request marked verified or unverified against Google's published ranges.
Which plans include Search Console?
Sitemap and Search Console are on every paid plan, from Solo at $19 a month, each with a 14-day trial. The Free plan covers one website, 500,000 requests a month and 30 days of history from your log source. Search Console itself is free from Google.
Start your 14-day trial and connect Search Console.
Search Console is on paid plans from $19 a month. Connect a log source first, then sign in with Google and choose your property.