Server log analysis for SEO

Google Search Console tells you what Google chooses to report: aggregated, filtered for privacy and row limits, and a few days behind. Server logs tell you what actually happened at your infrastructure. Every request, every bot, every response code.

A log line typically gives you the requested URL, timestamp, client IP, claimed User-Agent, HTTP status, response size, response time where it is recorded, and referrer where one is sent. That is enough to reconstruct a crawler's entire session.

JavaScript analytics tools answer a different question, and they answer it with a structural blind spot: they only fire when a browser executes JavaScript. They miss crawlers entirely, miss visitors who block scripts, and miss any request that never renders the tracking snippet, including files, feeds, and API endpoints.

Logs do not replace Search Console or analytics; Salience connects Google Search Console and Google Analytics beside the log data. Logs answer a question neither can: what did crawlers and other clients actually request from your site?

The fake bot problem

Not everything that calls itself Googlebot is Googlebot. User-Agent strings are supplied by the client and are trivially spoofed. A meaningful share of traffic claiming to be a legitimate crawler is something else: scrapers, SEO tools rendering your site as Google sees it, vulnerability scanners, or malicious bots borrowing a trusted name to slip past rules that whitelist it.

How much depends heavily on the site. Published studies put it between a few percent and about a fifth of claimed Googlebot requests: Incapsula measured one fake for every 24 genuine Googlebot visits across its network in 2014, and 21% on the customer sites it found targeted in 2012; Centinel measured 14% on the typical news site and 6% pooled across all news traffic between June and September 2026. On small or newly launched sites the ratio can invert entirely, because the real crawler has little reason to visit while automated scanners probe everything. Centinel found the same inversion for AI crawlers: GPTBot claims were 84% fake on the typical news site but 2% pooled.

The point is not the industry average. The point is that you do not know your own number until you verify by IP, and until you do, every Googlebot metric in your reporting is polluted by an unknown amount. Unverified bot data corrupts crawl frequency, response time averages, error rates, crawl budget analysis, and any access-control decision built on top of them. For a quick first look, you can check a list of IPs claiming to be Googlebot against Google's published ranges in the browser.

Crawl budget is the set of URLs a search engine can and wants to fetch from your site in a given period. Google describes it through two forces.

• Crawl capacity limit (formerly crawl rate limit, also called hostload). The total time Google is willing to spend holding connections open to your server, counting both parallel connections and their duration. Every site starts at the same conservative default. The limit rises when response times stay stable or improve, and falls when latency increases or when your server returns 5xx errors or 429 rate-limiting responses.
• Crawl demand. How much Google wants to crawl your URLs, driven by how many URLs it knows about, how often content changes, perceived popularity, duplicate, removed or unimportant URLs, and the quality and relevance of the content relative to other sites.

Two details matter specifically for log analysis:

• Capacity is shared across Google's crawlers. Google's guidance since July 2026 says the crawl capacity limit is shared across all its crawlers, so high demand from one can reduce the capacity available to others. Segment your logs by individual crawler, as the Bot Detection view in Salience does, not by a single Google bucket.
• Budget is per hostname. www.example.com and shop.example.com are separate sites for this purpose. Analyse them separately.

Google scopes its crawl budget guidance to large sites: a million or more pages that change about weekly, 10,000 or more pages that change daily, or a large share of URLs sitting in 'Discovered, currently not indexed'. In practice, for everyone else, apparent crawl problems are usually discoverability, internal linking, accidental blocking, or content quality rather than a hard ceiling.

That said, page count understates what crawling actually consumes. Every HTML page also triggers fetches for JavaScript, CSS, and images, all of which count against the same capacity. Logs are the only place you can see those fetches, which is why smaller sites sometimes hit a wall that page-count heuristics say should not exist.

What to look for in your logs

• Crawl activity over time. Verified crawler requests by hour, day, and week. A sustained decline can mean reduced demand, server or network problems, accidental blocking, a robots.txt change, rising error rates, or large-scale canonicalisation. Compare like periods and annotate releases, migrations, and incidents rather than reacting to daily noise.
• Crawl distribution. Break requests down by directory, page type, hostname, parameter, status code, crawler, and content type. This is where you find crawlers spending their time on filters, internal search results, duplicate paths, and parameter permutations instead of the pages that earn revenue. In Salience the split by directory and by template is the Site Sections and URL Patterns reports.
• Wasted crawls. Requests to non-canonical URLs, parameter variants, internal search, and faceted navigation. Note the common misconception here: noindex does not save a crawl, because Google has to fetch the page to read the directive. Of the directives you control, only a robots.txt disallow stops the fetch, and a disallowed URL can still be indexed without content. The Crawl Budget report groups these fetches by directory.
• Response codes. Not every 404 is a problem, since removed URLs should return 404 or 410. The concern is repeated crawler activity on URLs that should exist, large volumes of broken URLs generated by internal links or malformed parameters, and any sustained 5xx or 429 rate, both of which directly reduce your capacity limit. The Status Codes, Redirects and Soft 404s reports cover these in Salience.
• Response size. Googlebot fetches the first 2MB of an HTML or other supported file (64MB for a PDF), measured uncompressed, and sends only the downloaded part for indexing. Each referenced CSS or JavaScript file is bound by the same 2MB limit. Very few sites are affected, and the ones that are will never know without checking the bytes column.
• Crawl depth. Are crawlers reaching your important pages, or circling low-value areas? Measured from real fetches rather than theoretical click depth. Page Importance scores each URL from its verified recrawl rate.
• Response time percentiles. A site-wide average hides the problem. Product pages may be fast while category, search, and client-rendered pages are slow. Report median and upper percentiles, segmented by template and status code.

User-Agent classification is a reasonable first pass and worthless as proof. Any HTTP client can send Googlebot's User-Agent string, and your server will log it faithfully. Google says as much in its own documentation on verifying Googlebot and tells you to verify by matching the IP against its published lists, or by a reverse DNS lookup confirmed with a forward lookup.

Verification is best understood as a ladder of evidence, strongest first.

The Salience crawler directory lists every named crawler seen across monitored sites, with its user agent string, its robots.txt token and a recommended policy. Report verification as a spectrum, not a boolean. Salience labels each crawler with one of six states, described in the bot verification section of the help centre:

• Verified, with the tier it was verified at: IP ranges, reverse DNS, network owner, or CDN edge
• Consistent network: the address sits in the cloud provider this operator uses, but not in a published range
• Verifying: the reverse DNS check is queued and the verdict follows within a day
• Proxied: the request arrived through a CDN or proxy, so the origin address cannot be checked
• Not verifiable: the operator publishes no verification data
• Unverified: the claim failed every check, so treat it as possible impersonation

Those six states mean six different things. Only verified requests belong in authoritative crawl reporting. Verifying, proxied and not-verifiable claims should never be counted as real crawl activity, and a claim that failed verification is a security finding, not a crawl statistic.

Why verification matters

Without it you cannot:

• Measure true crawl activity. Your Googlebot crawl rate is inflated by an unknown number of imposters.
• Identify impersonators. Scrapers wearing Googlebot's name walk straight through rules that whitelist it. The security and threat detection guide covers how to block on verification instead.
• Apply access controls safely. Block by User-Agent and you may block the real crawler. Allow by User-Agent and you have allowed everyone.
• Compare crawlers meaningfully. Crawl rate, response time, and error rate comparisons across operators are worthless if identity is unreliable.

Logs tell you what search engines fetched. A crawl of your own site, from Salience's own site crawler or an uploaded crawl export, tells you what exists and how it is linked. Joined, they answer questions neither can alone:

• Crawl gaps. Pages your crawler finds and that are linked internally, but that verified search crawlers have never requested.
• Orphans. Pages crawlers fetch, or sitemaps list, that nothing in your internal link graph points to. Usually old URLs surviving on external links or a stale sitemap.
• Effective crawl depth. The real relationship between a page's internal depth and the crawler attention it receives.
• Redirect destinations. Where chains actually terminate from the bot's point of view, including loops and endpoints that differ from your intended canonical. Logs record only the redirect itself; the crawl supplies the destination.
• Crawl-attention concentration. Which directories and templates absorb the largest share of verified crawler requests, which you can then compare with where you want that attention spent.

This join is what separates a crawling problem from an architecture problem, and it is the distinction that determines whether you fix your server or your internal links. In Salience it is the Crawl Join report, with the Link Map for the internal link graph.

Your XML sitemap tells search engines which canonical URLs you consider important. It is a signal, not an instruction, and inclusion guarantees nothing. Your logs tell you which of those URLs crawlers actually requested. The gap between the two is where the work is.

Cross-reference sitemap URLs against verified Googlebot requests and every URL lands in one of three buckets, which is how the Sitemap Coverage report groups them.

Thirty days is a reasonable window for ecommerce and publishing, and it is the fixed window Salience uses. Slower-moving reference content can be crawled far less often without anything being wrong, so read the stale bucket against how often the content changes rather than against a round number.

Investigating never-observed URLs

Before you do anything, check whether the URL returns 200, is allowed by robots.txt, is internally linked, carries a self-referencing canonical, does not redirect, carries no noindex, sits in the right sitemap, is not duplicated elsewhere, and has been live longer than your log retention. Most never-crawled pages fail one of those tests, and the fix is the failure, not a submission to Search Console. Do not bulk-submit URLs you have not established are worth indexing.

Logs tell you whether a crawler requested a page. Search Console tells you how Google treated it, and connecting Search Console to Salience puts the two side by side. Together they give you a prioritisation matrix.

Two cautions that make this matrix trustworthy rather than merely neat:

• Index status must come from URL Inspection or the Page Indexing report, not from the absence of impressions. A page with no clicks is not necessarily unindexed; it may simply rank for nothing. The URL Inspection API allows 2,000 inspections per property per day, so URL-level index status at scale has to be sampled or scheduled.
• 'Not crawled' is bounded by your log retention. Say so in your reporting.

The not crawled and not indexed quadrant is usually where the largest wins sit: pages you built and submitted that Google has never discovered or has abandoned. Fixing their discoverability is the fastest route to getting them crawled, and crawling is the precondition for indexing the content. What happens after that is decided by content and links, so treat discovery fixes as unlocking the opportunity rather than guaranteeing the ranking.

Go to page level

Site-level Search Console totals hide the movement worth seeing. Pull daily impressions, clicks, CTR, and average position per URL, plus the queries behind each page, and you can place a crawl event beside a performance change for the same URL on the same day:

• Did performance change after a recrawl?
• Is a heavily crawled page earning no impressions at all?
• Is a high-performing page being refreshed less and less often?
• Did Google recrawl after the content update, or is it still serving the old version?
• Did a ranking decline coincide with an error or a redirect?

Coincidence is not causation, but the timeline tells you which five pages to investigate out of fifty thousand. In Salience the URL detail view shows a page's crawl history and its Search Console history together. Note that Search Console omits anonymised low-volume queries and truncates long reports, so treat page-and-query data as a strong sample rather than a complete record.

Search engines are no longer the only discovery channel visible in your logs, and AI User-Agents are not one category. Different bots from the same operator do different jobs: training-data collection, search index building, user-initiated retrieval at the moment someone asks a question, link previews, and automated browsing. Lumping them together produces a number that cannot be acted on. Identify operator, crawler, and likely purpose, then verify where verification data exists. The bot management and AI crawlers guide lists the main crawlers by operator and purpose, and the AI crawler impact report measures their share of a log file you upload.

The second half of the picture is human. On the web, ChatGPT tags many of the links it cites with utm_source=chatgpt.com, and Perplexity, Claude, Gemini and Copilot pass a referrer; the mobile apps and some ChatGPT surfaces strip it. Google added an AI Assistant channel to GA4 on 13 May 2026 for the referrers it recognises. Those landing pages, and what visitors do next, are SEO data now. Report landing page, assistant and requests, and, with Google Analytics connected, sessions, engagement, key events and revenue. Because referrers are not always passed, treat measured assistant traffic as an observable subset rather than a total.

Put together, the three questions are distinct:

• Crawler activity: which pages are AI systems fetching?
• Referral activity: which pages are assistants actually sending people to? Salience reports crawler activity under LLM Crawlers and referrals under AI referral traffic.
• Outcome: what do those visitors do when they arrive?

The gap between what the answer engines fetch and what they cite is usually a content problem, not a crawl problem, and you can only see it by holding the log data and the referral data side by side. Neither a fetch nor a referral proves that an answer cited the page, so read the two counts as related evidence rather than a linked record.

• Verified crawler volume. Googlebot and Bingbot request counts against the previous period. Investigate sustained changes, ignore single-day noise.
• Crawler response time. Median and upper percentiles, segmented by template and status code. Target under 500ms.
• Crawl error rate. Share of verified crawler requests returning 4xx, 410, 429, and 5xx. 5xx and 429 directly reduce crawl capacity, so treat them as urgent.
• Sitemap coverage ratio. Share of sitemap URLs fetched inside your window, read against content freshness and log retention.
• Crawl distribution drift. Is attention moving towards parameters, filters, internal search, redirects, or duplicate paths?
• Share of crawl by individual Google crawler. Because capacity is shared, watch for one crawler crowding out the main index crawl.
• Unverified bot share. Percentage of Googlebot and Bingbot claims failing verification. A spike usually means a new scraper has arrived, though CDN edge addresses can also produce unverified claims, so check the proxied count first. Salience's bot impersonation alert fires on a surge.
• New or changed User-Agents. Treat anything new as unverified until classified.
• Annotations. Log deployments, migrations, CDN changes, robots.txt edits, and outages, so next week's anomaly has context.

Mature log analysis joins five datasets:

1. 01Edge and origin logs, from whichever log source you connect: what was requested.
2. 02Verification data: who probably made the request.
3. 03A crawl of your own site: what exists and how it is linked.
4. 04Sitemaps: which canonical URLs you want discovered.
5. 05Search Console: what is indexed and what is earning visibility.

Together they answer what no single tool can: which important pages are never crawled, which crawled pages nothing links to, which sections absorb disproportionate attention, which pages are fetched but not indexed, which are indexed but never refreshed, and which crawl events line up with performance changes.

The objective is not more crawling. It is getting legitimate crawlers to spend a larger share of a finite budget on pages that are accessible, canonical, useful, and commercially important.