Guide

Server logs vs prompt tracking: what each tells you about AI search

AI crawler activity and AI search visibility are often discussed as though they measure the same thing. They don't.

Last updated

An AI crawler can request thousands of your pages without any of them appearing in an AI-generated answer. Equally, an AI assistant might mention your brand without making a new request to your website.

Server log analysis and prompt tracking help explain different parts of this behaviour.

Server logs tell you which AI systems request your website, which URLs they access and how your infrastructure responds. Prompt tracking tells you what a sample of AI-generated answers says about your brand, products and competitors.

There is also a third measurement: the people who click through from AI assistants to your website.

Understanding the difference matters because each requires different evidence, and none automatically proves the others.

Three different signals: access, visibility and referrals

SignalWhat it measuresWhere the evidence comes from
Crawler accessWhich bots and agents request your pages, how frequently and the HTTP responses they receiveServer and CDN logs
AI answer visibilityWhether selected AI-generated answers mention your brand, recommend your products or cite your contentPrompt tracking
AI referral trafficIdentifiable visits or requests arriving from AI assistantsWeb analytics, server logs and attribution parameters

A crawler request is not an AI citation. A citation is not a referral visit. And a referral does not necessarily tell you which crawler or source contributed to the answer.

These distinctions are important when investigating why AI systems appear to be ignoring your content.

What server logs measure

When a crawler requests a URL, your server or CDN can record the request.

Depending on the infrastructure and log format, the record may contain the requested URL, timestamp, source IP address, user-agent string, HTTP status code, response size, referrer and URL query parameters.

Analysing these records can answer questions such as:

  • Which AI crawlers are requesting our website?
  • Are they accessing product pages, articles, documentation or other sections?
  • How frequently are they returning?
  • Are important URLs responding with 200, redirecting, returning 404, or being blocked with 403 or 429?
  • Did crawl behaviour change after a deployment, migration or firewall update?
  • Are requests claiming to be from legitimate crawlers actually coming from those operators?

Unlike browser-based analytics, log analysis can observe crawler requests that never execute JavaScript or trigger an analytics tag.

However, coverage depends on where the records are collected.

Origin logs and CDN logs are not necessarily the same dataset. If a CDN or Web Application Firewall (WAF) blocks a request before it reaches your origin server, that request may never appear in your origin logs. Even edge collection can miss traffic intercepted before its collection point, and some logging services provide sampled rather than complete records.

This matters when investigating missing crawlers. No recorded activity does not always mean no attempted access.

Not all AI crawlers do the same job

One of the most important distinctions in AI log analysis is the purpose of the crawler.

Major AI operators use different agents for model development, search discovery and requests initiated by users.

PurposeExamplesWhat the activity indicates
Model development and trainingGPTBot, ClaudeBotContent is being collected for potential model development or training
AI search crawlingOAI-SearchBot, Claude-SearchBot, PerplexityBotContent is being requested to support AI search discovery or retrieval
User-triggered retrievalChatGPT-User, Claude-User, Perplexity-UserAn AI system is fetching content in response to a user-initiated action

The distinctions are documented by OpenAI, Anthropic and Perplexity.

For example, GPTBot requesting a product page does not establish that ChatGPT is preparing to recommend that product. It is a crawler associated with potential model training.

A verified ChatGPT-User request provides different evidence: a page was fetched as part of supported user-initiated activity. Even then, the log cannot reveal the exact prompt, whether the page contributed to the answer or whether it was ultimately cited.

Crawler purpose matters when deciding which bots to allow, which to restrict and how to interpret changing request volumes.

How do you know a crawler is genuine?

A user-agent string is a claim, not proof of identity.

Anyone can send a request identifying themselves as Googlebot or GPTBot.

Crawler verification can use published provider IP ranges, reverse DNS followed by forward DNS verification, or cryptographically signed requests where supported.

Google documents IP-range and DNS verification. Emerging approaches such as Web Bot Auth use cryptographic signatures.

Verification is not equally available for every operator. It also does not prove what an operator subsequently does with retrieved content.

A verified crawler identity establishes the source of a request using the available evidence, not that the requested page was indexed, cited or used in training.

Why an HTTP 200 does not mean an AI crawler read your content

A successful HTTP response does not guarantee that a crawler extracted the information you intended it to see.

Consider a JavaScript-heavy product page.

A crawler requests /products/example and receives 200 OK. Your server log records a successful request.

But if the initial HTML contains little more than an application shell, the product name, description, price and specifications may only appear after client-side JavaScript executes.

A crawler that does not execute that JavaScript may receive the page successfully without obtaining its meaningful content.

Research published by Vercel found that several major AI crawlers did not execute JavaScript in their tests. Crawler capabilities can change, so those findings should not be treated as a permanent rule for every agent.

Googlebot has its own rendering process, but that does not mean other AI systems behave identically.

The practical check is to inspect the initial HTML delivered to the crawler, not just the HTTP status code.

Standard access logs generally record request and response metadata, not the complete HTML response body.

What prompt tracking measures

Prompt tracking approaches AI search from the opposite direction.

Rather than recording requests to your website, it submits a selected set of questions to AI products and records their responses.

Depending on the tool, results may show:

  • Whether your brand appears in an answer
  • Which competitors are mentioned or recommended
  • Which websites and pages are cited
  • How your products, services or pricing are described
  • Whether answers contain outdated or inaccurate information
  • How these observations change across repeated measurements

This makes prompt tracking useful for examining brand visibility, competitive positioning and the sources AI assistants use to support their answers.

But it is important to understand what the results represent.

Prompt tracking is a sample, not a complete record of AI searches

Most prompt trackers monitor a selected set of questions. Those questions may be commercially relevant, but their frequency in a tracking tool does not establish how often real users ask them.

Answers can also differ between runs because of changes in models, retrieval behaviour, location, personalisation and other conditions.

A brand mentioned in six out of ten tracked responses has appeared in 60% of that particular sample. It does not necessarily have 60% visibility across real-world conversations on the topic.

Some vendors provide additional demand estimates or user-conversation datasets. These should be evaluated separately from results produced using scheduled prompts.

API results and consumer AI answers can differ

Another important distinction is how vendors collect responses.

Some use model APIs. Others capture answers from consumer-facing interfaces, such as the ChatGPT website or app.

These are not necessarily equivalent.

Differences in model configuration, search tools, retrieval systems and interface behaviour can change the brands mentioned and sources cited.

When evaluating a prompt tracker, ask whether its responses come from an API or the consumer interface, which model and region are used, and whether repeated runs are performed under consistent conditions.

Neither method reproduces every personalised answer an individual user might see.

Which questions should you answer with each tool?

Your questionBest evidence
Which AI crawlers request our website?Server logs
Is a crawler fetching content for training, search or user-triggered retrieval?Server logs, agent classification and verification
Is our WAF rejecting legitimate AI crawlers?CDN or WAF logs
Did AI crawler activity change after a deployment?Server logs
Is robots.txt preventing a crawler from accessing certain pages?robots.txt rules combined with log analysis
Did a crawler successfully access the important content on a page?Logs plus an HTML or rendering test
Does ChatGPT mention our brand for selected questions?Prompt tracking
Which competitors appear alongside us in AI answers?Prompt tracking
Which third-party websites are cited for our products or topics?Prompt tracking
Is an AI assistant describing outdated product information?Prompt tracking, supported by crawl and content checks
Which pages receive identifiable traffic from AI assistants?Referral analytics or appropriately filtered server logs
Why wasn't a particular page cited?Neither proves the answer alone; use both to investigate

The distinction is straightforward: logs help diagnose access and infrastructure behaviour, while prompt tracking examines sampled answer visibility.

Some questions require evidence from both, plus additional technical checks.

How server logs and prompt tracking work together

Example 1: Your brand has disappeared from AI answers

Suppose a prompt tracker shows that your brand has stopped appearing in answers about a particular product category.

Rather than immediately rewriting content, investigate the relevant access data.

First, identify the pages that should support those answers. Then check whether the relevant AI search crawler has requested them recently.

If requests are returning 403, 429 or 5xx responses, there may be an infrastructure problem preventing access.

If the crawler receives 200, inspect the initial HTML to confirm that the important information is actually present and accessible.

If crawling and content delivery appear healthy, review the competing sources cited in the tracked answers.

The issue may be related to content relevance, third-party information, retrieval selection or how the AI system constructs its answer.

The logs narrow down the possible causes. They do not prove why a model chose another source.

Example 2: AI crawler activity falls after a robots.txt change

There is an important difference between a firewall block and a robots.txt exclusion.

A firewall actively rejects or challenges a request. If the relevant edge logs capture the event, you can usually see that response.

A compliant crawler following robots.txt behaves differently. After reading a disallow rule, it may simply stop requesting the affected URLs.

That means you will not necessarily see a series of 403 responses. You may see no requests at all.

To investigate, compare the published robots.txt rules, access to /robots.txt, crawl activity before and after the change, and the relevant crawler's documented behaviour.

User-triggered fetchers introduce further differences. OpenAI says robots.txt rules may not apply to ChatGPT-User requests, while Perplexity says Perplexity-User generally ignores them. Anthropic documents robots.txt controls for its own bots, including Claude-User.

Changes can also take time to propagate.

Another complication is Google-Extended. This is a policy-control token in robots.txt, not a distinct crawler that appears as Google-Extended in your HTTP logs. Google continues to use existing crawler identities, including Googlebot.

Consequently, the absence of a Google-Extended user agent tells you nothing about whether that policy is in effect.

Example 3: AI crawlers fetch thousands of pages, but referrals remain low

A high volume of crawler requests does not imply high AI search visibility or human traffic.

One operator might repeatedly request thousands of pages for model development while another fetches a smaller number for search discovery.

Neither number establishes how frequently a brand appears in answers.

Prompt tracking can investigate the answer side. Referral analysis can separately identify the users who arrive through links.

Referrals also need careful interpretation.

An HTTP Referer header is not always passed, particularly when traffic comes from applications or privacy-restricted environments.

Some AI services include attribution parameters in outbound URLs. For example, OpenAI documents the use of utm_source=chatgpt.com in ChatGPT search referrals.

Analysing URL parameters alongside available referrer information can recover some otherwise unidentified traffic.

Raw logs must also distinguish primary page-navigation requests from assets, prefetches and other HTTP requests. Otherwise, one human visit may be counted more than once.

Logged requests are not the same as unique visitors or sessions. Use analytics when you need engagement, conversions and visitor-level reporting.

What neither tool can prove

Even when used together, these tools cannot fully reconstruct how an AI system arrived at an answer.

A page may be cited because it was already present in a search index, because a previously retrieved copy was available, or because information was obtained from another source.

An answer may also reflect knowledge acquired during earlier model development, without a fresh visit to the original website.

Seeing a crawler fetch a page does not prove that the page influenced an answer. Failing to observe a fetch during your reporting period does not prove the page was unavailable to the AI system.

This is why AI crawler requests, answer citations and human referrals should be reported as separate metrics rather than combined into one visibility score.

Choosing between server log analysis and prompt tracking tools

The software categories increasingly overlap.

Platforms such as Profound offer both AI visibility measurement and agent analytics. Established log-analysis platforms such as Oncrawl also classify AI crawlers, while Cloudflare offers AI crawler monitoring and controls at its own network edge.

A single platform may cover several requirements. The important question is whether it collects the underlying evidence you need.

When comparing tools, check:

  • Log coverage: Does it use origin logs, edge logs or both? Are records sampled, and can it see requests blocked by the WAF?
  • Crawler verification: Does it distinguish claimed identities from verified ones, and explain the evidence used?
  • Crawler purpose: Can it separate model-training crawls, AI search crawls and user-triggered requests?
  • Prompt methodology: Does it use APIs or consumer interfaces? How are prompts, regions, models and repeated runs controlled?
  • Referral measurement: Does it use referrers, URL parameters or analytics integrations? Are requests, visits and sessions clearly distinguished?

The presence of a crawler analytics dashboard does not automatically mean it provides complete first-party logs. Equally, a prompt tracking product with crawler monitoring does not necessarily provide the same request-level diagnostics as a dedicated log-analysis tool.

Where Salience fits

Salience focuses on the first-party access side of this measurement.

It analyses server and CDN request data to show which search engines, AI crawlers and agents request your website, which pages they access, when they return and what HTTP responses they receive.

Salience classifies recognised AI systems by their documented purpose, including model training, AI search crawling and user-triggered retrieval. For supported identities, it checks crawler claims against available provider verification data.

It also helps identify crawl errors, blocked requests, unusual activity and changes following website or infrastructure updates.

Alongside crawler requests, Salience reports identifiable AI referral activity from available source signals, keeping machine requests separate from human traffic.

Salience does not submit prompts to AI assistants or measure which brands appear in their generated answers. It is not a prompt tracking tool.

If you already use prompt tracking, Salience adds evidence about whether the relevant AI systems can actually access your website and how your infrastructure responds to them.

If your main concern is crawler access, verification, blocked URLs or changes in bot behaviour, start with logs. If you need to understand brand mentions, citations and competitors in AI answers, start with prompt tracking.

For many search teams, both are useful because they answer different questions.

Explore your own crawler data

You can check your website with Salience using the free plan, or explore the live demo before connecting a log source.

If you already have an access log, the free AI Crawler Impact Report provides a browser-based snapshot of crawler activity, including requested URLs and bandwidth usage. It groups requests by their claimed crawler identities; it is not a substitute for continuous bot verification.

For more detail, explore AI crawler and agent intelligence or agent analytics.

Your logs already show what Google and the AI crawlers are doing.

Free plan, no card, about ten minutes to your first request.