A
Agent analytics
Measuring the requests that AI agents and crawlers make to your website: which systems came, which URLs they asked for, when they returned and what your site sent back. It is built from your own server or CDN records, so it shows what reached your site. It does not show what an AI answer said afterwards.
Learn more: Agent analytics
AI crawler
Any bot an AI company runs to fetch web pages. One company often runs several, each with a different job, so what matters is what a crawler is doing on your site, not whether it counts as AI. Salience groups them as training, AI search or user-triggered, using the roles their operators describe.
Learn more: Crawler directory and AI crawler intelligence
AI referral traffic
Human requests that arrive from an AI assistant such as ChatGPT, Perplexity or Claude. Salience counts a human request as AI-referred when the referrer is an assistant's own address, or when the link carries a source tag such as utm_source=chatgpt.com. Counts are logged requests, not unique visitors. Some apps strip the referrer, so the figure is a subset of the real number, not a total.
Learn more: AI referral traffic in the help centre
AI search crawler
Collects pages so an AI product can search and retrieve from them. OAI-SearchBot, Claude-SearchBot and PerplexityBot are examples. A request from one shows the page was read for that purpose. It does not show the page was indexed or cited.
Learn more: LLM crawlers in the help centre
C
CDN log
Your content delivery network's own record of the requests it handles at the edge, for example on Cloudflare, CloudFront or Vercel. It includes requests the CDN answered or blocked without passing them to your own server, which a server log never sees. On supported CDN integrations the logs arrive as a stream. Cloudflare and Vercel show within seconds. CloudFront real-time logs take seconds to minutes and follow the sampling rate you set in AWS.
Learn more: Integrations
ChatGPT-User
The name OpenAI's agent uses when it visits a page because someone asked ChatGPT a question. OpenAI says robots.txt rules may not apply to these user-initiated requests, so Salience lists them apart from crawler robots.txt violations.
Learn more: Robots.txt in the help centre
ClaudeBot
Anthropic's crawler. Anthropic describes it as collecting web content that could contribute to training its models. The company also runs Claude-SearchBot for search and Claude-User for fetches made when someone asks Claude a question. Salience checks claimed ClaudeBot requests against Anthropic's published address ranges.
Learn more: Crawler directory
Crawl budget
The set of URLs a search engine can and wants to fetch from your site in a given period. Google describes it as a limit on how much load your server can take, together with how much demand there is for your pages. Google says it matters mostly for large sites, but logs show where crawl requests actually go, including filters, parameters and redirects.
Learn more: Understanding crawl budget and Crawl Budget report
F
Fake Googlebot
A request that sends Googlebot's user agent from an address that does not belong to Google. Scrapers and scanners do this to get past rules that let Googlebot through. A proxy or CDN that rewrites the client address can produce the same result, so check before you block.
Learn more: Free Fake Googlebot Checker
G
Google-Extended
A robots.txt token that tells Google whether content it crawls may be used for training its Gemini models and for grounding. It is not a separate crawler, so you will not see Google-Extended as a user agent in your logs. Google fetches with its usual Googlebot user agents.
Learn more: Bot management and AI crawlers
Googlebot
Google's crawler for Search. Salience tracks its variants separately (smartphone, desktop, image, video and news) and checks each request against Google's published ranges. Blocking Googlebot takes you out of Google Search, which is a different decision from the AI controls Google offers.
Learn more: Search crawler intelligence
GPTBot
OpenAI's crawler for content that may be used to train its models. OpenAI uses separate agents for ChatGPT search (OAI-SearchBot) and for user-triggered visits (ChatGPT-User). Blocking GPTBot can limit its future requests. It does not undo earlier use of your content.
Learn more: Bot management and AI crawlers
I
IP ranges
The address lists that crawler operators publish for their bots. Google, Bing, OpenAI, Anthropic, Perplexity, Apple, Amazon, DuckDuckGo and Common Crawl are among those that publish them. Checking a request's address against the list is the strongest and quickest evidence of identity, but some operators publish no list at all.
Learn more: Bot verification
L
llms.txt
A plain file some sites publish at /llms.txt for automated readers such as AI coding tools. Salience's nightly site checks look for the file and check that it is open to plain clients, so a rule meant for bad bots does not shut out tools that should read it. We do not claim that having one changes whether AI systems fetch or cite your pages.
Learn more: Site Checks in the help centre
M
MCP (Model Context Protocol)
An open protocol that lets AI assistants connect to tools and data. Salience runs an MCP server, so Claude, Cursor, ChatGPT and other clients can ask questions about your traffic and crawler data in plain language. It is listed in the official MCP Registry as com.salience/logs.
Learn more: MCP server setup and the registry listing
P
PerplexityBot
Perplexity's crawler for surfacing and linking sites in its search results. Perplexity says it is not used to crawl content for AI foundation models. Its separate fetcher, Perplexity-User, visits a page when someone asks Perplexity a question.
Learn more: Crawler directory
Prompt tracking
Asking AI assistants a chosen set of questions on a schedule and recording what the answers say. That covers whether your brand is mentioned, which sources are cited and how you compare with competitors. It measures AI answers from the outside and does not see requests to your website. Salience does not do prompt tracking today. It records the website side, and some teams use the two together.
Learn more: Server logs and prompt tracking
R
Request
One call to your server for one URL. A single page load usually triggers several, for the HTML, stylesheets, scripts and images, so request counts run higher than visitor counts. Salience counts requests, not visitors.
Learn more: Key concepts in the help centre
Reverse DNS verification
Checks a crawler's identity when there is no published address list. You look up the host name for the address, check it belongs to the operator's domain, then look that name up again and confirm it returns the same address. The second lookup matters, because a reverse record alone can be set by whoever controls the address.
Learn more: Bot verification
robots.txt
A public text file at the root of your site that asks crawlers which paths they may fetch. It is a request, not a lock. Well-behaved crawlers follow it, others ignore it, and anyone can read it. Your logs show whether a crawler actually stopped.
Learn more: Robots.txt report and Compliance monitoring
S
Server log
The file your web server or host writes, with one line per request. A line typically holds the URL, the time, the client address, the claimed user agent, the status code, the response size and the referrer if one was sent. It records what happened at your infrastructure, whether or not a browser ran any scripts.
Learn more: Why server logs matter for SEO
Sitemap
An XML file listing the canonical URLs you want search engines to find. It is a signal, not an instruction. Salience compares it with your logs and sorts each URL as recently crawled, stale (not fetched in 30 days) or never crawled.
Learn more: Sitemap Coverage
Soft 404
What you get when a page returns a 200 status but looks like an error or is empty. Common causes are empty category pages, out-of-stock products and searches with no results. Crawlers must download the page to find out it is empty, so a real 404, 410 or redirect is usually better.
Learn more: Soft 404s report
Status code
Every server response carries a three-digit number saying what happened. 2xx means success, 3xx a redirect, 4xx an error on the request such as 404, and 5xx an error on the server. Google lowers its crawl capacity when a server returns 5xx or 429 responses, so the codes your crawlers receive are worth watching.
Learn more: Status codes in the help centre
T
Training crawler
Any bot that collects content that may be used to build or train AI models. Salience counts requests from crawlers classed this way, and a request does not prove that a page was used for training.
Learn more: LLM crawlers in the help centre
U
User agent
The label a client sends with each request to say what it is, such as Googlebot or a browser name. Anyone can send any label, so a user agent is a claim, not proof. That is why identity needs a second check.
Learn more: Bot verification
User-triggered fetch
What happens when someone asks an AI assistant about a page and it fetches that page there and then. ChatGPT-User, Claude-User and Perplexity-User are examples. Your logs show the request, but not the person or the question.
Learn more: Agent analytics
V
Verified bot
A request whose claimed crawler identity passed a check, such as its address falling inside the operator's published ranges. Salience shows which tier of evidence was used (IP ranges, reverse DNS, network owner or CDN edge). It also keeps separate states for checks still pending, proxied requests and crawlers with no published check, instead of one yes or no.
Learn more: Bot verification
W
Web Bot Auth
A proposal for crawlers to sign their requests cryptographically, so a server can check identity without relying on IP addresses. It builds on HTTP Message Signatures (RFC 9421) and is still an IETF working-group draft. Cloudflare supports it for verified bots, and Google describes its own implementation as experimental.
Learn more: Bot verification