Scenariosearch and development team
“An AI crawler is getting 404s on our product URLs. Which pages, and what do we fix?
An error response traced to the pages behind it
- 1Spot the issueAn identified AI crawler is receiving 404s on product URLs.
- 2Inspect the affected pagesReview the recorded paths and responses, then ask your team to check the pages and redirects.
- 3Check what happens after the fixWhen the crawler returns, review the subsequent requests to see how your website responds.
The recorded paths and responses give the team the URLs to check, and the crawler's later requests show how the site responded after the fix. Salience does not control when a crawler returns.
Scenariocontent team
“Visits from ChatGPT and Perplexity are rising, but we can't tell which of our pages the AI systems are actually fetching.
AI referrals set beside the pages AI systems fetch
- 1MondayThe AI referral report shows 126 visits from chatgpt.com, perplexity.ai and claude.ai in the last week, landing on 34 pages.
- 2Same viewThe request record shows OAI-SearchBot, PerplexityBot and ChatGPT-User fetched 1,045 pages in the same week, the 34 landing pages among them.
- 3Tuesday11 guides were fetched more than 50 times each with no referred visit. The team reviews those guides first.
The fetched pages and the referred visits sit side by side, so the team works from recorded requests rather than guessing which pages AI systems read.