Site Checks phase 2: active probes and serving-quality checks
Site Checks now includes active probes: Salience fetches your site directly to verify what logs can't show, like whether robots.txt is served statically, crawlers are kept out of A/B tests, and HTTPS canonicalisation is clean.
Some serving problems never appear in your logs, because they live in the responses your site sends, cookies set on robots.txt, missing cache headers, crawlers being bucketed into A/B experiments. Site Checks now covers these with active probes: Salience fetches your site directly (a handful of requests nightly, clearly identified as SalienceBot/1.0) and inspects what comes back.
New probe checks
- Site reachable by our checker, if your firewall or bot manager blocks our probe, we tell you plainly rather than guessing. Blocked probes never masquerade as site problems: the dependent checks show as not evaluated, with instructions to allowlist the
SalienceBotuser agent. - robots.txt served statically and cacheably, 200,
text/plain, no cookies, cacheable, stable ETag/Last-Modified. A dynamically-served robots.txt makes crawlers refetch it constantly and trust it less. - Sitemap served properly, correct XML content type, no cookies, cacheable. We follow the
Sitemap:directive in your robots.txt to find it. - Crawlers kept out of A/B tests, we fetch your homepage twice with a crawler user agent; if experiment cookies or variant headers differ between fetches, search engines are seeing your experiments instead of your pages.
- HTTPS and host canonicalisation, http→https and www/apex should each be a single permanent redirect.
- robots.txt responds quickly, time to first byte on your most-fetched utility file.
New serving-quality checks from your logs
- Crawlers are not hitting server errors (5xx rate to verified crawlers)
- Crawler response times are healthy (where your traffic source carries timings)
- Soft 404 level and redirect hygiene for crawlers
Probes are on by default and documented; you can switch them off per site in Website Settings → Site Check Probes. The probe also settles an old ambiguity: when vulnerability scanners hit paths like /.env and get a 200, we now verify whether your site simply serves a catch-all page for every unknown path, so you're only alerted about sensitive-file exposure when it's likely to be real.
On your dashboard, with history
Your site dashboard now shows a Site Checks summary, how many checks are passing, with any failing checks called out in detail and a link to the full page. And every check now keeps a verdict history: a day-by-day strip on the Site Checks page, plus a "Recent changes" list so you can see exactly when a problem started or cleared.