Know when your firewall starts refusing crawlers, and what it lets through
Salience now spots the day your edge policy changed, says whether a block is aimed at our checker or at every automated visitor, lists the AI crawler claims you are not verifying and the real crawlers you refuse, and flags GA sessions your server never served.
A firewall or bot-manager change rarely announces itself. The first sign is usually a crawler quietly getting a 403, a tool that stops working, or analytics that no longer add up. Four additions today make those signs visible in Salience without anyone going looking.
Edge policy change alert. Every night Salience compares the share of refused requests (401, 403 and 429) for each kind of automated visitor with the previous two weeks: verified search crawlers, verified AI crawlers, unverified bot claims, SEO tools and scrapers, and plain HTTP clients. When two or more of them are suddenly refused far more than usual on the same day, or a verified search or AI crawler is on its own, you get one alert naming the date, the groups affected and the crawlers hit. It is critical when verified search crawlers are affected, and it appears in the SEO, engineering and AI digests.
Site Checks now say who is blocked. When our nightly checker is refused, it sends one more request shaped like a normal browser from the same address. The result tells you whether a user-agent rule is refusing SalienceBot specifically, whether the rule is on our address or on every automated visitor, or whether the site is rate limiting. A new check reads robots.txt, your sitemap, llms.txt and a linked Markdown document the way a plain client does, so you find out when a rule meant for bad bots is also keeping out the search and AI crawlers, and the AI coding tools, those files exist for.
Two new recommendation lists. AI claims not verified shows AI crawlers such as GPTBot where a share of the traffic using that name came from addresses outside the operator's published ranges and your site served it anyway, with the networks, sample IPs and a ready-to-paste Cloudflare rule (plus AWS WAF guidance) that challenges the claim without touching the real crawler. Refused crawlers shows verified search, AI and social crawlers your rules are partly refusing, with the status split and the paths it happens on. Both are on the Recommendations page and in the public API, the MCP server and the CLI (salience recommendations ai-claims and refused-crawlers).
Google Analytics sessions checked against your logs. The GA sync now brings in sessions by country, and the Google Analytics page compares each country's share of sessions with its share of real human page loads in your server logs. When a country runs at a large share of GA sessions for three or more days that your server never served, Salience says so and shows engagement with and without it, so you can filter the traffic in GA4 and report on the real figures. When both GA and the logs saw the surge, it tells you the requests are real and points you at the networks behind them.
The recommendation lists work on your existing data straight away. The alert, the new checks and the GA comparison start reporting after the next nightly run.