Exports you can trust: filters that apply, hostnames, and honest warnings
Request-level exports now honour bot and status filters properly, include a hostname column for multi-host feeds, and tell you explicitly when a filter you asked for doesn't apply, plus a new feed_scope field so you always know what a site's feed captures.
Feedback from a large agency engagement (a 10-million-request crawler analysis run end to end on Salience) surfaced some sharp edges in our export pipeline. All fixed today:
- Filters now apply on request-level exports. The
seo-requestsexport
accepted bot_name and status_code filters, named the file after them, and then quietly returned the full unfiltered dataset. It now honours both, along with status_category, countries and user_agent. Naming a bot explicitly also overrides the default search-engine scope, so a "GPTBot" export means GPTBot.
- No more silently ignored filters, anywhere. Every export type now
declares which filters it supports. Ask for one it can't apply and the response says so up front, with the filter listed in an ignored_filters field and a plain-English warning, instead of handing you a file that reads as filtered when it isn't.
- A
hostcolumn on request-level exports. If your feed covers several
hostnames (www, subdomains, apex), path analysis no longer mixes them together, each exported request row now says which host it hit.
- A new
feed_scopefield on website details. If a site uses traffic
retention filtering (storing bots only, for instance), the API now says exactly what the feed captures, which traffic classes are stored or dropped, and when that configuration last changed, so a scope change mid-analysis is a documented fact, not something you reverse-engineer from the data.
These fixes apply across the dashboard, public API, MCP server and CLI. More from the same feedback batch is coming: documented bot scope for every export type, an all-bots request export, and size-managed detail endpoints.