Saved segments, with compare-to-previous

Define page groups once, blog, products, categories, old URLs, and filter every page to one of them; the Segments page shows each group's bot activity against the previous period, which is the migration view.

Segments (SEO → Segments). Define the parts of your site once with simple rules, path starts with, contains, exactly, matches a regex, or query string contains, with exclusions and nesting. Add the common ones in one click, preview what a rule matches before saving, and choose whether groups are exclusive.

Applied everywhere. A segment picker sits in the header next to the country filter. Pick one and every page, chart and export, and the API, MCP and CLI via segment=, filters to it. Segments are evaluated on your logs when you ask, so they cover all history, including before you created them.

Compare with previous. The Segments page shows every group over the period: requests, humans, verified search-bot hits and share, Googlebot, AI bots, unique paths, crawl coverage and errors, each with the change against the previous period of the same length. Define old URLs and new URLs and you have migration monitoring.

← Crawl Join, a crawler of our own, and a Crawl Audit crossed with your logsOperator logos next to every bot name →
← All changelog entries