Request-level exports for every bot, size-safe endpoints, short download links
A new all-bots request export with a bot_scope selector, a max_rows cap that makes huge detail endpoints usable in agent tooling, short stable download links for exports, and documented bot scope for every export type.
The second batch from our agency feedback, all shipped across the API, MCP server and CLI together:
- A new
requestsexport type. Request-level rows for all bot
traffic, not just search engines. A bot_scope selector narrows it to search, ai, or all (default), so an AI-crawler request dump is now one call: create_export with export_type=requests, bot_scope=ai. Like every request-level export it includes the host column.
max_rowson every public read endpoint. Detail endpoints over long
windows can return millions of characters, unusable inline in agent tooling. Pass ?max_rows=N and every list in the response is capped at N entries, with a _truncated map reporting exactly what was cut and the original counts. The MCP server also enforces a hard response-size guard, so an oversized payload is truncated with an explicit note instead of blowing the client's tool-response limit.
- Short, stable export download links.
GET /public/v1/websites/{id}/exports/{job_id}/download redirects to the file (add ?redirect=false for JSON), no more passing around fragile 1,500-character presigned URLs. The CLI gains loglens download-export to save a completed export straight to a local file.
- Bot scope is documented everywhere. The API docs, MCP tool
descriptions and CLI help now state exactly which bots each export type covers, seo-requests is search engines only, requests is everything.
MCP server v1.4.0 is live; CLI v1.5.0 ships alongside.