Save focused crawl settings and pause large crawls
Reuse named crawl profiles, check selected URLs, and pause compatible crawls without starting over.
Save named crawl profiles for each website, with page limits, path rules, depth and speed settings. Check a specific list of URLs when you need a focused investigation. Each crawl keeps the settings it started with, even if you later edit its profile. Weekly crawl settings stay separate.
Pause compatible new crawls and resume from saved progress. The app distinguishes a requested pause from a completed pause, so you can see when work has stopped. Existing results remain available; older active jobs and uploaded crawls do not gain pause/resume support.
Use Site Crawler for profiles and crawl controls. The public API also accepts the new run options and pause/resume actions; profile management is in the app. Existing MCP and CLI start/cancel commands continue to work, but do not expose these new controls.
API integrations must send a page limit between 100 and 50,000. Invalid limits now return an error rather than being silently adjusted. See the API reference and crawler help for scope rules and recovery behaviour.