AI assistants and agents: are they getting past your homepage?
A new table on the Bots page shows the AI assistants and agents that visit for a user (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent), how far into the site each one gets, and where it is refused. Google-Agent is now named and verified against Google's published addresses.
Google has published a guide to allowing Google-Agent, the agent that browses a site for a user and can search, fill in a form or add to a basket. Its troubleshooting advice is that allowing the first page is not enough: the pages after it, the background calls and the page's own files have to be allowed too, and firewall rules often block those without anyone noticing. A crawl of the homepage will not show it. The log will.
What changed
- AI assistants and agents on the Bots page, in two groups. Assistants are fetchers that read one page to answer a question (ChatGPT-User, Claude-User, Perplexity-User). Agents are browsers driven for a user (Google-Agent). Each bot's requests are placed at four depths (home, pages, calls, resources), with the share refused or challenged at each and the paths refused most.
- A finding when the first page is fine and the rest is not: "deeper pages blocked" when the homepage is served but 30% or more of other pages and calls are refused or challenged, and "resources blocked" when an agent is served the page but not the scripts and styles it needs.
- Impostors are left out. Scanners use the assistants' names to probe for secrets and are rightly refused, which would read as the site blocking the assistant. Requests that fail the identity check, or come from an address that asked for a secrets file, are counted apart.
- Google-Agent is named and verified. Both forms of its user agent are recognised (one was previously filed as a generic crawler), and its requests are checked against the address list Google publishes for it.
GET /public/v1/websites/{id}/bots/agentscarries the same figures; the MCP server addsget_ai_agents(1.35.0) and the CLIbots-agents(2.16.0).
A block at a CDN or firewall is only visible on feeds that record the edge's action (Cloudflare Logpush and Vercel). On other feeds the table says the check is not measured. Agents that run in a person's own browser look like any other visitor and are not counted.
See AI assistants and agents in the help.