Salience

How Are Publishers Controlling AI Crawlers?

Explore how news and media publishers around the world allow or block AI crawlers. Compare robots.txt policies across thousands of websites, including AI training, search and retrieval bots, with changes tracked daily.

Or start from a crawler:

Share of readable sites

Last refreshed

Crawlers, ranked by how often they are blocked

Select a crawler to list the publishers blocking it.

  • Blocked
  • Partial
  • Open
  • Unknown

Find a publisher

Loading publishers
  • Blocked
  • Partial
  • Open
  • One square per crawler
Publishers and their declared crawler policy

Latest policy changes

Listed only when a tracked crawler's verdict moves. Edits that leave every verdict the same are stored and left out.

    Compare regions, countries and owners

    Share of readable sites blocking at least one crawler in each category. Select a row to filter the chart and the publisher list.

    Blocking by group

    How the index works

    Every figure here comes from robots.txt files. A rule in that file is a site's stated position.

    Why a robots.txt rule is not proof of crawler behaviour

    robots.txt is a request. A crawler can ignore it, and a site can block a crawler at its CDN or server with 403s, challenges or rate limits that never appear in the file.

    Only server logs show which crawlers arrive, whether they are genuine and what they receive. Salience measures that for connected sites.

    What a block means

    Each crawler token is tested against the site root and a representative article path (a sample article), using Google's robots.txt rules.

    Blocked
    Disallowed on both paths.
    Partial
    Disallowed on one of the two.
    Open
    Allowed on both, including sites with no robots.txt, where no rule applies.
    Unknown
    The file could not be read, so no policy is shown.

    Percentages on this page count blocked and partial together.

    How crawlers are grouped
    Which sites are included

    News and media sites, each listed with a country and an owner.

    How the index is refreshed

    robots.txt files are read daily with a declared Salience user agent.

    If a file cannot be read (403, timeout, challenge page), the last readable policy is kept and the site is marked unreadable. A fetch failure is never reported as a policy change.

    Site and operator favicons are copied into Salience's own storage weekly, so nothing on this page loads from a third-party icon service.

    See which AI crawlers actually reach your site, which are genuine and what your server gives them.

    Connect Salience
    Filters