Salience

Find the products search engines never reach, and the bots reading your catalogue.

Salience joins your CDN or server logs to your sitemap and Search Console, per URL. It shows which product pages search engines have yet to request, where crawl goes to filters and out-of-stock URLs, and which bots are copying your prices. It works for any store, Shopify included, with no app and no theme change.

Connect a log sourceExplore the demo

Free plan, no card, nothing added to your pages. First requests in about ten minutes on CDN integrations, and within the hour on Shopify.

The Log Explorer for shop.example, newest request first: Googlebot requested /products/wireless-headphones (200), /products/trail-shoes?colour=red&size=9 (200) and /products/old-sku-4471 (404); GPTBot requested /products/wireless-headphones (200); a client identified only as Generic Bot requested /products/wireless-headphones and /products/trail-shoes one second apart; Chrome, Safari and Edge requests sit between them. The alert: Generic Bot requested 4,102 URLs under /products/ since 13:05. Next step: Review the source IPs before deciding whether to block the range at your WAF.

Works with stores on
  • Shopify
  • BigCommerce
  • Adobe Commerce
  • Salesforce Commerce Cloud
  • commercetools

Salience connects at your CDN or server

Salience collects at the layer you control: a Cloudflare zone, a Vercel Drain, CloudFront real-time logs, Netlify, Kinsta, or the Vector agent on your own Apache or Nginx servers. Setup happens in that account, so your theme, your code and your checkout stay exactly as they are, and the shopper's response is served as before while data is sent.

  • Shopify and Shopify Plus connect on any Shopify plan, with no app and no theme change
  • BigCommerce, Adobe Commerce, Salesforce Commerce Cloud, commercetools and headless sites connect through the Cloudflare, Vercel, CloudFront, Netlify or Kinsta account in front of them
  • Collection is read-only, so every decision about blocking stays in your CDN and WAF
  • Stores on other CDNs import Apache, Nginx or CloudFront log files for the same analysis

Server log monitoring for online stores

Server logs are the one record of every request your store answered, whoever or whatever made it. Shopify exposes none to merchants on any plan, and on BigCommerce, Adobe Commerce, Salesforce Commerce Cloud and headless stores they sit at the CDN or server in front of the platform, as raw lines. Theme and tag analytics see the browsers that ran your JavaScript, so search crawlers, AI agents and scrapers pass through unrecorded. Salience records every request to your store pages at the CDN or server, with no app installed and no theme change, and reports it per URL with crawler verification, the sitemap and Search Console join, and alerts.

  • Every request to products, collections, search, sitemap and robots.txt, with the status your store returned
  • Googlebot, Bingbot and AI crawlers verified against their published ranges, with impersonators marked
  • Scrapers walking product and collection URLs, grouped by source and path
  • Alerts on crawler and error changes that name the paths and addresses involved

What this looks like in practice

Scenarioe-commerce SEO

My platform doesn't give me server logs. Search Console is always two to three days behind. I'm flying blind on what crawlers are doing.

Holiday pages Googlebot had not visited

  1. TuesdayThe Black Friday collection pages go live.
  2. FridaySalience shows no Googlebot request to any of them.
  3. FridayThe cause is a Disallow: /collections/holiday-* rule left in robots.txt from last year's campaign, and it is removed with ten days to traffic week.

The sitemap join lists the pages awaiting a first crawl, and the robots.txt history shows the rule that held them back.

ScenarioSEO lead on a heavily scraped site

Everyone claims to be Googlebot. Half of them aren't. My analytics tool can't tell the difference.

Every Googlebot claim checked against Google's ranges

  1. Every request claiming to be Googlebot is checked against Google's published IP ranges.
  2. Requests that fail the check are marked unverified, and an alert fires on any surge in unverified crawler traffic.
  3. The WAF allowlist is built from the verified Googlebot addresses, so genuine crawl is never rate-limited.

Crawl analysis counts only verified requests, and scraper traffic borrowing the Googlebot name is handled as scraper traffic.

Compare the catalogue you publish with the catalogue crawlers read

Salience fetches the product and collection URLs in your sitemap and joins them to every crawler request, so you can see, per URL, where the catalogue you publish and the catalogue search engines read differ.

  • Product URLs in the sitemap still waiting for their first search crawler request, with the last crawl date where one exists
  • New collections and their first crawl after a launch
  • Discontinued product URLs that crawlers still request, with the status your store returned
  • Crawled URLs that could be added to the sitemap

Path Analysis on shop.example, last 24 hours: Total Requests 34,163, Unique Paths 17,700, 4xx Errors 3,272, 5xx Errors 28. Top paths: / 1,704 requests, 5.0% of total, importance 7; /collections/sale 1,245 requests, 3.6% of total; /products/wireless-headphones 376 requests, 1.1% of total; /collections/all 202 requests, 0.6% of total, importance 5; /products/running-shoes 183 requests, 0.5% of total; /cart 161 requests, 0.5% of total, importance 5; /pages/shipping 120 requests, 0.4% of total.

See how much of the crawl reaches a product page

Crawl budget is the time Google spends on your store, and it matters most on large catalogues that change often and generate URLs through filters, sorting and search, which describes most online stores. When a new line lands and demand is moving with a trend, what counts is how quickly Googlebot reaches the new product pages, and every second spent on faceted, sorted and out-of-stock URLs delays that. Salience breaks Googlebot's time down by the URL patterns your catalogue produces, reports the share that reached product pages and the share that went to faceted, sorted and out-of-stock URLs, and shows when a new collection was first crawled.

  • Requests to product pages alongside requests to filtered and sorted collection URLs
  • Out-of-stock and discontinued product URLs still absorbing crawl
  • Internal search and campaign parameters requested by crawlers
  • URL patterns detected automatically, with high-cardinality sections flagged
  • Products, categories, search results, pagination and parameter URLs saved as segments, with a one-click library and period-on-period comparison per segment

The Sitemap Coverage page for shop.example: Total Active URLs 31,041, Sitemap checked 15/09/2026; Recently Crawled 22,514, Within 30 days; Stale 7,912, 30+ days old; Never Crawled 615; Coverage Rate 73%, Recently crawled / total. /, first seen 23/01/2026, 135 recorded crawls, last crawled 15h ago by Bingbot, Crawled; /collections/sale, first seen 25/02/2026, 2 recorded crawls, last crawled 25/02/2026 by Bingbot, Stale; /products/linen-shirt-navy, first seen 29/04/2026, 1 recorded crawls, last crawled 22/03/2026 by Googlebot Smartphone, Stale; /products/trail-runner?variant=41, first seen 14/02/2026, 2 recorded crawls, last crawled 14/02/2026 by Googlebot, Stale; /products/wool-overshirt-olive, first seen 12/09/2026, 0 recorded crawls, last crawled —, Never Crawled.

Catch broken product pages after a release or migration

A theme edit, an app install or a URL change can break paths quietly, and crawlers meet the change first. Salience watches how crawlers are answered against your own baseline and alerts when that changes.

  • New 404s on /products/* and 5xx responses on collection pages after a deployment
  • Redirect chains and orphaned URLs after a catalogue migration or replatform
  • Slow commercial sections, ranked by response time
  • Crawler activity dropping away from a section

Status Codes on shop.example, last 24 hours, the Status Codes Over Time chart: 24,447 2xx Success (71.6%), 6,416 3xx Redirect (18.8%), 3,272 4xx Client Error (9.6%), 28 5xx Server Error (0.1%), in half-hour buckets, stacked by status class.

Find the automation copying your prices and holding your stock

Scrapers, cart bots and credential testers skip your theme's JavaScript, so analytics apps that run in the browser never see them. The server-side request stream records every one. Salience groups automated requests by source, path and timing and reports what your store returned to each.

  • Systematic reads of every product handle from one network, whatever user agent each request carries
  • Bursts on /cart/add, account and login paths from many addresses in a short window
  • Requests carrying Googlebot's name from addresses outside Google's published ranges
  • Repeated requests against gift-card, balance or stock-check endpoints
  • Repeated requests to checkout and payment paths from many addresses in a short window, with the status each was answered
  • Bursts on account registration paths, with the networks behind them and the seven-day history of each address
  • A scored list of addresses to block, with the rule written for Cloudflare, CloudFront or Nginx

The Recommendations page for shop.example, IPs to block tab, 9 addresses with a ready-to-paste rule for your platform. 45.133.5.188, 1,412 requests, seen as scraper on /products.json, /collections/all/products.json, last seen 12m ago; 185.218.86.24, 968 requests, seen as scraper on /collections/all?page=47, /collections/all?page=48, last seen 25m ago; 88.166.28.185, 611 requests, seen as scraper on /products/trail-runner.json, /products/linen-shirt-navy.json, last seen 1h ago; 62.60.130.228, 173 requests, seen as hacking probe on /cart/add.js, /cart/add.js, last seen 38m ago; 34.12.128.52, 284 requests, seen as scraper on /collections/all?page=12, last seen 4h ago; 68.183.52.128, 96 requests, seen as hacking probe on /checkouts/, /cart/add.js, last seen 15h ago.

Decide which automated clients to let in

A blanket block on bots protects prices and bandwidth and also removes your products from the AI answers and shopping agents that increasingly make the recommendation. Salience separates the clients your store receives into classes, verifies the ones whose operators publish IP ranges, and reports each so the decision can differ by class.

The Bots and Crawlers page for shop.example, last 24 hours: 15,541 bot requests, 45.5% of all 34,163 requests · same filters. 12,233 verified, 655 failed verification, 2,653 other status. By category: Search Engines 1,781, Social Media 92, AI Crawlers 6,309, Monitoring 654, Scrapers 4,788, Other Bots 1,917. Bot traffic over time in fifteen-minute buckets, stacked by category.

What Salience reports for your store

Every figure comes from your logs, or from your logs joined to your sitemap and Search Console.

The Salience dashboard for shop.example, last 24 hours. Website setup 5 of 6 completed: Logs receiving, Google Search Console connected, Google Analytics connected, Sitemap checked, Robots.txt checked, First site crawl optional. Traffic Over Time in fifteen-minute buckets, human and bot requests stacked, with two spikes above 1,800 requests. Total requests 34.2K, up 7.0% on the previous period; Unique IPs 15.3K, up 2.0% on the previous period; Bot traffic 45.5%, down 1.9pp on the previous period; AI crawlers 18.5%, down 8.0pp on the previous period. Site Health for the last 30 minutes: traffic 639 normal, 5xx errors 1 with the baseline still learning, bots 52% normal, crawlers 45 normal, latency 0ms normal, 404s 0.2% critical.

See which products AI assistants send shoppers to

A visit is counted as AI-referred when the shopper arrived from ChatGPT, Perplexity, Claude, Gemini, Copilot or another assistant, or the URL carried a matching source tag. Salience lists the product and collection pages those shoppers land on, what they open next, and sets each AI company's fetches of your catalogue against the shoppers it sent back.

  • Landing products and collections per assistant, with the share of human traffic each accounts for
  • Fetches made by each AI company against the shoppers its answers sent back
  • Products that earn impressions in Search that no AI system has fetched
  • With Google Analytics 4 connected, sessions, key events and revenue for AI-referred shoppers next to organic search, per landing page

Connect ahead of peak trading

Black Friday, Christmas, launches and migrations change the request stream, and a code freeze can hold a fix until January. Salience is configured in your CDN account, so it can be connected during a freeze, and a baseline built before the peak is what makes the peak readable. Add site events for launches and freezes, and the alerts compare crawler and error behaviour against your own normal.

How different teams use it

E-commerce leadership

Which commercial sections search engines are reaching, which are broken, and since when, in plain terms.

SEO

Product URLs awaiting a first crawl, the crawl spent on filters and dead stock, and whether the fix changed the split.

Platform and engineering

The paths, status codes and latency behind every issue on the store, tied to the release that preceded it.

Security and fraud

Scraper, cart and login bursts with the source addresses, ready for a rule in the CDN or WAF.

Merchandising

Confirmation that a new collection or launch is being crawled, before the campaign spend starts.

Why not the tools you already have

Most stores already run Search Console, platform analytics, a site crawler and a CDN. Each answers a different question from Salience, and each keeps its place.

Search Console and Merchant Center

What it shows
Google's aggregated verdict on crawling, indexing and feed health, with example URLs and reporting about two days behind.
What Salience adds
Every request from every crawler, per product URL, as your server answered it, joined to Search Console.

Shopify Analytics and analytics apps

What it shows
Browsers that ran your theme's JavaScript.
What Salience adds
Server-side requests from every client, whether or not it ran a script, kept separate from shoppers.

Site crawlers and log-file tools

What it shows
What a crawler could fetch when you run one, or a log export you analyse by hand.
What Salience adds
A Salience crawl or an uploaded crawl joined to what external crawlers did request, continuously, against your sitemap and site events, with verified identity and alerts.

CDN, WAF and bot management

What it shows
Blocks and challenges at the edge, reported per rule.
What Salience adds
Read-only explanation of what was allowed as well as refused, grouped by source, usable by SEO and trading teams alongside security.

PIM, digital shelf and reviews platforms

What it shows
Manage, syndicate and display product content and customer reviews.
What Salience adds
Whether the pages that content lands on are reached by search and AI crawlers, alongside the PIM and reviews platform you keep.

Common questions

Do I need server access to monitor my store's logs?

No. Salience collects at the CDN or server in front of the store, so nothing is installed on the site and no theme or platform code changes. On your own Apache or Nginx servers an agent reads the log file; on a CDN the integration is configured in that account.

Can I monitor server logs on Shopify?

Yes. Shopify does not expose access logs to merchants on any plan, and Salience records every request to your store pages on any Shopify plan, with no app and no theme change. You get the same per-URL request record, crawler verification, sitemap and Search Console join and alerting as a store on Cloudflare, Vercel or its own servers.

Does it work with Shopify Plus, BigCommerce, Adobe Commerce, Salesforce Commerce Cloud or a headless site?

Shopify and Shopify Plus connect on any Shopify plan, with no app and no theme change. BigCommerce, Adobe Commerce, Salesforce Commerce Cloud, commercetools and headless sites connect through the CDN or server in front of them: Cloudflare, Vercel, AWS CloudFront, Netlify, Kinsta, or Apache and Nginx through an agent. A headless frontend on Vercel or behind Cloudflare connects the same way as any other site on that platform.

Does it add latency or touch the checkout?

Nothing is added to your pages and the shopper's response is served as before. CDN integrations forward request metadata after the response, and the Apache and Nginx agent reads the log file. On Shopify, checkout is handled entirely by Shopify, and Salience covers the store pages: products, collections, search, sitemap and robots.txt.

How are product variants and SKUs matched?

Salience works at URL level, so there is nothing to map. Variants that share a URL pattern, for example /products/handle?variant=, are grouped by that pattern, and the sitemap join shows which product URLs are being crawled and which are still waiting.

Which data does it join to my logs?

Your sitemap, robots.txt, Search Console and Google Analytics 4. Search Console brings search performance and index coverage per URL, the sitemap join lists the product URLs awaiting a first crawl, and GA4 brings sessions, key events and revenue per landing page, with organic and AI-referred sessions counted separately. Feed and listing health stay in Merchant Center.

How does it handle out-of-stock, seasonal and discontinued products?

It reports what crawlers requested and what your store returned, per URL pattern, so an out-of-stock product still answering 200 to Googlebot, or a discontinued one answering 404 to hundreds of requests, appears as crawl on that pattern. Once you act, the same report shows the crawl moving.

Can it be connected during a code freeze?

Yes. CDN integrations are configured in the CDN account, and the Shopify integration changes no theme or application code, so a freeze on either leaves setup untouched. The setup guide walks through connecting a live store.

Can it show card testing or automated sign-ups?

It shows the request pattern: volume on checkout, payment and registration paths against each path's own baseline, the addresses and networks behind it, the seven-day history of each address and the status your store returned to each request. Whether a card is fraudulent or two accounts belong to one shopper is decided by your payment provider and fraud tools, not from the log.

Can Salience identify price scrapers?

It identifies scraper patterns: high-volume systematic access to product URLs, user-agent rotation from one network, suspicious hosting ranges and impersonation of trusted crawlers, with the addresses and paths involved, ready for a rule at the CDN.

Does it classify AI shopping agents today?

Known AI crawlers are classified by purpose: model training, AI search indexing, user-triggered retrieval, and AI assistants and agents. Supported ones are verified against their operator's published IP ranges. New agents appear as unknown automation until they have a label.

How does it work with our WAF or bot-management platform?

Alongside it. Salience reports and explains, showing what was allowed as well as what was refused, grouped by source, which is the evidence a rule in your CDN, WAF or bot manager needs.

Is there a catalogue size, request or retention limit?

Plans are priced on requests, from 500,000 a month on Free to 500M on Pro and more on Enterprise, so catalogue size and SKU count are open. History runs from 30 days on Free to four years on Pro, and a seasonal year-on-year comparison fits Growth and above. Multiple stores and international domains run as separate sites in one account.

What does the free plan include?

One website, 500,000 requests a month, 30 days of history, real-time analytics, bot and AI crawler detection, all 21 site checks and two email alerts. Solo at $19 a month adds the sitemap and Search Console joins, all 16 alerts by email and Slack, log import and six months of history; webhooks and shared dashboards start on Starter at $49. Every plan except Solo has unlimited users, and no card is needed to start.

Trust & data protection

Privacy and data protection

You are the controller

We process only on your instructions. GDPR Art. 28 DPA on every account, nothing to sign.

UK data residency

AWS eu-west-2 (London). Encrypted in transit (TLS 1.2+) and at rest (AES-256).

Server-side collection

No browser tracking script and no client-side pixel.

No sale, no pooling

Your logs are never sold, never used for advertising, never shared between customers. DPA, sub-processor list and security overview available.

Related

Your logs already show what Google and the AI crawlers are doing.

Free plan, no card, nothing added to your pages. First requests in about ten minutes on CDN integrations, and within the hour on Shopify.

Connect a log sourceExplore the demo