Shopify server log monitoring
See every crawler, bot and error on your Shopify store.
Shopify Analytics shows what shoppers do. Salience shows the Googlebot requests, AI crawlers, scrapers and error responses those reports never record.
- Any Shopify plan
- No Shopify app
- Cloudflare connection required
Log Explorer
shop.example · requests in real time
GooglebotVerified/products/linen-shirt200
PerplexityBotVerifiedAI search/collections/summer200- GooglebotUnverified/products.json?limit=250200
- GooglebotVerified/products/canvas-tote-natural404
ClaudeBotVerifiedTraining/pages/size-guide200
BingbotVerified/collections/all?page=48200
Requests to /products.json from 198.51.100.24, an address outside Google's published ranges.
Recommended: review the address in Recommendations and add the generated WAF rule.
Shopify Analytics and Salience answer different questions
Shopify Analytics runs in the shopper's browser, through your theme's JavaScript, and reads your orders. Crawlers, AI bots and scrapers request the page and leave without running any JavaScript, so the only record of them is the request itself.
Shopify Analytics
Your theme's browser script and Shopify's order records
- How many people visited, and from which channels?
- Which products sold, and what was the conversion rate?
- Which campaigns brought in revenue?
- What did shoppers do between landing and checkout?
Orders, revenue and customer records stay in Shopify.
Salience
Every request to your domain, recorded at your Cloudflare edge
- Which crawlers requested which product URLs, and were they who they said they were?
- What status code did each request get back?
- Which AI systems request your catalogue, and which pages do AI assistants send people to?
- Which addresses are working through the catalogue page by page?
Salience reads no order, payment or customer records.
Questions your Shopify reports can't answer
Four examples from the four teams that use Salience on a store. The store, addresses and figures in the panels are examples.
01Search and SEO
Can Googlebot reach your products?
Shopify creates a URL for every page of every collection and for each filter a shopper applies. Googlebot can spend much of its time on those and reach new or changed products late, and Shopify Analytics records no crawler visits at all.
Salience records every verified Googlebot request with the URL, the status code it received and the time, groups the requests by section, and compares the URLs Googlebot fetched with the URLs in your sitemap.
Find sitemap products Googlebot has not fetched, see how much crawling goes to paginated and filtered collections, and see error responses to Googlebot on the day they start.
Verified Googlebot by section
shop.example · last 7 days
| Section | Requests | URLs | Unfetched |
|---|---|---|---|
| Products/products/* | 4,812 | URLs fetched1,903 | Sitemap URLs not fetched41 |
| Collections/collections/* | 1,236 | URLs fetched214 | Sitemap URLs not fetched0 |
| Paginated collections?page= | 2,941 | URLs fetched1,388 | Sitemap URLs not fetchedNot in sitemap |
| Filtered collections?filter. | 1,507 | URLs fetched1,122 | Sitemap URLs not fetchedNot in sitemap |
Sources requesting the catalogue
shop.example · last 24 hours
Requests per hour from unverified automation
peak 2,734
Unverified automated requests ran at about 100 an hour, rose to between 2,544 and 2,734 an hour from 03:00 to 10:59, then returned to about 100.
- 203.0.113.0/24Not a crawler18,406
Hosting provider · 4 rotating browser user agents · Every /products/* URL in order, no images or scripts
- 198.51.100.24Unverified3,120
Hosting provider · Googlebot (claimed) · /products.json?page=1 to 62
- GooglebotVerified1,962
Google · Googlebot · Products and collections, follows robots.txt
- 192.0.2.77Not a crawler1,284
Hosting provider · python-requests/2.32 · /collections/all?page=1 to 96
02Merchandising and catalogue protection
Who is requesting your whole catalogue?
Price and catalogue scrapers request every product page, often from datacentre addresses, rotating browser user agents or borrowing Googlebot's name. They run no JavaScript, so Shopify Analytics never sees them.
Salience groups requests by source address and user agent, checks every claimed crawler name against the operator's published addresses, and shows each source's paths, request rate and timing. The evidence is the address, the network, the user agent and the pattern of requests.
Rate-limit or block the sources in your own Cloudflare account. Recommendations list the addresses and fake crawler user agents and generate a WAF custom-rule expression for you to paste.
03AI search and discovery
Which AI systems are requesting your products?
AI companies run separate crawlers for collecting training data, for building an AI search index and for fetching a page when someone asks an assistant a question. Each one is a separate robots.txt decision.
Salience puts every AI crawler in one of those three classes, verifies it where the operator publishes its addresses, and lists the products and collections each one requested.
A recorded request shows that a page was fetched. It does not show whether an answer used or cited the page.
The AI referral traffic report counts the human visits that arrive from ChatGPT, Perplexity, Claude, Gemini, Copilot and other assistants, by landing page.
AI crawlers by purpose
shop.example · last 7 days · verified requests
GPTBot2,104- ClaudeBot1,388
Most requested /products/linen-shirt
- OAI-SearchBot612
- PerplexityBot498
Most requested /collections/summer
- ChatGPT-User233
- Perplexity-User41
Most requested /pages/size-guide
04Engineering and operations
What broke after your last store change?
A theme release, an app or a change to your URLs can start returning errors on pages nobody on your team is checking.
Salience tracks status codes per path and per type of client against your own history. Site Events mark your releases and changes on every chart, and the Elevated 404s and Crawler Error Spike alerts fire when error rates rise above that baseline.
The alert lists the affected paths and which clients received the errors, with a recommended next step.
404 responses by client
shop.example · daily share · 14 days
- Verified search crawlers
- 0.6% 12.3%
- Verified AI crawlers
- 0.9% 15.1%
- Human visitors
- 0.4% 7.9%
- Verified search crawlers
- Verified AI crawlers
- Human visitors
- Site event · new theme published
404s rose after the new theme was published. Its navigation links to 6 collection URLs that do not exist, and shoppers and crawlers are following them.
Recommended: correct the menu links in the theme, or add URL redirects in Shopify for the 6 paths.
Example investigation
Googlebot started receiving 404s on 12 product URLs
An example of how an alert becomes a fix, step by step. The store, URLs and figures are illustrative.
- 1
The alert fires
At 10:15 a Crawler Error Spike alert reports that verified Googlebot requests to product URLs are returning 404 far above the store's usual rate.
- 2
The crawler is Google's
Every request came from an address inside Google's published Googlebot ranges, so these are Google's own fetches being refused a page.
- 3
The affected URLs
76 requests across 12 product URLs. Each URL is a product handle that existed before the morning's product import.
- 4
When it started
The first 404 arrived at 10:02, 22 minutes after the import your team recorded as a Site Event at 09:40. The requests continued through the day.
- 5
The next step
Create URL redirects in Shopify from the old handles to the current product URLs, then check that Googlebot receives a 301 and then a 200 on its next visits.
Crawler Error Spike
shop.example · opened 10:15 · Googlebot · 404
GooglebotVerified66.249.66.0/24 · in Google's published ranges- /products/linen-shirt-white40411
- /products/linen-shirt-navy4049
- /products/canvas-tote-natural4048
- /products/canvas-tote-black4048
- /products/merino-crew-oat4047
- /products/merino-crew-charcoal-grey-long-sleeve4046
- 6 more URLs · 27 requests
09:40 Site event · product import
No 404s in the 09:00 hour before the import at 09:40; then 4, 9, 12, 14, 11, 10, 9 and 7 per hour from 10:00 to 17:00, 76 in total.
Recommended next step
Redirect the 12 old product handles to their current URLs with Shopify URL redirects, then confirm Googlebot receives 301 followed by 200.
What Salience shows for a Shopify store
Every report comes from the same request data. Search Console and Google Analytics 4 add search and session figures to the same URLs if you connect them.
URL detail
shop.example/products/linen-shirt · last 28 days
/products/linen-shirt
- LogsVerified Googlebot fetchesLast 200 at 08:1214
- LogsAI page fetchesChatGPT-User, Perplexity-User6
- LogsAI-referred visitschatgpt.com, perplexity.ai4
- Search ConsoleImpressions, 28 daysIndexed, last inspected 3 Oct3,180
- Google AnalyticsSessions, 28 daysOrganic search 71%612
Who is requesting
Verified crawler identities
200+ crawler identities, checked against published address ranges where the operator publishes them.
AI crawlers and AI referrals
Training, AI search and user-triggered fetches separated, with AI-referred visits by landing page.
Scraper and automation patterns
Systematic catalogue requests grouped by address, network and user agent.
What they request
Every request, URL by URL
Log Explorer and a URL page for each path, with status, client, timing and source.
Products, collections and parameters
One click adds segments for products, collections, search results, pagination and parameter URLs, applied to all history.
Sitemap and robots.txt
Checked automatically when the store is added, with crawl activity set against sitemap coverage.
What happens next
Status codes and errors
Status codes per path, soft 404s, redirects and response times against your own baseline.
Alerts and recommendations
Alerts by email, Slack or webhook with a recommended next step, and a WAF rule generated from the recommendations.
Reports
A weekly email report, a daily digest and a written technical SEO report.
Salience also runs its own site crawler and crawl audit, and has a public API, a CLI and an MCP server.
How to set up server log monitoring on Shopify
Setup takes about 10 minutes. The app walks you through each step and checks the connection at the end.
- 1
Enter your store domain and create your account
Use the custom domain shoppers see, such as yourstore.com. The Free plan needs no card.
- 2
Connect Cloudflare and the Salience Worker
Add your domain to a free Cloudflare zone, point your store's hostnames at Shopify with proxied CNAME records (Shopify's supported Orange-to-Orange setup), then deploy the Salience Worker and set its routes to Fail open.
- 3
Confirm requests are arriving and start investigating
The connection check sends three test requests to your store and confirms they reach Salience. From then on, requests appear in Log Explorer within seconds.
What you need
- A custom domain on any Shopify plan
- Access to your domain's DNS at the registrar
- A Cloudflare account, on the free plan or above
What stays the same
- No Shopify app to install
- No theme or code changes
- Shopify keeps serving every page
Before you switch
- Copy your MX and verification TXT records into Cloudflare first, so email keeps working
- Set Fail open on every Worker route. It lets requests through to Shopify if the Worker's daily allowance runs out
- Collection covers every path except /checkout, where Cloudflare does not run Workers
- History starts when collection is connected
Someone else manages your DNS? Send them a delegated setup link from the app. It is limited to this store, and they complete the Cloudflare and Worker steps without access to your Salience account.
How your request data is collected and controlled
What is collected
- Timestamp, method, host and path
- Status code and response size
- Client address and user agent
- Referrer, country and cache status
What is not collected
- Requests to /checkout
- Order, payment and customer records
- Request and response bodies
- Shopify's own origin logs
Where it is processed
- AWS eu-west-2 (London), with encrypted S3, DynamoDB and Aurora storage
- A GDPR Article 28 DPA applies to every account
- Never sold, used for advertising or shared between customers
Who controls it
- The Worker runs in your own Cloudflare account
- Removing its route stops collection straight away
- Reverting your DNS takes the domain out of Cloudflare
The Cloudflare Worker allowance
Cloudflare's Workers Free plan allows 100,000 Worker requests a day across your Cloudflare account, resetting at 00:00 UTC. The Worker runs on every request to your store's domain, including images, scripts and bots, so a busy store can use the allowance within hours; after that, Salience receives no requests until midnight UTC and Fail open keeps the store serving. Workers Paid costs $5 a month and includes 10 million requests.
Shopify questions
Can I access server logs from Shopify?
No. Shopify runs the servers behind every store and no plan, app or API gives merchants their access logs. Salience records the requests to your own custom domain instead, through a Cloudflare zone you own in front of Shopify, which is a configuration Shopify supports.
Does this work with any Shopify plan?
Yes, on any plan, as long as the store uses a custom domain whose DNS you control. A yourstore.myshopify.com address cannot be connected, because the setup runs through your domain's DNS.
Do I need a developer or a Shopify app?
There is no Shopify app and no theme or code change. The setup is DNS and Cloudflare work, which anyone comfortable editing DNS records can do in about 10 minutes. If someone else manages your DNS, send them a delegated setup link from the app; they complete the steps without access to your Salience account.
Will configuring Cloudflare affect my store?
Shopify keeps serving the store and the Worker does not hold a shopper's response while it sends request data. The settings that matter are in the setup guide: SSL/TLS mode Full, Always Use HTTPS off, mail records left unproxied, and Fail open on every Worker route, which lets requests through to Shopify when the Worker's daily allowance is used up. Fail open does not cover an error in the Worker or at Shopify. Removing the Worker route stops collection, and reverting your DNS undoes the change.
Can I see Googlebot and AI crawlers?
Yes. Every request from Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and 200+ other crawlers is recorded with the URL and the status code it received, and checked against the operator's published addresses where they publish them. AI crawlers are split into training, AI search and user-triggered fetches. A recorded fetch shows that a page was requested. It does not show whether an answer used it.
Can I see historical logs or checkout data?
Collection starts when the Worker is connected; Shopify keeps no request history that can be imported. Requests to /checkout are not collected, because Cloudflare does not run Workers on that path, and Salience reads no order, payment or customer data from Shopify.
What does monitoring cost?
The Free plan covers one website, 500,000 requests a month and 30 days of history, with no card. Paid plans start at $19 a month, each with a 14-day trial. On the Cloudflare side the zone is free, Workers Free allows 100,000 Worker requests a day, and Workers Paid is $5 a month for 10 million.
Start monitoring your Shopify store for free.
The Free plan needs no card. Setup needs a Cloudflare account and access to your domain's DNS.