Free PDF
Get the guide by email
Five chapters, the weekly and quarterly checklists, and the 2026 crawler table. We email you the PDF link; nothing else unless you ask.
What is inside
| Chapter | What you will be able to do |
|---|---|
| 1 · SEO and crawl optimisation | Read crawl budget from your logs, verify Googlebot properly (published ranges, reverse DNS, network consistency), find the sitemap URLs Google never visits, and run the four-quadrant crawled-versus-indexed analysis with Search Console. |
| 2 · Security and threat detection | Spot scrapers and vulnerability scanners by their log signature, tell a blocked impersonator from a blocked real bot, run the 21 site safety checks, and set alerts that name the address, the path and the fix. |
| 3 · Bot management and AI crawlers | The 2026 crawler table by purpose: training, search indexing and on-demand answering. Measure what each bot costs you, check robots.txt compliance in the logs, and work out your give-and-take ratio with each AI operator. |
| 4 · Traffic intelligence | Why JavaScript analytics misses 30–50% of traffic, how to measure your own gap against Google Analytics, and how to see the humans that AI assistants are now sending you. |
| 5 · Putting it into practice | The weekly, monthly and quarterly cadence, the three alerts to start with, and what to automate. |
Who it is for
SEO leads, security and platform engineers, and anyone who owns a website's traffic numbers. It assumes you can read a URL and a status code, nothing more.
Read the chapters online
Server log analysis for SEO
Crawl budget, bot verification, sitemap coverage and Search Console correlation.
Security and threat detection
Fake bots, scrapers, scanners, safety checks and alerts that act in minutes.
Bot management and AI crawlers
The 2026 crawler landscape by purpose, crawl impact and robots.txt policy.
Traffic intelligence
The JavaScript blind spot, the measurement gap and humans arriving from AI.