Server Log Bot Verification

Any client can send a Googlebot user agent string, and on most public sites a meaningful share of the hits claiming to be one are scrapers, monitoring tools and AI crawlers wearing the name. Crawl budget work built on an unverified log is wrong before the analysis starts, and firewall rules written from it block the wrong things. This applies whenever logs are the evidence, and always before anyone blocks a crawler at the edge.

Le fichier de compétence

---
name: server-log-bot-verification
description: Use when crawl analysis, a bandwidth spike or a proposal to block a crawler rests on log lines that say Googlebot, and nobody has checked whether they were.
---

# Server Log Bot Verification

Any client can send a Googlebot user agent string, and on most public sites a meaningful share of the hits claiming to be one are scrapers, monitoring tools and AI crawlers wearing the name. Crawl budget work built on an unverified log is wrong before the analysis starts, and firewall rules written from it block the wrong things. This applies whenever logs are the evidence, and always before anyone blocks a crawler at the edge.

## Ce qu’il vous faut d’abord

- Raw access logs with IP, user agent, status, bytes and timestamp, 14 days minimum and unsampled
- The ability to run reverse and forward DNS lookups from the log host, or the published IP ranges for the crawlers you care about
- robots.txt plus any CDN or WAF bot rules already in force
- A list of which crawlers matter commercially, search and AI assistants kept separate

## Méthode

1. Group requests by distinct user agent string with counts and bytes. Sorting by claimed identity first is what makes the impersonation visible at all.
2. Verify every hit claiming to be a search crawler by reverse DNS on the IP, then forward-resolve the returned hostname and confirm it maps back to the same IP. Where reverse DNS is not offered, match against the operator's published IP range file, refetched rather than remembered.
3. Tag each request verified, unverified or other, and keep the unverified set rather than deleting it. It is your scraping and load picture, and it is the only place a spoofer is visible.
4. Recompute crawl frequency, crawl depth and status distribution on the verified segment only. Every conclusion about crawl budget comes from this segment and no other.
5. Quantify what the unverified traffic actually costs in origin response time, 5xx rate under load and bandwidth. Rate-limit where the source may still be a tool someone relies on, and reserve outright blocking for volume that is demonstrably hurting the origin.
6. Decide AI and training crawlers separately from search crawlers, and write down the reason. This is genuinely contested: allowing them buys visibility in assistant answers that may never send a click, blocking them protects content that is being used without return, and neither position is settled.
7. Repeat after 30 days. Impersonators change user agent, legitimate crawlers change IP ranges, and a verification done once decays into a false sense of a clean log.

## Ce que ça produit

A verified-crawler log segment, a per-crawler allow, rate-limit or block decision with its stated reason, and the residual unverified traffic quantified in requests and bandwidth.

## Là où ça dérape

- Verifying by user agent, or against an IP range file downloaded months ago, so a scraper stays inside your crawl budget numbers and outside your rate limits
- Blocking at the WAF before trying a robots.txt Disallow, which turns a crawler that would have obeyed a rule into a stream of 403s you now have to explain
- Sampling one day in seven, since crawl arrives in bursts and a sampled day is either all burst or no burst
- Reading a Googlebot string from a cloud hosting range as proof of interference, when third-party emulators and testing tools send the same string legitimately

---

Extrait de la bibliothèque de compétences QuQi - https://www.quqi.io/fr/skills/server-log-bot-verification

Ce qu’il vous faut d’abord

• Raw access logs with IP, user agent, status, bytes and timestamp, 14 days minimum and unsampled
• The ability to run reverse and forward DNS lookups from the log host, or the published IP ranges for the crawlers you care about
• robots.txt plus any CDN or WAF bot rules already in force
• A list of which crawlers matter commercially, search and AI assistants kept separate

Méthode

1. 01 Group requests by distinct user agent string with counts and bytes. Sorting by claimed identity first is what makes the impersonation visible at all.
2. 02 Verify every hit claiming to be a search crawler by reverse DNS on the IP, then forward-resolve the returned hostname and confirm it maps back to the same IP. Where reverse DNS is not offered, match against the operator's published IP range file, refetched rather than remembered.
3. 03 Tag each request verified, unverified or other, and keep the unverified set rather than deleting it. It is your scraping and load picture, and it is the only place a spoofer is visible.
4. 04 Recompute crawl frequency, crawl depth and status distribution on the verified segment only. Every conclusion about crawl budget comes from this segment and no other.
5. 05 Quantify what the unverified traffic actually costs in origin response time, 5xx rate under load and bandwidth. Rate-limit where the source may still be a tool someone relies on, and reserve outright blocking for volume that is demonstrably hurting the origin.
6. 06 Decide AI and training crawlers separately from search crawlers, and write down the reason. This is genuinely contested: allowing them buys visibility in assistant answers that may never send a click, blocking them protects content that is being used without return, and neither position is settled.
7. 07 Repeat after 30 days. Impersonators change user agent, legitimate crawlers change IP ranges, and a verification done once decays into a false sense of a clean log.

Ce que ça produit

A verified-crawler log segment, a per-crawler allow, rate-limit or block decision with its stated reason, and the residual unverified traffic quantified in requests and bandwidth.

Là où ça dérape

• Verifying by user agent, or against an IP range file downloaded months ago, so a scraper stays inside your crawl budget numbers and outside your rate limits
• Blocking at the WAF before trying a robots.txt Disallow, which turns a crawler that would have obeyed a rule into a stream of 403s you now have to explain
• Sampling one day in seven, since crawl arrives in bursts and a sampled day is either all burst or no burst
• Reading a Googlebot string from a cloud hosting range as proof of interference, when third-party emulators and testing tools send the same string legitimately

Utiliser cette compétence dans votre propre IA

Le fichier téléchargé est un simple markdown dont l'en-tête porte le nom et le déclencheur. Quand un assistant sait charger des compétences tout seul, c'est cet en-tête qu'il lit pour décider que celle-ci s'applique.

Plus dans SEO technique