---
name: ai-crawler-access-diagnosis
description: Use when deciding which AI agents may fetch your site, or when you allow them in robots.txt and still suspect something further down the stack is refusing them.
---

# AI Crawler Access Policy

Blocking AI bots gets treated as one decision, but the agents do different jobs: some collect training data, some fetch a page live at the moment a user asks a question. Block the second kind and you cannot be cited at all, however good the page is. The harder failure is silent - robots.txt allows the agent while a WAF rule, bot manager or CDN challenge returns 403 to it, and nobody reads the logs by user agent.

## What you need first

- Current robots.txt, including any rules injected at the CDN edge or inherited by subdomains
- Raw server or CDN logs with user agent and response status, 30 days minimum, not sampled analytics
- The bot management or WAF rule set, and the name of whoever can change it
- A stated business position on training use as distinct from retrieval use

## Method

1. List the agents that actually hit you, taken from the logs rather than from a blog post. Names change between versions and a rule for an agent that never visits is noise you will maintain forever.
2. Split them by function: training collectors, live retrieval fetchers that run when a user asks a question, and search crawlers that feed AI answers inside a search engine. One blanket rule forces you to get at least two of the three wrong.
3. Decide training separately from retrieval and record the reason. This one is genuinely contested: a publisher with a licensable archive and a vendor that wants to be recommended reach opposite answers, and both are defensible.
4. Write the robots.txt groups per user agent, most specific first, with a comment naming the decision date and owner so the file is not quietly reverted at the next migration.
5. Replay a request against a real URL sending each allowed agent string, and record the status code. This is the step that exposes a WAF or bot manager overriding your robots.txt.
6. Watch status codes by agent for a fortnight after any change. A sustained 403 or challenge rate for an agent you deliberately allowed means the block is below robots.txt and the rule you wrote is decorative.
7. Re-run the whole list quarterly. New agents appear, and an agent renamed between versions inherits none of your rules.

## What this produces

A one-page access policy listing every AI agent seen in the logs, the allow or block decision, the reason, and a verified status code proving the server behaves that way.

## Where this goes wrong

- Treating robots.txt as the whole answer when a bot management vendor blocks the agent regardless and no dashboard surfaces it
- Blocking a training collector and a live retrieval fetcher with one rule, which removes you from answers to win an argument about training data
- Copying robots.txt from a publisher whose business model makes being uncitable the point
- Reading robots.txt as enforcement - it is a request, and unnamed scrapers ignore it entirely

---

From the QuQi skill library - https://www.quqi.io/skills/ai-crawler-access-diagnosis