Sign in Start free

AI Crawler Access Policy

Blocking AI bots gets treated as one decision, but the agents do different jobs: some collect training data, some fetch a page live at the moment a user asks a question. Block the second kind and you cannot be cited at all, however good the page is. The harder failure is silent - robots.txt allows the agent while a WAF rule, bot manager or CDN challenge returns 403 to it, and nobody reads the logs by user agent.

CATEGORY
AI search (GEO)
FORMAT
ai-crawler-access-diagnosis.md
STEPS
7
PRICE
Free - no account
WHEN TO REACH FOR THIS

Use when deciding which AI agents may fetch your site, or when you allow them in robots.txt and still suspect something further down the stack is refusing them.

The skill file

ai-crawler-access-diagnosis.md 
---
name: ai-crawler-access-diagnosis
description: Use when deciding which AI agents may fetch your site, or when you allow them in robots.txt and still suspect something further down the stack is refusing them.
---

# AI Crawler Access Policy

Blocking AI bots gets treated as one decision, but the agents do different jobs: some collect training data, some fetch a page live at the moment a user asks a question. Block the second kind and you cannot be cited at all, however good the page is. The harder failure is silent - robots.txt allows the agent while a WAF rule, bot manager or CDN challenge returns 403 to it, and nobody reads the logs by user agent.

## What you need first

- Current robots.txt, including any rules injected at the CDN edge or inherited by subdomains
- Raw server or CDN logs with user agent and response status, 30 days minimum, not sampled analytics
- The bot management or WAF rule set, and the name of whoever can change it
- A stated business position on training use as distinct from retrieval use

## Method

1. List the agents that actually hit you, taken from the logs rather than from a blog post. Names change between versions and a rule for an agent that never visits is noise you will maintain forever.
2. Split them by function: training collectors, live retrieval fetchers that run when a user asks a question, and search crawlers that feed AI answers inside a search engine. One blanket rule forces you to get at least two of the three wrong.
3. Decide training separately from retrieval and record the reason. This one is genuinely contested: a publisher with a licensable archive and a vendor that wants to be recommended reach opposite answers, and both are defensible.
4. Write the robots.txt groups per user agent, most specific first, with a comment naming the decision date and owner so the file is not quietly reverted at the next migration.
5. Replay a request against a real URL sending each allowed agent string, and record the status code. This is the step that exposes a WAF or bot manager overriding your robots.txt.
6. Watch status codes by agent for a fortnight after any change. A sustained 403 or challenge rate for an agent you deliberately allowed means the block is below robots.txt and the rule you wrote is decorative.
7. Re-run the whole list quarterly. New agents appear, and an agent renamed between versions inherits none of your rules.

## What this produces

A one-page access policy listing every AI agent seen in the logs, the allow or block decision, the reason, and a verified status code proving the server behaves that way.

## Where this goes wrong

- Treating robots.txt as the whole answer when a bot management vendor blocks the agent regardless and no dashboard surfaces it
- Blocking a training collector and a live retrieval fetcher with one rule, which removes you from answers to win an argument about training data
- Copying robots.txt from a publisher whose business model makes being uncitable the point
- Reading robots.txt as enforcement - it is a request, and unnamed scrapers ignore it entirely

---

From the QuQi skill library - https://www.quqi.io/skills/ai-crawler-access-diagnosis
Free to download · no account, no email

What you need first

  • Current robots.txt, including any rules injected at the CDN edge or inherited by subdomains
  • Raw server or CDN logs with user agent and response status, 30 days minimum, not sampled analytics
  • The bot management or WAF rule set, and the name of whoever can change it
  • A stated business position on training use as distinct from retrieval use

Method

  1. 01 List the agents that actually hit you, taken from the logs rather than from a blog post. Names change between versions and a rule for an agent that never visits is noise you will maintain forever.
  2. 02 Split them by function: training collectors, live retrieval fetchers that run when a user asks a question, and search crawlers that feed AI answers inside a search engine. One blanket rule forces you to get at least two of the three wrong.
  3. 03 Decide training separately from retrieval and record the reason. This one is genuinely contested: a publisher with a licensable archive and a vendor that wants to be recommended reach opposite answers, and both are defensible.
  4. 04 Write the robots.txt groups per user agent, most specific first, with a comment naming the decision date and owner so the file is not quietly reverted at the next migration.
  5. 05 Replay a request against a real URL sending each allowed agent string, and record the status code. This is the step that exposes a WAF or bot manager overriding your robots.txt.
  6. 06 Watch status codes by agent for a fortnight after any change. A sustained 403 or challenge rate for an agent you deliberately allowed means the block is below robots.txt and the rule you wrote is decorative.
  7. 07 Re-run the whole list quarterly. New agents appear, and an agent renamed between versions inherits none of your rules.

What this produces

A one-page access policy listing every AI agent seen in the logs, the allow or block decision, the reason, and a verified status code proving the server behaves that way.

Where this goes wrong

  • Treating robots.txt as the whole answer when a bot management vendor blocks the agent regardless and no dashboard surfaces it
  • Blocking a training collector and a live retrieval fetcher with one rule, which removes you from answers to win an argument about training data
  • Copying robots.txt from a publisher whose business model makes being uncitable the point
  • Reading robots.txt as enforcement - it is a request, and unnamed scrapers ignore it entirely

Use this skill in your own AI

The download is a plain markdown file with the name and trigger in its frontmatter. Where an assistant supports skills it can load itself, that frontmatter is what it reads to decide this one applies.

Claude Code Save it as ~/.claude/skills/ai-crawler-access-diagnosis/SKILL.md and Claude loads it on its own when what you are doing matches the trigger line. Put it in .claude/skills inside a project instead if the whole team should have it.
Claude Upload the file in the skills section of your settings. Once it is there it applies itself in any conversation where the trigger fits, so you do not have to remember it exists.
ChatGPT There is no skills format to install into, so paste the file contents into a Project instruction or a Custom GPT instead. It then applies to every chat in that project rather than only the one you paste it into.
Anything else Paste the markdown into the chat before your question. It works in any assistant, it just has to be pasted again each time.

Questions about this skill

When do I need a policy rather than a robots.txt edit?

When you are deciding which AI agents may fetch you, or when robots.txt already allows them and you suspect something lower down refuses. Editing robots.txt and moving on is the usual approach, and it hides the common failure: a WAF rule, bot manager or CDN challenge returning 403 to an agent you deliberately allowed, with no dashboard showing it.

What do I need in hand before starting?

Raw server or CDN logs with user agent and status code, thirty days minimum and not sampled analytics; the current robots.txt including rules injected at the edge; the bot management rule set and whoever can change it; and a stated position on training use as distinct from retrieval. Without logs you write rules for agents that never visit and maintain them forever.

What do I end up with, and which part gets used?

A one-page policy listing every agent actually seen in the logs, the allow or block decision, the reason, and a verified status code proving the server behaves that way. The status code is the part that counts. A policy with no replayed request against a real URL records an intention, not the behaviour a fetcher will meet.

What ruins this most often?

Blocking training collectors and live retrieval fetchers with a single rule. The second kind fetches a page at the moment a user asks a question, so blocking it removes you from answers entirely in order to win an argument about training data. Whether to allow training use is genuinely contested, but it is a separate decision and should be recorded separately.

More in AI search (GEO)

Answer extractability pass

Use when a page ranks well but is never quoted by AI assistants, and you need to make its claims...

AI citation monitoring

Use to find out whether assistants actually cite you, since this traffic is largely invisible in...

Entity consistency check

Use when assistants describe your company inaccurately, conflate you with another brand, or stat...

llms.txt authoring

Use when you want to tell assistants which of your pages are authoritative, and be honest with y...