AI SEARCH (GEO)

Decide which AI crawlers to allow

Use when you need a defensible position on which AI crawlers may fetch your site.

ai-crawler-access-policy.md

Download .md

You are advising on crawler access policy. You are not predicting traffic and you are not writing a public statement.

Site and what it publishes: {{SITE_AND_CONTENT}}
User agents seen in server logs, with hit counts: {{AGENT_LOG_SUMMARY}}
Our commercial position on assistants using our content: {{BUSINESS_STANCE}}
Content that is licensed, gated or sold: {{RESTRICTED_CONTENT}}

Produce four sections.

1. A table: | User agent | Hits in {{AGENT_LOG_SUMMARY}} | Paths it concentrates on | Recommended rule (allow / allow with path limits / disallow) | What allowing costs us | What blocking costs us |
2. A robots.txt block written out in full, with one comment line above each group giving the reason in plain terms.
3. The paths from {{RESTRICTED_CONTENT}} that should be disallowed for every agent, and whether each also needs server-side enforcement.
4. Which decisions in {{BUSINESS_STANCE}} are reversible within days and which are not, with the reason.

Constraints:
- Judge every agent from {{AGENT_LOG_SUMMARY}} and {{SITE_AND_CONTENT}} only. Do not add agents from memory or state what a named agent does with what it fetches.
- Do not present robots.txt as enforcement. Name the agents in the log that would need rate limiting or a firewall rule instead.
- Flag any rule that would also block a path assistants need in order to describe us accurately.

Fill in before running

Replace each placeholder with your own detail. The more specific you are, the less the model invents.

{{SITE_AND_CONTENT}}
{{AGENT_LOG_SUMMARY}}
{{BUSINESS_STANCE}}
{{RESTRICTED_CONTENT}}

Getting a better result

Pull the agent list from your own access logs; blocking agents that never visit is theatre.
Decide the gated content rule first, since that is the one with a contract behind it.
Recheck when a new user agent string appears in the log rather than on a calendar.

Questions about this prompt

When do I need a policy rather than a default setting?

When someone has asked whether to block AI crawlers and you need a position you can defend. Whether blocking costs visibility in assistant answers is genuinely contested and depends on which agents fetch for retrieval rather than training, so the prompt weighs both costs per agent instead of ruling for you.

What do I need in front of me?

A user agent summary from your own access logs with hit counts and the paths each concentrates on, a written commercial stance, and the list of content that is licensed, gated or sold. The prompt will not add agents from memory, so anything missing from your log summary gets no rule.

What comes back, and what is the useful part?

A per-agent table showing what allowing and blocking each would cost, a commented robots.txt block, the paths to disallow for everyone, and which decisions are reversible within days. The two cost columns are the useful part, because they force the trade-off to be stated rather than assumed by whoever edits the file.

What is the mistake that costs me?

Believing robots.txt enforces anything. It is a request, and agents that ignore it need rate limiting or a firewall rule, which is why section three asks which paths also need server-side enforcement. Watch too for a rule blocking the pages assistants need in order to describe you accurately.