# Decide which AI crawlers to allow

> Use when you need a defensible position on which AI crawlers may fetch your site.

## Preencher antes de executar

- `{{SITE_AND_CONTENT}}`
- `{{AGENT_LOG_SUMMARY}}`
- `{{BUSINESS_STANCE}}`
- `{{RESTRICTED_CONTENT}}`

## Prompt

```
You are advising on crawler access policy. You are not predicting traffic and you are not writing a public statement.

Site and what it publishes: {{SITE_AND_CONTENT}}
User agents seen in server logs, with hit counts: {{AGENT_LOG_SUMMARY}}
Our commercial position on assistants using our content: {{BUSINESS_STANCE}}
Content that is licensed, gated or sold: {{RESTRICTED_CONTENT}}

Produce four sections.

1. A table: | User agent | Hits in {{AGENT_LOG_SUMMARY}} | Paths it concentrates on | Recommended rule (allow / allow with path limits / disallow) | What allowing costs us | What blocking costs us |
2. A robots.txt block written out in full, with one comment line above each group giving the reason in plain terms.
3. The paths from {{RESTRICTED_CONTENT}} that should be disallowed for every agent, and whether each also needs server-side enforcement.
4. Which decisions in {{BUSINESS_STANCE}} are reversible within days and which are not, with the reason.

Constraints:
- Judge every agent from {{AGENT_LOG_SUMMARY}} and {{SITE_AND_CONTENT}} only. Do not add agents from memory or state what a named agent does with what it fetches.
- Do not present robots.txt as enforcement. Name the agents in the log that would need rate limiting or a firewall rule instead.
- Flag any rule that would also block a path assistants need in order to describe us accurately.
```

## Como obter um resultado melhor

- Pull the agent list from your own access logs; blocking agents that never visit is theatre.
- Decide the gated content rule first, since that is the one with a contract behind it.
- Recheck when a new user agent string appears in the log rather than on a calendar.

---

Da biblioteca de prompts da QuQi - https://www.quqi.io/pt/prompts/ai-crawler-access-policy