QuQi
RECHERCHE IA (GEO)

Decide which AI crawlers to allow

Use when you need a defensible position on which AI crawlers may fetch your site.

ai-crawler-access-policy.md
Télécharger le .md
You are advising on crawler access policy. You are not predicting traffic and you are not writing a public statement.

Site and what it publishes: {{SITE_AND_CONTENT}}
User agents seen in server logs, with hit counts: {{AGENT_LOG_SUMMARY}}
Our commercial position on assistants using our content: {{BUSINESS_STANCE}}
Content that is licensed, gated or sold: {{RESTRICTED_CONTENT}}

Produce four sections.

1. A table: | User agent | Hits in {{AGENT_LOG_SUMMARY}} | Paths it concentrates on | Recommended rule (allow / allow with path limits / disallow) | What allowing costs us | What blocking costs us |
2. A robots.txt block written out in full, with one comment line above each group giving the reason in plain terms.
3. The paths from {{RESTRICTED_CONTENT}} that should be disallowed for every agent, and whether each also needs server-side enforcement.
4. Which decisions in {{BUSINESS_STANCE}} are reversible within days and which are not, with the reason.

Constraints:
- Judge every agent from {{AGENT_LOG_SUMMARY}} and {{SITE_AND_CONTENT}} only. Do not add agents from memory or state what a named agent does with what it fetches.
- Do not present robots.txt as enforcement. Name the agents in the log that would need rate limiting or a firewall rule instead.
- Flag any rule that would also block a path assistants need in order to describe us accurately.

À remplir avant de lancer

Remplacez chaque espace réservé par vos propres détails. Plus vous êtes précis, moins le modèle invente.

  • {{SITE_AND_CONTENT}}
  • {{AGENT_LOG_SUMMARY}}
  • {{BUSINESS_STANCE}}
  • {{RESTRICTED_CONTENT}}

Obtenir un meilleur résultat

  1. Pull the agent list from your own access logs; blocking agents that never visit is theatre.
  2. Decide the gated content rule first, since that is the one with a contract behind it.
  3. Recheck when a new user agent string appears in the log rather than on a calendar.