البحث بالذكاء الاصطناعي (GEO)
Decide which AI crawlers to allow
Use when you need a defensible position on which AI crawlers may fetch your site.
ai-crawler-access-policy.md
You are advising on crawler access policy. You are not predicting traffic and you are not writing a public statement.
Site and what it publishes: {{SITE_AND_CONTENT}}
User agents seen in server logs, with hit counts: {{AGENT_LOG_SUMMARY}}
Our commercial position on assistants using our content: {{BUSINESS_STANCE}}
Content that is licensed, gated or sold: {{RESTRICTED_CONTENT}}
Produce four sections.
1. A table: | User agent | Hits in {{AGENT_LOG_SUMMARY}} | Paths it concentrates on | Recommended rule (allow / allow with path limits / disallow) | What allowing costs us | What blocking costs us |
2. A robots.txt block written out in full, with one comment line above each group giving the reason in plain terms.
3. The paths from {{RESTRICTED_CONTENT}} that should be disallowed for every agent, and whether each also needs server-side enforcement.
4. Which decisions in {{BUSINESS_STANCE}} are reversible within days and which are not, with the reason.
Constraints:
- Judge every agent from {{AGENT_LOG_SUMMARY}} and {{SITE_AND_CONTENT}} only. Do not add agents from memory or state what a named agent does with what it fetches.
- Do not present robots.txt as enforcement. Name the agents in the log that would need rate limiting or a firewall rule instead.
- Flag any rule that would also block a path assistants need in order to describe us accurately.
املأها قبل التشغيل
استبدل كل فراغ بتفصيلة من عندك. كلما كنت أدقّ، قلّ ما يخترعه النموذج.
- {{SITE_AND_CONTENT}}
- {{AGENT_LOG_SUMMARY}}
- {{BUSINESS_STANCE}}
- {{RESTRICTED_CONTENT}}
كيف تحصل على نتيجة أفضل
-
1
Pull the agent list from your own access logs; blocking agents that never visit is theatre.
-
2
Decide the gated content rule first, since that is the one with a contract behind it.
-
3
Recheck when a new user agent string appears in the log rather than on a calendar.