Use when server logs are available and you want to know where crawl budget is going.
log-file-crawl-analysis.md
You are analysing server log data for crawl behaviour.
Site: {{SITE_URL}}
Log summary (URL or directory, bot hits, status codes, date range): {{LOG_SUMMARY}}
Site structure and which sections make money: {{SITE_STRUCTURE}}
Total indexable URL count: {{INDEXABLE_COUNT}}
Output:
1. A crawl distribution table: Section or pattern | Bot hits | Share of total crawl | Share of indexable URLs | Commercial value (High / Medium / Low) | Verdict (Under-crawled / Balanced / Wasted).
2. "Crawl waste" - the patterns absorbing crawl with no value, ordered by hits, each with the mechanism (parameters, faceted navigation, pagination, redirect chains, soft 404s, calendars, session ids).
3. "Never crawled but important" - value sections with low or zero bot hits, with the likely reason.
4. A fix list: Action | Method (robots, noindex, canonical, parameter handling, internal linking, removal) | Expected effect on crawl distribution.
Constraints:
- Do not confuse low crawl with low indexation. Say which the data actually shows.
- Verify bot identity is claimed in the data before treating hits as genuine search crawler traffic; if verification status is unknown, say so.
- Do not recommend blocking a pattern in robots.txt if the pages need to be de-indexed first. Explain the ordering.
Replace each placeholder with your own detail. The more specific you are, the less the model invents.