QuQi
SEO

Log file crawl budget analysis

Use when server logs are available and you want to know where crawl budget is going.

log-file-crawl-analysis.md
Download .md
You are analysing server log data for crawl behaviour.

Site: {{SITE_URL}}
Log summary (URL or directory, bot hits, status codes, date range): {{LOG_SUMMARY}}
Site structure and which sections make money: {{SITE_STRUCTURE}}
Total indexable URL count: {{INDEXABLE_COUNT}}

Output:
1. A crawl distribution table: Section or pattern | Bot hits | Share of total crawl | Share of indexable URLs | Commercial value (High / Medium / Low) | Verdict (Under-crawled / Balanced / Wasted).
2. "Crawl waste" - the patterns absorbing crawl with no value, ordered by hits, each with the mechanism (parameters, faceted navigation, pagination, redirect chains, soft 404s, calendars, session ids).
3. "Never crawled but important" - value sections with low or zero bot hits, with the likely reason.
4. A fix list: Action | Method (robots, noindex, canonical, parameter handling, internal linking, removal) | Expected effect on crawl distribution.

Constraints:
- Do not confuse low crawl with low indexation. Say which the data actually shows.
- Verify bot identity is claimed in the data before treating hits as genuine search crawler traffic; if verification status is unknown, say so.
- Do not recommend blocking a pattern in robots.txt if the pages need to be de-indexed first. Explain the ordering.

Fill in before running

Replace each placeholder with your own detail. The more specific you are, the less the model invents.

  • {{SITE_URL}}
  • {{LOG_SUMMARY}}
  • {{SITE_STRUCTURE}}
  • {{INDEXABLE_COUNT}}

Getting a better result

  1. Aggregate by directory or pattern before pasting - raw log lines waste the context window.
  2. Use at least 30 days of logs so weekly crawl cycles do not distort the picture.
  3. Pair this with an indexation report to see which under-crawled sections are actually missing.