# Log file crawl budget analysis

> Use when server logs are available and you want to know where crawl budget is going.

## Fill in before running

- `{{SITE_URL}}`
- `{{LOG_SUMMARY}}`
- `{{SITE_STRUCTURE}}`
- `{{INDEXABLE_COUNT}}`

## Prompt

```
You are analysing server log data for crawl behaviour.

Site: {{SITE_URL}}
Log summary (URL or directory, bot hits, status codes, date range): {{LOG_SUMMARY}}
Site structure and which sections make money: {{SITE_STRUCTURE}}
Total indexable URL count: {{INDEXABLE_COUNT}}

Output:
1. A crawl distribution table: Section or pattern | Bot hits | Share of total crawl | Share of indexable URLs | Commercial value (High / Medium / Low) | Verdict (Under-crawled / Balanced / Wasted).
2. "Crawl waste" - the patterns absorbing crawl with no value, ordered by hits, each with the mechanism (parameters, faceted navigation, pagination, redirect chains, soft 404s, calendars, session ids).
3. "Never crawled but important" - value sections with low or zero bot hits, with the likely reason.
4. A fix list: Action | Method (robots, noindex, canonical, parameter handling, internal linking, removal) | Expected effect on crawl distribution.

Constraints:
- Do not confuse low crawl with low indexation. Say which the data actually shows.
- Verify bot identity is claimed in the data before treating hits as genuine search crawler traffic; if verification status is unknown, say so.
- Do not recommend blocking a pattern in robots.txt if the pages need to be de-indexed first. Explain the ordering.
```

## Getting a better result

- Aggregate by directory or pattern before pasting - raw log lines waste the context window.
- Use at least 30 days of logs so weekly crawl cycles do not distort the picture.
- Pair this with an indexation report to see which under-crawled sections are actually missing.

---

From the QuQi prompt library - https://www.quqi.io/prompts/log-file-crawl-analysis