Search engines allocate a finite amount of crawling to any site. On a catalogue or a site with faceted navigation, that budget is routinely spent on near-duplicate URLs while the pages you care about wait. This skill finds where the budget is going and reclaims it.
WHEN TO REACH FOR THIS
Use when a large site has pages that stay unindexed for weeks, or when log files show crawlers spending their time on parameters, filters and pagination instead of the pages that matter.
---
name: crawl-budget-audit
description: Use when a large site has pages that stay unindexed for weeks, or when log files show crawlers spending their time on parameters, filters and pagination instead of the pages that matter.
---
# Crawl budget audit
Search engines allocate a finite amount of crawling to any site. On a catalogue or a site with faceted navigation, that budget is routinely spent on near-duplicate URLs while the pages you care about wait. This skill finds where the budget is going and reclaims it.
## What you need first
- Server access logs for at least 14 days, or Search Console crawl stats
- A full URL inventory (sitemap or crawl export)
- robots.txt
## Method
1. Pull crawl hits per URL pattern from the logs, grouping by path segment and query parameter rather than by individual URL.
2. Rank the patterns by share of total crawl hits. Anything above 5% that is not a page you want ranked is a leak.
3. For each leak, decide the correct instrument: robots.txt Disallow for URLs that must never be fetched, canonical for duplicates that must stay reachable, noindex for pages that must be crawled but not listed, and parameter rules only where the first three do not fit.
4. Check for crawl traps: infinite calendars, session IDs in URLs, sort orders that generate unbounded combinations.
5. Compare crawl frequency against last-modified dates. Pages that change often but are crawled rarely need better internal linking, not a directive.
6. Re-measure after 14 days and confirm the reclaimed share landed on the pages you intended.
## What this produces
A ranked table of crawl leaks with the specific directive to apply to each, plus a before/after share of crawl spent on revenue pages.
## Where this goes wrong
- Blocking a URL in robots.txt that already carries links - it keeps its equity but can no longer pass it on
- Using noindex on pages you also blocked from crawling, so the noindex is never seen
- Treating crawl budget as a problem on a 200-page site, where it almost never is
---
From the QuQi skill library - https://www.quqi.io/skills/crawl-budget-audit
Free to download · no account, no email
A ranked table of crawl leaks with the specific directive to apply to each, plus a before/after share of crawl spent on revenue pages.