Use when you need someone to read a robots.txt and sitemap setup line by line for mistakes.
You are reviewing crawl directives for errors and unintended consequences.
robots.txt contents: {{ROBOTS_TXT}}
Sitemap index and child sitemap structure: {{SITEMAP_STRUCTURE}}
Sample of URLs in the sitemaps: {{SITEMAP_SAMPLE}}
URL patterns that should NOT be indexed: {{EXCLUDE_PATTERNS}}
Site size (approximate URL count): {{SITE_SIZE}}
Output three sections.
A. robots.txt line review - a table: Line | What it does | Verdict (Correct / Risky / Broken) | Consequence if left | Suggested replacement.
B. Sitemap review - check for: URLs that are blocked in robots.txt, non-canonical URLs, non-200 URLs implied by the patterns, redirect chains, missing lastmod, files over the 50,000 URL or 50MB limit, and whether the structure allows useful indexation reporting.
C. Recommended robots.txt, rewritten in full, with a comment line above each rule explaining it.
Rules:
- Flag any Disallow that would also block a resource needed for rendering.
- State clearly where a Disallow has been used when noindex was the correct tool, and explain the difference in one sentence.
- Do not assume URLs exist that are not shown. If a check needs data you do not have, list it under "Cannot verify from what was supplied".
Replace each placeholder with your own detail. The more specific you are, the less the model invents.