QuQi

Robots.txt Generator

Write crawl rules search engines actually follow, with your sitemap wired in.

Instant No signup

  

robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it - use a noindex meta tag to keep a page out.

OR STOP DOING IT BY HAND

This checks one page, once. The agents check every page, every week, and fix what they find.

Add your site - free

More free tools