Robots.txt Generator
Build a robots.txt file with allow/disallow rules.
About this tool
Search crawlers read robots.txt before deep crawling. This free robots.txt generator helps you draft User-agent and Disallow/Allow rules. Configure paths locally, copy the file contents, and place them at your site root—then verify in Search Console.
How to use
- Choose user-agent scope (often * for all bots).
- Add Disallow or Allow paths for private or public sections.
- Include a Sitemap URL if you have one.
- Copy the generated robots.txt to your website root.
Frequently asked questions
No. It asks well-behaved crawlers not to fetch paths. URLs can still be linked or guessed—use auth for true privacy.