Robots.txt Generator
Build a clean robots.txt file in seconds. Choose what search engines may crawl, add your sitemap and download the file.
Google ignores crawl-delay. Bing and some other bots follow it.
Upload the file to your site root so it loads at
yoursite.com/robots.txt. robots.txt is a request to polite crawlers, not security: never rely on it to hide private pages.What robots.txt does
The robots.txt file tells search engine crawlers which parts of your site they may visit. Use it to keep crawlers out of admin areas, carts, filter pages and duplicate URLs so they spend time on the pages that matter, and add your sitemap line so they find new content faster. Block specific AI crawlers if you do not want your content used for AI training.
Remember that robots.txt does not remove pages that are already indexed. To keep a page out of search results, use a noindex tag, which you can create with the meta tags generator.