Robots.txt Generator
Build a clean robots.txt file to control how search engines crawl your site.
Paths that crawlers should be allowed to access (optional).
Paths you want to block from crawling (e.g. admin, thank-you pages).
Paste the full URL of your XML sitemap so crawlers can find it easily.
Click to upload or drag & drop
File content will fill the field above
Generating…
Your robots.txt
What is robots.txt?
robots.txt is a plain-text file placed at the root of your website (e.g. https://yoursite.com/robots.txt). It tells search-engine crawlers which parts of the site they may or may not request. It is not a security mechanism—malicious bots can ignore it—but legitimate crawlers such as Googlebot respect it.
Primary purpose
Manage crawl budget, hide non-public sections, and point crawlers to your sitemap.
Important limit
robots.txt prevents crawling, not indexing. A blocked URL can still appear in search if linked from elsewhere.
Where it lives
Must be at the domain root over HTTP/HTTPS. Subfolder robots.txt files are ignored.
How to use this tool
- List any paths you want explicitly allowed (rarely needed unless you also block broader paths).
- List paths to disallow—admin areas, cart, checkout, internal search results, staging folders, etc.
- Optionally paste your sitemap URL.
- Generate, then Copy or Download the file and upload it as
/robots.txt. - Test with Google Search Console’s robots.txt tester.
Best practices
- Never block CSS, JS or image assets that Google needs to render the page.
- Do not use robots.txt to hide private pages that contain sensitive data—use authentication instead.
- Keep the file small and readable; comments start with
#. - After changes, request a recrawl in Search Console if critical pages were previously blocked.
Frequently asked questions
Can I block an entire site?
Yes:
User-agent: * Disallow: /. Use this only for staging or development environments, never on a live site you want indexed.Does robots.txt stop Google from indexing a page?
No. To prevent indexing use a noindex meta tag or X-Robots-Tag header while allowing the page to be crawled.
How long until changes take effect?
Crawlers cache robots.txt for up to a day or more. You can request a fresh fetch in Google Search Console.