Place robots.txt at the root of your domain: https://yourdomain.com/robots.txt
How it works
Convert in three steps
1. Configure your crawl rules
Select which crawlers the rules apply to (all robots or specific ones like Googlebot), then set which paths to allow or disallow.
2. Add your sitemap URL
Enter your sitemap.xml URL to include the Sitemap: directive — this tells crawlers where to find your sitemap automatically.
3. Download robots.txt
Preview the final file and download it. Upload to your site root so it's accessible at yourdomain.com/robots.txt.
FAQ
Common questions
Robots.txt is a plain text file at your site root that tells web crawlers which pages or sections they’re allowed to access. It follows the Robots Exclusion Protocol — a convention that well-behaved crawlers like Googlebot respect. It controls crawl access, not indexing — a page blocked in robots.txt can still appear in search results if other pages link to it.
These control different things. A Disallow rule in robots.txt tells crawlers not to visit the page at all. A noindex meta tag tells crawlers they can visit but should not include the page in search results. If you block a page in robots.txt, Google may never see the noindex tag — so use noindex for pages you want deindexed, and Disallow for pages you don’t want crawled (like staging paths, internal search results, or large parameter-driven URL sets).
Common paths to disallow include /wp-admin/ (keep /wp-admin/admin-ajax.php allowed for WordPress), /search?, internal account pages like /cart/ and /checkout/, and any staging or development directories. Avoid disallowing CSS and JS files — Google needs to render your pages to assess them properly.
It affects any crawler that respects the Robots Exclusion Protocol — Google, Bing, and most reputable crawlers do. Malicious bots and scrapers may ignore it entirely. Robots.txt is a directive for compliant crawlers, not a security measure.
Keep going
Related tools
XML Sitemap Generator
Build the sitemap.xml you'll reference in your robots.txt.
Schema Markup Generator
Add structured data to the pages you're allowing crawlers to access.
Meta Title Generator
Optimize titles on pages you want indexed.
URL Slug Generator
Build clean URL paths before configuring crawl rules.
About the Robots.txt Generator
Robots.txt is one of those foundational site files that often gets set up once and forgotten — but getting it wrong can quietly block important pages from being crawled, or leave admin paths exposed to every bot on the web. This generator walks through the main configuration options: which crawlers the rules apply to, which paths to allow or disallow, and where your sitemap lives. The output is a clean, correctly formatted robots.txt file that follows the Robots Exclusion Protocol. Download it and upload to your site root — it should be the first file you set up on a new site and the first thing you check when something isn’t getting indexed.
Benefits
- Covers all key directives: User-agent, Allow, Disallow, and Sitemap.
- Target specific crawlers or apply rules to all bots at once.
- Includes sitemap reference directive — helps crawlers discover your sitemap automatically.
- Download-ready plain text file — upload directly to your site root without editing.
Privacy
All robots.txt generation runs in your browser. The paths and URLs you configure are processed locally and never transmitted to any server.