Quick-Start Presets
One click for Allow everything, Block everything, or a sensible WordPress setup. Tweak from there.
Robots.txt Generator
Create a robots.txt file in seconds. Set crawler rules, block AI bots with one click, add your sitemap, and copy the result. Free and fully in your browser.
Guide
How to use the robots.txt generator
1
Pick a Starting Point
Use a quick-start preset like Allow everything, Block everything, or WordPress, or build your own rules from scratch.
2
Set Your Rules
Add a user-agent, list the paths to disallow or allow, check any AI crawlers to block, and paste your sitemap URLs. The file updates as you type.
3
Copy or Download
Copy the robots.txt or download the file, then upload it to the root of your site so it loads at yoursite.com/robots.txt.
Control What Search Engines and Bots Can Crawl
Every search engine crawler looks for a robots.txt file before it visits your pages. That small text file is where you tell bots which parts of your site to crawl and which to leave alone, from admin folders to private sections you do not want in search.
Writing it by hand is easy to get wrong. A stray slash or a misplaced rule can block your whole site or open up pages you meant to hide. The format is strict, and one bad line can undo the rest.
This generator builds the file for you. Pick a preset or set your own rules, block the AI crawlers you do not want, add your sitemap, and the correct robots.txt appears as you go. Copy it, drop it at your site root, and you are done.
Benefits
Why Use Our Robots.txt Generator?
Block AI Crawlers
Tick a box to keep GPTBot, ClaudeBot, CCBot, Google-Extended, and other AI bots out of your content.
Multiple Rule Groups
Set different Allow and Disallow rules for each user-agent, with as many groups as you need.
Crawl-delay Support
Add a crawl-delay for bots that respect it, to ease load on busy servers.
Sitemap Lines
List one or more sitemap URLs and they are added to the file in the right format.
Live Preview
See the finished robots.txt update as you type, then copy or download it in one click.
Explore More
Similar Tools
More SEO tools you might find useful.
FAQ
Frequently Asked Questions
Common questions about robots.txt and crawler control.
What is a robots.txt file?
A robots.txt file sits at the root of your site and tells search engine crawlers which pages or folders they can and cannot access. It is the first file most bots check before crawling.
Where do I put the robots.txt file?
Upload it to the root of your domain so it loads at yoursite.com/robots.txt. It will not work from a subfolder.
Does robots.txt stop a page from being indexed?
Not on its own. Disallow blocks crawling, but a blocked page can still be indexed if other sites link to it. To keep a page out of search, use a noindex tag instead.
Can I block AI crawlers with robots.txt?
Yes, for bots that honor it. Checking the AI crawler boxes adds a Disallow-all rule for each one. Well-behaved bots follow it, though it cannot force compliance.
What does an empty Disallow mean?
An empty Disallow line means nothing is blocked, so the crawler is allowed everywhere. That is how you allow full access for a user-agent.
Is this robots.txt generator free?
Yes, it is completely free with no signup and no limits. It runs entirely in your browser, so nothing is uploaded or stored.