Robots.txt Generator
Generate a valid robots.txt file visually. Add user-agents, control allow/disallow rules, set crawl delays, and specify your sitemap URL — no manual editing required.
What is Robots.txt Generator?
The robots.txt file sits at the root of your domain and tells search engine crawlers which pages they can and cannot access. It's one of the first files a bot checks when it visits your site.
A well-configured robots.txt prevents search engines from wasting crawl budget on admin pages, staging paths, duplicate content, and internal search results — directing them toward the content that actually matters.
This tool lets you build a robots.txt file visually without having to memorise the syntax. Add as many user-agent / rule blocks as you need, preview the output in real time, and copy it with one click.
How to use
Frequently asked questions
It's a convention, not a technical barrier. Well-behaved bots like Googlebot respect it; malicious scrapers may ignore it entirely.
No — disallowing a URL only stops Google from crawling it. If other pages link to it, Google may still index it. Use the noindex meta tag to prevent indexing.
It must be at the root of your domain: https://example.com/robots.txt. Subdirectory placement does not work.
Yes — these paths should be blocked from crawlers. They provide no SEO value and waste crawl budget.
A single slash blocks the entire site from the specified user agent. Only use this on staging environments, never on production.