Robots.txt Generator

Generate robots.txt rules to manage search engine crawler access.

Important SEO Rule: robots.txt controls crawling, not indexing. If a URL is disallowed in robots.txt but other pages link to it, Google may still index the bare URL without content! To exclude a page from Google, allow the crawler and serve <meta name="robots" content="noindex"> in HTML.

Crawler Directives

Group #1

User-agent

Crawl Delay (sec)

Disallow Paths

Disallow:

Group #2

User-agent

Crawl Delay (sec)

Disallow Paths

Disallow:

Sitemap URL

robots.txt Preview

# robots.txt generated by AltHunt Free SEO Tools
# Notice: robots.txt blocks CRAWLING, not INDEXING.

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard

User-agent: GPTBot
Allow: /
Disallow: /private/

Sitemap: https://althunt.io/sitemap.xml

Key Capabilities

5 capabilities

#	Capability
01	Configure rules for Google, Bing, and AI crawlers (GPTBot, ClaudeBot, etc.)
02	Add Allow and Disallow paths with live validation
03	Specify sitemap XML locations
04	Crawl delay parameter support
05	One-click Copy & Download robots.txt

How to Use the Robots.txt Generator

Step 01

Select Crawlers

Choose default crawlers (*) or specific search and AI bots.

Step 02

Define Directives

Add allowed directories and disallowed private paths (e.g., /admin, /api).

Step 03

Export robots.txt

Add your sitemap URL and download your production-ready robots.txt file.

Frequently Asked Questions

Does robots.txt stop Google from indexing a page?

No! robots.txt controls CRAWLING, not INDEXING. If other pages link to a disallowed path, Google may still index the bare URL without content. To prevent indexing, allow crawling and use a <meta name='robots' content='noindex'> tag.

Where should the robots.txt file be uploaded?

It must always reside in the top-level root of your domain: https://example.com/robots.txt.

Related Free SEO Utilities

Browse all 20 tools →

Technical SEO