Advanced Robots.txt Generator

Create customized robots.txt files to control search engine crawlers and improve your website's SEO

Generator
Instructions
Examples

Basic Configuration

Enter your website's primary domain (including https://)

Recommended: 5-10 seconds to prevent server overload

Comma-separated list of URL parameters to block

User Agent Rules

Define specific rules for different search engine crawlers

All User Agents (*)

Sitemap Configuration

Add your XML sitemap URLs to help search engines discover your content

Advanced Options

Example: Request-rate: 1/10s # 1 page every 10 seconds

What is a robots.txt file?

A robots.txt file is a text file that tells search engine crawlers which pages or files they can or cannot request from your site. It's used primarily to manage crawler traffic to your site.

How to use this generator:

  1. Enter your primary domain (including https://)
  2. Set a crawl delay if needed (recommended for large sites)
  3. Add disallowed URL parameters (like tracking parameters)
  4. Configure rules for different user agents (search engine crawlers)
  5. Add your sitemap URLs
  6. Set any advanced options or custom directives
  7. Click "Generate Robots.txt"
  8. Copy the generated code and upload it to your site's root directory

Best Practices:

  • Always place your robots.txt file in your root directory (e.g., https://www.example.com/robots.txt)
  • Keep the file under 500KB
  • Use specific rules rather than blocking everything
  • Don't use robots.txt to hide sensitive information (use authentication instead)
  • Regularly test your robots.txt file with Google Search Console

Basic Example:

User-agent: *
Disallow: /private/
Disallow: /tmp/
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml

E-commerce Site Example:

User-agent: *
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /search?
Allow: /search?q=*
Crawl-delay: 10
Clean-param: ref /products/
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-products.xml

Media Site Example:

User-agent: *
Disallow: /uploads/temp/
Disallow: /admin/
Disallow: /config/
Allow: /media/
Crawl-delay: 5

User-agent: Googlebot-Image
Allow: /images/
Disallow: /images/private/

User-agent: Bingbot
Crawl-delay: 15

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-videos.xml