Robots.txt Manager

Customize your site’s robots.txt file directly from the WordPress admin panel. Control which parts of your site search engines can crawl, add sitemap references, and manage crawler behavior—all without FTP access or file editing.

Use Cases#

• Block search engines from crawling admin, login, and private areas
• Add your XML sitemap location for better search engine discovery
• Control crawl rate to prevent server overload from aggressive bots
• Block specific user agents (bad bots, scrapers)
• Manage staging site visibility during development

Where to Find It#

Configure your robots.txt content in the Switchboard module settings. Your changes take effect immediately at yoursite.com/robots.txt.

How It Works#

1. Open the Robots.txt Manager module settings
2. Edit the robots.txt content in the textarea
3. Save your changes
4. WordPress serves your custom content at /robots.txt

The module overrides WordPress’s default virtual robots.txt generation with your custom content.

Basic Syntax#

Robots.txt uses a simple directive format:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap.xml

User-agent#

Specifies which crawler the rules apply to.

Disallow#

Tells crawlers not to access a path.

Allow#

Explicitly allows access (useful for exceptions within disallowed areas).

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap#

Points crawlers to your XML sitemap for better indexing.

Sitemap: https://yoursite.com/sitemap.xml

Common Configurations#

Standard WordPress Setup#

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/

Sitemap: https://yoursite.com/sitemap.xml

With Feed and Trackback Blocking#

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /feed/
Disallow: /trackback/
Disallow: /comments/feed/

Sitemap: https://yoursite.com/sitemap.xml

Block Specific Directories#

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /private/
Disallow: /members-only/
Disallow: /staging/
Disallow: /test/

Sitemap: https://yoursite.com/sitemap.xml

Block Specific Bots#

# Allow all normal crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Block bad bots
User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: MJ12bot
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml

Staging Site (Block Everything)#

User-agent: *
Disallow: /

Sitemap Reference#

Always include your sitemap location:

Sitemap: https://yoursite.com/sitemap.xml

If using multiple sitemaps:

Sitemap: https://yoursite.com/post-sitemap.xml
Sitemap: https://yoursite.com/page-sitemap.xml
Sitemap: https://yoursite.com/category-sitemap.xml

This helps search engines discover and index your content efficiently.

Testing Your Robots.txt#

View Live File#

Visit yoursite.com/robots.txt in your browser to see the active content.

Google’s Testing Tool#

1. Go to Google Search Console
2. Select your property
3. Use the robots.txt tester to check for issues

Test URL Blocking#

In Search Console’s robots.txt tester:

1. Enter a URL path
2. See if it’s blocked or allowed
3. Identify which rule affects it

Important Considerations#

It’s a Suggestion, Not Security#

Robots.txt is a convention that well-behaved crawlers follow. It does not:

• Prevent access to files
• Hide content from determined visitors
• Provide security or privacy
• Block malicious bots that ignore rules

For actual access control, use password protection or login requirements.

Don’t Block Important Resources#

Avoid blocking CSS, JavaScript, or images that crawlers need to render your pages:

# Bad - may hurt SEO
Disallow: /wp-content/

Search engines need to access these files to properly understand and rank your content.

Check for Physical File#

If you have a physical robots.txt file in your WordPress root directory, it takes precedence over this virtual file. Delete the physical file to use this module’s content.

FAQ#