WordPress Robots.txt: Quick Fixes for Better SEO

Having a correctly configured robots.txt file is essential for any website, especially from an SEO and performance perspective. It helps guide search engine crawlers to the most important content while keeping them away from pages that don’t need to be indexed, such as admin areas, filters, or internal search results. This not only prevents duplicate content and index bloat but also improves crawl efficiency — ensuring that search engines spend their time crawling and indexing the pages that matter most. For larger or dynamic sites like eCommerce stores, a well-optimised robots.txt can even help improve site speed and resource usage by reducing unnecessary crawl activity.

What is robots.txt file?

A robots.txt file is a small text file placed in the root directory of your website that tells search engine crawlers which pages or sections of your site they are allowed to crawl and index. It acts like a set of instructions or rules for bots, helping you control and manage how your website is accessed by search engines.

What Should You Include? (Beginner’s Guide)

If you’re running a WordPress website, your robots.txt file can help control what search engines like Google crawl and index. By setting it up properly, you can improve your SEO and reduce unnecessary crawling of parts of your site that don’t need to be seen.

What Does the Default WordPress Robots.txt Look Like?

Here’s what WordPress includes by default:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This setup is safe, but it’s not ideal. Let’s improve it.

Always Include Your Sitemap

Sitemaps help search engines find all the important pages on your site. You should add your sitemap URL like this:

Sitemap: https://yourwebsite.com/sitemap_index.xml

If you have more than one, list each one.

Don’t Block These Folders

In the past, people used to block these parts of WordPress:

/wp-includes/
/wp-content/plugins/
/wp-content/uploads/

Don’t do this. Here’s why:

• • • Google needs access to CSS and JavaScript files to understand how your site looks.
    • Blocking uploads could stop images and videos from showing in search results.

What About Staging Sites?

If you’re working on a staging version of your site (a private copy for testing), you don’t want it showing up in Google.

To block it completely, use:

User-agent: *
Disallow: /

Also, go to Settings > Reading in WordPress and check the box:
“Discourage search engines from indexing this site.”

Important: Don’t forget to undo this when your site goes live!

Clean Up Useless Paths

There are a few parts of WordPress that aren’t useful to search engines. You can block them to reduce crawl waste:

Disallow: /trackback/
Disallow: /comments/feed/
Disallow: */embed/
Disallow: /cgi-bin/
Disallow: /wp-login.php

Block Tag Pages and Search Results (Optional)

If your site has tag pages or internal search results, and they don’t offer much value, you can block them too:

Disallow: /tag/
Disallow: /page/
Disallow: /?s=

But if these pages are important for your SEO strategy, don’t block them.

What to block on an e-commerce website?

On an e-commerce website, it’s important to block low-value or sensitive pages from being crawled to keep your site efficient and focused. This includes cart, checkout, login, account pages, and internal search results. You should also block filtered URLs with parameters (like ?color=, ?size=, etc.) and duplicate paths such as tag pages or paginated URLs, unless they serve a clear SEO purpose. This helps prevent index bloat and ensures search engines focus on your product and category pages.

# Block private or low-value pages

Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /login/
Disallow: /admin/

Check How It’s Working

After you set up your robots.txt file:

• • • Use Google Search Console to check crawl stats.
    • Use the URL Inspection Tool to see if anything important is blocked.
    • Check your Sitemap to make sure only useful pages are listed.

WordPress doesn’t give you the best robots.txt file by default — but it’s easy to fix.

Spending just few minutes to update this file can:

• • • • Save your server from wasting resources
        
      • Help search engines focus on the right pages
        
      • Avoid SEO issues in the future

Take control of your crawl budget and give your SEO a small but powerful boost.

If you need technical SEO services, including help with your robots.txt file setup, feel free to reach out. We’re here to optimise your website’s performance, improve search engine rankings, and ensure your site runs smoothly for both users and search engines. Don’t hesitate to get in touch to discuss how we can boost your online presence together.