skip to Main Content

Suddenly increased a lot of Server Error in Crawl Errors

Maybe you received such a message in the search console of the webmaster tools:

Googlebot found an increase in server errors on http://www.domain.com/
Googlebot has identified that an increased number of URLs on http://www.domain.com/ return a server error HTTP 5xx. As a result, users might not be able to see your content at all, and Google might not be able to Index and show your pages in search results.

Looking on Webmaster Tools, there seems to be a lot of crawl errors that have arisen over the last few days. All of them are from the WordPress wp-includes folder.

The cause is automatic indexes on folders on your server. For example http://www.domain.com/wp-includes/ shows an index of all the files in the folder.

The solution is to turn of indexes

Add this to the top of .htaccess

Options -Indexes

If you can’t do this then tell your host to disable indexes

In addition, you can add the following line to robots.txt


User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Then go to Crawl Errors and select these links, click the Mark as fixed button.

It may take a few days for google to update these results.

Crawl Errors Removed

Leave a Reply

Back To Top