Google needs to crawl your website so it can index it and ultimately rank it to show it to visitors when they search for relevant keywords. However, it can encounter errors or issues that make this process impossible to complete. Every website owner should know what these errors are, and fortunately, the Google Search Console provides helpful information on the errors it encounters. In this article, we will go through the most common errors displayed within the tool’s coverage reports and how to fix them.
Sitemap “Couldn’t Fetch” Error
One of the steps you need to complete to get Google to crawl and index your website is adding a sitemap to your Google Search Console. The sitemap contains all links on your website, including posts, pages, attachments, and different media. Google’s crawlers use it to navigate your website to discover relevant content and the relationship between it as indicated by your internal links.
You can create the sitemap yourself, but it is much easier to use the various tools available to do so. Regardless of the method you use, you should ensure that it is accurate and accessible.
Your sitemap is likely located at https://yoursite.com/sitemap.xml. In almost all cases, Google will find your sitemap easily and start crawling your pages.
However, it can encounter a fetch error. There are several reasons for this.
First, the sitemap might not be using the standard /sitemap.xml naming convention. This means the crawler will not find the sitemap where you told it to look. You can verify its name location by checking using a browser and through your web host.
Checking with the browser entails trying different naming conventions. Some web hosts use a format that looks like yourwebsite.com/sitemap_index.xml, which you will typically see if you have more than one sitemap, or yourwebsite.com/sitemep1.url. Try this in your browser to see what works. If not, turn to your website host.
Your web hosting provider can direct you to your file manager within your hosting account to find your sitemap. Once there, ensure you are in the same folder where your website resides and look for the sitemap there. Find it and then add it to your Google Search Console account.
5XX Server Errors
The Googlebot crawler should be able to access your website to crawl it. However, it may encounter a server error that the Search Console presents as a 5XX error. There are three main types of 5XX errors to know about:
- 500 or internal server errors – These occur when a technical issue causes a delay in processing requests. There could be several reasons for this error, including coding errors, an issue with your CMS and the themes and plugins you are using, or a different reason.
- 502 or bad gateway errors – These errors indicate the request could not be completed due to an error in an upstream connection. That could be on the same machine or a separate one. You are more likely to see this issue with a badly configured content management system installation.
- 503 or service unavailable errors – These errors indicate the server is down entirely, down temporarily, or is too busy to process requests. All these issues lead to the Googlebot waiting too long for the request and presenting the error.
There are many fixes for these issues depending on the underlying cause. In many cases, waiting a bit or reloading the page might help. You can also talk to your hosting provider or IT team so they can find out what is going on and fix it.
If all else fails, consider moving to a different provider. This might be the only option if you are consistently seeing 503 errors that indicate the issue is not on your end but the server’s.
Redirect Errors
There are legitimate reasons you may want to redirect a page, such as when switching to a new URL. However, this can cause several issues that present as redirect errors within Google Search Console. These errors include:
- Redirect loops: This happens when one URL redirects to another that redirects to the first one, causing a loop. This happens a lot with HTTP and HTTPS redirects.
- Empty or bad URLs in the chain – If any URL in the chain fails, the Googlebot will show an error. You might see a 404 error or blank URL when the page does not exist.
- The redirected URL exceeded the character limit – URLs have a limit on the number of characters they can include. If they exceed this limit, Googlebot will not crawl them, leading to errors.
To fix these errors, you should find the faulty URL chain, and the originating and destination URLs. Next, know where each is redirecting to eliminate redirect loops and bad or empty redirect URLs. Also, only use a maximum of two URLs for redirects, the originating and destination URLs.
The Submitted URL Was Blocked by Robots.txt
The robots.txt file directs bots and crawlers on how to navigate your website. It also includes a line that tells the Googlebot whether it should crawl your website. If you have not allowed this, the bot will not be able to crawl your website, and you will end up with this error. You might also have a misconfigured robots.txt file, which happens a lot on WordPress when website owners use certain SEO plugins.
The first way to fix the issue is to download the file and search for the blocked URL; Google Search Console should provide it. If you cannot find the whole URL, search for its snippets. Once you find it, remove it and reupload the file.
Sometimes, you do not want this URL to be crawled but have included it in your sitemap.xml file. Search for the URL within the file, delete it, and reupload the file.
Google Search Console is a fantastic tool for helping website owners know of any issues causing their pages not to be crawled and indexed. It does this through various errors, some more technical than others. Fortunately, it also provides helpful additional information that makes it easy to find and rectify the indicated issues.