How to Fix Crawl Errors in Google Search Console (Step-by-Step)

Crawl errors are one of the most common — and most misunderstood — issues in technical SEO. If Googlebot can’t reach your pages, those pages can’t rank. It’s that simple. Yet we regularly see site owners ignore the “Pages” report in Google Search Console for months, letting fixable errors silently drain their organic visibility. In this guide, we’ll walk you through every major type of crawl error, show you exactly where to find them in Google Search Console, and give you a priority-based system to fix crawl errors in Search Console efficiently — starting with the ones that actually hurt your rankings.

What Are Crawl Errors — and Why Do They Matter?

A crawl error occurs when Googlebot tries to access a URL on your site and fails. Instead of indexing the page and showing it in search results, Google logs the failure and moves on. A handful of crawl errors on a large site is normal. But when errors pile up, they signal bigger problems:

• Wasted crawl budget. Google allocates a limited number of requests per crawl session. Every failed request is a wasted opportunity for a real page to get crawled.
• Lost rankings. If a page that previously ranked returns a server error, Google will eventually drop it from the index.
• Poor user experience. Visitors who land on broken pages bounce immediately, which can indirectly harm your SEO performance.

According to a 2023 study by Semrush, broken internal links and crawl errors appear on over 42% of websites audited — making them one of the most prevalent technical SEO issues on the web.

Types of Google Crawl Errors (and What Causes Each)

Before you can fix crawl errors, you need to understand what you’re looking at. Here are the most common types you’ll encounter in Google Search Console.

404 Errors (Not Found)

A 404 means the URL doesn’t exist on your server. Googlebot requested the page and got nothing back. Common causes: - A page was deleted without setting up a redirect - A URL was changed (e.g., slug updated) without redirecting the old URL - A typo in an internal link or sitemap entry - External sites linking to a URL that never existed or has since been removed Impact: If the deleted page had backlinks, rankings, or traffic, a 404 throws all of that away. If the page was never important, the 404 is mostly cosmetic.

Soft 404 Errors

A soft 404 is when a page returns a 200 (OK) status code but the content is essentially an empty or “not found” page. Google is smart enough to detect this mismatch. Common causes: - A CMS that serves a blank template for missing pages instead of a proper 404 response - Search or filter pages that return zero results - Product pages where the item is out of stock and the page shows no meaningful content Impact: Soft 404s confuse Google and waste crawl budget because the bot keeps re-crawling pages it can never actually index.

5xx Server Errors

A 5xx error (500, 502, 503, etc.) means your server failed to respond. This is almost always a server-side problem, not a content problem. Common causes: - Server overload or insufficient resources - Broken server-side scripts (PHP errors, database timeouts) - Hosting downtime or maintenance - Misconfigured .htaccess or server rules Impact: These are the most urgent crawl errors. If Googlebot hits repeated 5xx errors, it will slow its crawl rate and may eventually deindex affected pages.

Redirect Errors

Redirect errors occur when Googlebot follows a redirect chain that fails — either because it loops, chains too many times, or leads to a dead end. Common causes: - Redirect loops (Page A → Page B → Page A) - Long redirect chains (more than 3-4 hops) - Redirects pointing to a page that itself returns a 404 or 5xx - Mixed HTTP/HTTPS redirect issues Impact: Google will stop following the redirect after a few hops. The original page won’t get indexed, and any link equity in the chain gets diluted or lost.

Blocked by robots.txt

This isn’t technically an “error” — it means your robots.txt file is telling Googlebot not to crawl the URL. But if the page appears in your sitemap or has inbound links, Google flags the conflict. Common causes: - Overly broad Disallow rules in robots.txt - Leftover rules from a staging or development environment - Blocking entire directories that contain pages you actually want indexed Impact: Blocked pages can’t be crawled or indexed. If you’re blocking important pages by accident, it’s a silent ranking killer. Check our robots.txt guide for best practices on writing rules that don’t backfire.

How to Find Crawl Errors in Google Search Console

Google restructured Search Console in recent years. The old “Crawl Errors” report is gone. Here’s where to find everything now:

1. Log in to Google Search Console and select your property.
2. Go to “Pages” (under the Indexing section in the left sidebar). This replaced the old “Coverage” report.
3. Click on “Not indexed” to see all pages Google tried to crawl but couldn’t or chose not to index.
4. Review the specific error categories. Google groups errors by type — “Not found (404),” “Server error (5xx),” “Redirect error,” “Blocked by robots.txt,” and more.
5. Click into each category to see the list of affected URLs.

Pro tip: Also check the “Crawl Stats” report under Settings → Crawl Stats. This shows you Googlebot’s crawl activity, response times, and any host-level issues. If your average response time is spiking, that often correlates with 5xx errors. For a deeper analysis beyond what GSC shows, a full SEO audit using tools like Screaming Frog or Sitebulb can uncover crawl errors that Google hasn’t reported yet — including orphan pages and links buried deep in your site architecture.

Which Crawl Errors to Fix First: A Priority System

Not all crawl errors deserve equal attention. Here’s how we prioritize them when working with clients:

Priority	Error Type	Why It’s Urgent
🔴 Critical	5xx Server Errors	Signals your server is failing — affects entire site health
🔴 Critical	Soft 404s on important pages	Pages are live but not being indexed
🟠 High	Redirect errors	Link equity and traffic are being lost
🟠 High	404s on pages with backlinks or traffic	Rankings and authority are being wasted
🟡 Medium	Blocked by robots.txt (unintentional)	Pages you want indexed can’t be crawled
🟢 Low	404s on intentionally deleted pages	Cosmetic — no real SEO impact

Rule of thumb: If the broken URL has backlinks, organic traffic, or is linked to from your sitemap, fix it immediately. If it’s a page you intentionally removed and nobody links to it, you can safely leave it or serve a 410.

How to Fix Each Type of Crawl Error

Fixing 404 Errors

If the content moved to a new URL: Set up a 301 (permanent) redirect from the old URL to the new one. In Apache, add this to your .htaccess: Redirect 301 /old-page/ /new-page/ In Nginx: rewrite ^/old-page/$ /new-page/ permanent; If the content was intentionally removed: Return a 410 (Gone) status code instead of a 404. A 410 tells Google the page is permanently gone, and the bot will stop re-crawling it sooner. If the page never existed: Check where the bad URL is being linked from. Fix the source — update the internal link, correct the sitemap entry, or reach out to external sites linking to the wrong URL.

Fixing Soft 404 Errors

• Add real content to thin pages, or
• Return a proper 404 or 410 status code if the page should be removed, or
• Use a noindex tag if you want to keep the page live but out of Google’s index

Fixing 5xx Server Errors

These require server-side investigation:

1. Check your server error logs for the specific error messages.
2. Test whether the issue is intermittent (server overload) or persistent (broken code).
3. Upgrade hosting resources if the problem is capacity-related.
4. Fix broken scripts, database connections, or misconfigured server rules.
5. If your site uses caching, flush the cache and test again.

Fixing Redirect Errors

1. Map out the full redirect chain using a tool like Redirect Checker.
2. Replace chains with a single 301 from the original URL directly to the final destination.
3. Remove any loops by identifying the circular reference and breaking it.
4. Ensure all redirects land on a page that returns a 200 status code.

Fixing “Blocked by robots.txt”

1. Open your robots.txt file (yourdomain.com/robots.txt).
2. Identify the Disallow rule that’s blocking the URL.
3. Remove or narrow the rule so it no longer blocks pages you want indexed.
4. Use Google’s txt Tester in Search Console to verify your changes before deploying.

For a detailed walkthrough on writing effective robots.txt rules, see our robots.txt guide.

How to Validate Fixes and Request Re-Crawling

After fixing a crawl error, don’t just wait and hope Google notices. Here’s how to speed things up:

1. Go to the Pages report in Google Search Console.
2. Click into the error category you fixed.
3. Click “Validate Fix.” Google will re-crawl a sample of the affected URLs over the following days.
4. Use the URL Inspection tool for individual URLs — paste the URL, click “Request Indexing,” and Google will prioritize re-crawling it.

Important: Validation can take up to two weeks. Don’t re-submit multiple times — one request is enough. Also, make sure your XML sitemap is up to date. Remove any URLs that now return 404 or 410 status codes, and ensure all live, indexable pages are included. A clean sitemap helps Google crawl your site more efficiently.

Preventing Crawl Errors Before They Happen

The best crawl error is one that never occurs. Here are preventive measures we recommend:

• Set up redirects before deleting or moving pages. Make this part of your content workflow.
• Audit internal links regularly. Broken internal links are the #1 source of 404 errors.
• Monitor server performance. Use uptime monitoring tools to catch 5xx issues early.
• Keep your sitemap clean. Only include URLs that return 200 status codes.
• Test staging changes before pushing to production. Many crawl errors come from botched migrations or redesigns.

Running a quarterly SEO audit catches crawl errors before they snowball into ranking losses. It’s one of the highest-ROI activities in technical SEO.

Frequently Asked Questions