Why a Website Is Not Indexed by Google: Causes

A page has been published, opens normally in a browser, appears in sitemap.xml, but is nowhere to be found in Google. This is a familiar situation — and it is often where chaotic SEO begins: resubmit the URL for indexing, add more text, check robots.txt, update the sitemap again, buy a few links.

The problem is that all of these actions address different causes.

Google may not have discovered the URL yet. It may have discovered it but not crawled it. It may have successfully loaded the page and decided not to add it to the index for now. It may consider another URL to be the primary canonical version. And sometimes the page is already indexed, but the user does not see it for the target query and mistakenly assumes there is an indexing problem.

That is why indexing diagnostics should not begin with robots.txt or the Request Indexing button.

The first question is simple:

At which stage has the page stopped?

The process can be simplified as:

Discovery → Crawling → Rendering → Indexing → Ranking.

If Google does not yet know the URL exists, rewriting the Title will not help. If Googlebot cannot access the page because of a 403 or 5xx response, adding another thousand words will not help either. And if the URL is already indexed, the problem is no longer indexing — it is ranking.

That is why proper website indexing analysis is about diagnosing the current state, not following a universal list of SEO recommendations.

What Is Website Indexing?

Website indexing is the process in which a search engine analyzes a page and may add information about it to its search index.

Before that happens, however, Google first needs to discover that the URL exists and then access the page.

This is where three different concepts are often confused:

crawling ≠ indexing ≠ ranking.

Google may discover a page without crawling it yet.

It may crawl the page but choose not to add it to the index.

It may index the page but not rank it highly for the target query.

So the fact that “I cannot see the page at the top of Google” does not mean the page is not indexed.

This distinction is fundamental. Otherwise, you may spend a long time trying to “fix indexing” for a page that is already in the index but simply lacks sufficient relevance or competitiveness for the target queries.

How to Check Whether a Page Is Really Not Indexed

The simplest mistake at this stage is diagnosing indexing only through ordinary Google Search.

Checking Through Google Search

For a quick check, you can use the operator:

site:domain.com

For a specific URL, you can search for the exact address or use a site: query.

This is useful for a quick overview, but the site: operator should not be treated as an exact indexing report. It does not replace Google Search Console and should not be the only basis for a technical conclusion.

URL Inspection in Google Search Console

For a specific page, the primary tool is URL Inspection.

It shows what Google knows about that exact URL and its current status.

It is especially useful to check:

  • whether the URL is indexed;

  • when it was last crawled;

  • whether crawling is allowed;

  • whether indexing is allowed;

  • which canonical the website declares;

  • which Google-selected canonical the search engine has chosen.

If you need to check the current version of the page rather than only data from the latest crawl, use Test Live URL.

This distinction is important. Indexed version data shows what Google already knows, while the live test helps determine whether the current page is accessible right now.

Page Indexing

When the issue affects dozens or hundreds of URLs rather than one page, the Page Indexing report is a better place to start.

It shows which groups of URLs Google is not indexing and which statuses are associated with them.

The logic is simple:

one specific page → URL Inspection;

large-scale problem → Page Indexing.

How a Page Gets Into Google

To interpret Search Console statuses correctly, it helps to understand the process itself.

Discovery

First, Google needs to learn that the URL exists.

Sources may include internal or external links, a sitemap, and other signals.

Crawling

Googlebot then attempts to retrieve the page.

At this stage, robots.txt, server issues, firewalls, authorization, redirect loops, or other technical barriers may prevent access.

Rendering

For modern websites, loading the HTML is not always enough. Google also needs to correctly obtain the page content after JavaScript is executed.

A user may see the full content in a browser while the crawler receives an almost empty page after rendering or cannot access the key content.

Indexing

After analyzing the page, Google decides whether to add it to the search index and which URL should be treated as the primary version among similar or duplicate alternatives.

Ranking

Only after that does the question arise of which queries the page can appear for and at which positions.

That is why “Google does not know this URL” and “the URL is indexed but does not rank” require completely different actions.

Google Has Not Discovered the Page Yet

For a new page, this is one of the simplest explanations.

The URL exists, but Google has not yet received enough signals to discover it.

This often happens with pages that:

  • have just been created;

  • have no internal links;

  • are orphan pages;

  • are missing from the sitemap;

  • sit very deep in the site structure.

A particularly suspicious situation is when the page effectively exists only because you know its URL.

If neither users nor crawlers can reach it from other pages on the site, its integration into the site structure is weak.

What to Do

Add a relevant internal link from an already accessible page, include the canonical URL in sitemap.xml, and make sure the sitemap itself is accessible to Google.

For a specific important page, you can then use Request Indexing through URL Inspection.

However, Request Indexing is a request to recrawl the URL, not a command telling Google that it must add the page to the index.

The Page Is Blocked in robots.txt

robots.txt is often mentioned as a cause of indexing problems, but there is an important nuance.

robots.txt primarily controls crawling.

A Disallow rule tells crawlers which URLs should not be crawled.

This is not the same as a noindex directive.

In some cases, Google may know about a blocked URL from other sources and may even show the URL itself in the index without having access to the page content.

That is why using robots.txt as a universal way to “prevent indexing” is the wrong approach.

There is another practical nuance: if you want Google to see a noindex directive, it generally needs to be able to crawl the page and read that directive.

A noindex Directive Is Still Present on the Page

This is a very common scenario after development, redesign, or migration.

Pages are blocked from indexing on staging, and the noindex directive is accidentally carried over to production.

Everything looks normal to the user: the page opens, the content is there, and the form works.

For the search engine, however, the signal is different:

this page should not be included in the index.

If Page Indexing or URL Inspection shows that the URL is excluded because of noindex, and the page should be available in search, you need to find the source of the directive and remove it.

It may come from:

  • meta robots;

  • X-Robots-Tag;

  • a CMS rule;

  • an SEO plugin;

  • a template;

  • settings for a specific page.

There is no point in repeatedly clicking Request Indexing while the underlying cause remains.

Googlebot Receives a 401 or 403 Response

Another group of problems occurs when the page is accessible to you but inaccessible to Googlebot.

This does not necessarily mean there is a standard login form.

Possible causes include:

  • staging rules;

  • Cloudflare;

  • firewall;

  • WAF;

  • anti-bot protection;

  • server rules;

  • access errors;

  • IP or user-agent restrictions.

In this situation, the problem is not the SEO copy.

Google simply cannot retrieve the page properly.

That is why Test Live URL and checking the actual server response are much more useful here than rewriting the content.

5xx Server Errors

If Googlebot repeatedly receives 5xx errors, the issue is page availability.

Possible causes include server overload, application problems, unstable hosting, timeouts, or CDN configuration issues.

From an SEO perspective, this is a serious problem, but it is not solved through meta tags.

First, the page needs to return a stable response to both users and crawlers.

Only after that does it make sense to analyze indexing.

Page with Redirect and Redirect Error Are Not the Same Thing

These statuses in Page Indexing should be interpreted differently.

Page with Redirect

If a URL permanently redirects users to another page, the original address normally should not be indexed as a standalone page.

For example:

/old-service/ → 301 → /new-service/

In this case, the important thing is to verify that the final URL is correct rather than trying to index the old one.

Redirect Error

This is an actual problem.

Possible causes include a redirect loop, an overly complex redirect chain, an incorrect URL, or another issue preventing Google from following the redirect properly.

The redirect itself needs to be fixed.

Canonical: The Page May Be Accessible, but Google Indexes Another URL

One of the most important areas in indexing diagnostics is canonicalization.

A website may contain several URLs with identical or very similar content. The search engine needs to determine which one should be treated as the primary version.

It is important to remember that a canonical is a strong signal, not a strict command.

Google may choose another URL if other site signals contradict the declared canonical.

Alternate Page with Proper Canonical Tag

This status is often not a problem at all.

For example, an alternative or duplicate URL exists and correctly points its canonical to the primary page.

Google indexes the main version — exactly as intended.

Nothing needs to be fixed if the canonical is correct.

Duplicate Without User-Selected Canonical

Google sees duplicate or very similar pages, but the website has not provided a clear canonical signal.

Google then determines which version should be treated as the primary one.

In this situation, check:

  • whether the pages are actually duplicates;

  • which URL should be indexed;

  • which Google-selected canonical has been chosen;

  • whether an explicit canonical is needed;

  • whether the duplicate has any separate SEO value.

Google Chose Different Canonical Than User

Here, the website declared one canonical URL, but Google chose another.

This should not automatically be treated as a search engine bug.

Instead, check whether different site signals contradict each other.

For example, a page may declare URL A as canonical, while:

  • internal links mostly point to B;

  • B appears in the sitemap;

  • redirects support B;

  • the content on A and B is almost identical;

  • other signals also point more strongly to B.

In that case, a single <link rel="canonical"> may not be enough.

What Crawled — Currently Not Indexed Means

This is one of the most common and most frequently misunderstood statuses.

It means Google has already crawled the page but has not currently added it to the index.

This is an important distinction: the problem is not that the crawler has failed to discover the URL.

That is why endlessly repeating Request Indexing usually does not help.

Instead, ask a different question:

Why does this URL need its own place in the index?

  • Check:

  • whether the page has standalone value;

  • whether it duplicates other URLs;

  • whether it serves a distinct search intent;

  • whether the canonical is configured correctly;

  • whether it is sufficiently integrated through internal linking;

whether Google can see the main content after rendering.

It is important not to reduce this status to the simplistic conclusion that “the content is bad.”

Crawled — currently not indexed does not, by itself, reveal the cause.

Thin content may be a problem. Duplication may also be a problem. Weak or unclear intent may be a problem as well. But the conclusion should be based on analysis of the specific URL.

steps_image

What Discovered — Currently Not Indexed Means

This situation is different.

Google already knows about the URL but has not crawled it yet.

In other words, the page stopped at an earlier stage than in the Crawled — currently not indexed scenario.

For one newly created page, this may simply be temporary. But if the status accumulates across a large group of URLs, the broader site structure should be reviewed.

Potential factors include:

  • weak internal linking;

  • large numbers of technical URLs;

  • parameters and filters;

  • site scale;

  • sitemap setup;

  • unstable server performance;

  • too many pages that Google does not consider a crawling priority.

Crawl budget matters mainly for large and technically complex websites.

For a small corporate website, crawl budget should not automatically be blamed for every indexing problem.

There Are No Technical Blocks, but the Page Is Still Not Indexed

This is where many SEO myths begin.

robots.txt allows crawling.

There is no noindex.

The server returns 200.

The canonical looks correct.

The URL is in the sitemap.

What now?

The worst answer is: “Add another 1,000 words.”

Google is not required to index every technically accessible page.

You need to look at the role of the URL within the website.

Does the page provide standalone value compared with existing pages?

Does it almost completely duplicate another page?

Does it have a clear, independent search intent?

Does it look like a soft 404?

Are there internal links showing its place in the site architecture?

Can the crawler see the main content after JavaScript rendering?

For example, an ecommerce site may have hundreds of filter URLs that technically load, return 200, and are crawlable. That does not mean every one of them deserves separate indexing.

What Is a Soft 404 and How Does It Affect Indexing?

A soft 404 is a situation where a page technically returns a 200 status but, based on its content, looks missing, empty, or unable to provide a meaningful standalone result.

For example, a product is unavailable and instead of a properly handled page, the user sees an almost empty template with a “nothing found” message.

The server says:

the page exists.

The content says:

there is effectively nothing here.

That is why an HTTP 200 response does not guarantee that Google will treat the URL as a complete page worth indexing.

Why sitemap.xml Does Not Guarantee Indexing

This is one of the most persistent SEO myths.

sitemap.xml helps Google discover URLs and understand which canonical pages the website considers important.

But including a URL in the sitemap does not mean Google will automatically add it to the index.

And that is normal.

A sitemap is a discovery source and a structural signal, not a list of pages that the search engine is required to index.

That is why the logic “the URL is in the sitemap, so indexing cannot be the problem” is incorrect.

The same applies to:

URL added to sitemap → guaranteed appearance in Google.

There is no such guarantee.

Why Request Indexing Does Not Solve Every Problem

The Request Indexing button is useful for important new or substantially updated pages.

But it does not remove the root cause.

If the page still contains noindex, the request will not remove it.

If the URL is blocked from crawling, the button will not change robots.txt.

If the server returns 5xx, the canonical points elsewhere, or the URL is a duplicate, resubmitting it will not automatically make it suitable for indexing.

The correct order is:

diagnose → fix → test live URL → use Request Indexing if needed.

Not the other way around.

Step-by-Step Diagnostics When a Page Is Missing from Google

If you need a specific sequence of actions, it can look like this.

1. Check the URL Through URL Inspection

Start with the actual status of the specific page rather than assumptions.

2. Read the Page Indexing Status

Crawled — currently not indexed, Discovered — currently not indexed, Page with redirect, Duplicate, and Blocked by robots.txt are different issues.

Do not treat them the same way.

3. Run Test Live URL

This lets you check what is happening with the current version of the page right now.

It is especially useful after fixing a technical issue.

4. Check the HTTP Status

For a page that should be indexed, the expected response is usually a stable 200.

A 301, 404, 403, or 5xx response takes the diagnosis in a completely different direction.

5. Check robots.txt and noindex

Do not confuse them.

robots.txt relates to crawling.

noindex relates to indexing.

6. Compare Canonicals

Check the user-declared canonical and Google-selected canonical.

If they differ, determine why Google considers another URL the stronger primary version.

7. Check Internal Linking and the Sitemap

Are there proper internal links to the URL?

Is it integrated into the site structure?

Does the sitemap contain the correct canonical address?

8. Evaluate Duplication and Standalone Value

Does this page actually need to be indexed as a separate result?

This is especially important for tags, filters, parameterized URLs, and very similar landing pages.

9. Check Rendering

This is especially important for websites that rely heavily on JavaScript.

In the live test, make sure Google can see the primary content rather than an empty or partially loaded template.

10. Give Google Time After the Fix

Request Indexing may help place a specific URL back into the crawl queue, but it does not guarantee an immediate result.

There is no need to start another round of changes after a few hours just because the status has not updated yet.

Should Every Page on a Website Be Indexed?

No.

This is an important part of correctly interpreting Page Indexing.

The goal of SEO is not to achieve a beautiful “100% Indexed” number.

Some URLs are perfectly fine to keep out of the index.

For example:

  • duplicates;

  • utility pages;

  • technical parameters;

  • filters without separate search intent;

  • deleted URLs;

  • alternate pages with a correct canonical.

A real problem occurs when Google fails to index important canonical pages that should participate in search.

So a sharp increase in the Not indexed section is not always a disaster. First, check which URLs actually ended up there.

How to Speed Up Indexing Without SEO Myths

There is no magic “index now” button.

However, you can make it easier for Google to discover, crawl, and correctly understand an important page.

First, the page should be properly integrated into the website through internal links.

Its canonical URL should be included in an up-to-date sitemap.xml.

Googlebot should have stable technical access to the page.

Issues involving robots.txt, noindex, server errors, redirects, and canonical conflicts need to be fixed.

The content should have sufficient standalone value rather than being another nearly identical page with no separate purpose.

For an important new or updated URL, Request Indexing can be used after these checks.

What does not make sense is building the process around questionable ping services, mass backlink submissions, or other supposed “indexing accelerators” when the underlying reason the page is not being indexed has not been resolved.

How to Monitor the Result After Fixing the Issue

The work is not finished once the technical change is made.

For a specific URL, return to URL Inspection and check its status after Google recrawls it.

For broader issues, monitor Page Indexing and the indexed / not indexed trend.

Check the sitemap status separately.

And once the page appears in the index, review Performance in Search Console: is the URL receiving impressions, and for which queries?

At this point, it is important to separate two concepts again:

indexing ≠ traffic.

The fact that Google has added a URL to the index only means that the page is eligible to participate in search.

The next task is different: does the search engine consider the page relevant and competitive enough to show it for the target queries?

If the Page Is Indexed but Does Not Appear for the Target Query

This is no longer an indexing problem.

First, use URL Inspection to confirm that the page is actually indexed and that the Google-selected canonical matches the expected URL.

If everything is correct, analyze:

  • search intent;

  • content relevance;

  • competitors;

  • internal linking;

  • external signals;

  • page structure;

  • overall site quality.

Clicking Request Indexing in this situation will not “improve the ranking.”

This is simply a different SEO task.

Good Indexing Diagnostics Start with the Status, Not a List of Tips

If an important page does not appear in Google, generic advice such as “check robots.txt, the sitemap, and the content” provides very little value.

First, establish what actually happened.

Has Google not discovered the URL yet?

Has it discovered the URL but not crawled it?

Can Googlebot not retrieve the page?

Does the page contain noindex?

Does it redirect?

Does Google consider another URL canonical?

Has the page already been crawled but not yet added to the index?

Or is it actually indexed, while the real problem is ranking?

Only after answering that question does it become clear what needs to be fixed.

That is why website indexing is not a race to maximize the number of indexed URLs. The goal is much more specific: to make sure Google can discover, crawl, understand, and index the canonical pages that genuinely matter to users and search.

FAQ

What Is Website Indexing?

Website indexing is the process in which Google analyzes a page and may add information about it to its search index. Before that, the URL first needs to be discovered and usually crawled. Indexing should not be confused with ranking: a page can be indexed and still fail to achieve noticeable positions.

How Can You Check Website Indexing in Google?

For an individual page, the best tool is URL Inspection in Google Search Console. For analyzing many URLs, use Page Indexing. A site: search can only be used as a quick approximate check.

Why Is a New Page Not Being Indexed?

Google may not have discovered the URL yet or may not have crawled it yet. Check internal linking, sitemap.xml, page accessibility for Googlebot, and the status in URL Inspection. If the page is important and technically accessible, you can use Request Indexing after the check.

How Long Does Google Take to Index a New Page?

There is no fixed timeframe. The speed depends on discovery, crawl frequency, site structure, technical accessibility, and other signals. Request Indexing also does not guarantee immediate inclusion in the index.

What Does Crawled — Currently Not Indexed Mean?

Google has already crawled the page but has not currently added it to the index. This does not automatically mean the content is poor. Check duplication, canonicalization, search intent, standalone URL value, internal linking, and rendering.

What Does Discovered — Currently Not Indexed Mean?

Google knows about the URL but has not crawled it yet. For one new page, this may be temporary. If the status appears across a large group of URLs, check site structure, internal links, sitemap configuration, technical URLs, and server performance.

Does sitemap.xml Guarantee Indexing?

No. A sitemap helps Google discover URLs and understand the site structure, but it does not force the search engine to index every page listed in it.

Does Request Indexing Help a Page Appear in Google Faster?

It can help place an important URL into the recrawl queue, but it does not fix technical or content-related causes and does not guarantee inclusion in the index.

Can robots.txt Prevent a Page from Appearing in Google?

robots.txt controls crawling and is not a reliable tool for preventing indexing. A URL that Google knows about from other sources may, in some situations, remain known to the search engine even without normal crawling. Indexing is controlled through different mechanisms, including noindex.

Why Did Google Choose a Different Canonical?

Google treats canonical tags as signals rather than absolute commands. If internal links, the sitemap, redirects, duplication, or other signals point to another URL, the search engine may choose that URL as the Google-selected canonical.

Should Every Page on a Website Be Indexed?

No. Duplicates, technical parameters, unnecessary filters, utility URLs, deleted pages, and alternate pages with a correct canonical may reasonably remain outside the index. What matters is that the necessary canonical pages are indexed.

What Should You Do If a Page Is Indexed but Does Not Appear for the Target Query?

In that case, the problem is no longer indexing. You need to analyze search intent, page relevance, the competitive landscape, internal linking, content, and other ranking factors.

Check out our blog
All publications
The tour is over.
Let's get to work!
Fill out the form and buckle up — we'll take the lead now!
Fill out the form