Indexed, Though Blocked by robots.txt: What It Means and How to Fix It

Search Console says a page is Indexed, though blocked by robots.txt. That sounds like a contradiction. You told Google not to go there, and it put the page in its index anyway.

It isn't a bug. robots.txt controls crawling, not indexing, and Google can index a URL it has never opened if other pages link to it. Once you see that difference, the fix is simple, and it's usually the opposite of what people try first: you remove the robots.txt block, not add a stronger one.

This guide covers what the warning means in Google's own words, robots.txt vs noindex, which pages you can leave alone, how to fix it step by step, and how to fix it on WordPress, Shopify and Next.js. It is the seventh part of my SEO checklist series, after page titles, meta descriptions, canonical tags, H1 tags, image alt text and pages Google hasn't indexed. While writing it, I found the same pattern on my own site, and decided to keep it.

The short answers

• What does "Indexed, though blocked by robots.txt" mean? Google found links to the page and indexed the URL, but robots.txt stops it from reading the page. It knows the address, not the content.
• Why would Google index a blocked page? Because robots.txt only says "don't crawl". In Google's words, it "doesn't necessarily prevent indexing if someone else links to your page."
• Is it an error? It's a warning. It only matters if the page should be in search, or if it should be completely out of it.
• How do I fix it? If the page should be in search, remove the rule that blocks it. If it should be out of search, remove the rule and add noindex. Google has to crawl a page to see its noindex.
• Can I put noindex in robots.txt? No. Google says "Specifying the noindex rule in the robots.txt file is not supported by Google." It stopped reading it in 2019.
• How long does it take to clear? Google has to fetch your robots.txt again and then recrawl the page. That can be days for busy pages and much longer for quiet ones.

What the warning means

Here is Google's definition from the Page indexing report:

"The page was indexed despite being blocked by your website's robots.txt file. Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page. Google won't request and crawl the page, but we can still index it, using the information from the page that links to your blocked page."

Two things happen separately in Google:

1. Crawling. Googlebot requests the page and reads it. robots.txt can stop this.
2. Indexing. Google stores the URL so it can show it in results. robots.txt cannot stop this.

So when your robots.txt says Disallow: /cart/ and another site links to https://example.com/cart/, Google respects the rule and never fetches the page. But it still knows the URL exists, and it knows the link text pointing to it. That's enough to index the bare URL.

What it looks like in search

A blocked page that gets indexed usually shows up with its URL and a title guessed from links, but no real description. Google's robots.txt guide says: "If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description." In practice the snippet often reads "No information is available for this page."

That's the worst of both: the page is in search, and it looks broken.

robots.txt vs noindex: the difference

This is the whole topic in one table:

The important line is the fourth one. Google's noindex documentation puts it plainly:

"For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results."

So blocking a page in robots.txt and adding noindex to it is not belt and braces. The block hides the noindex. You've turned off the one rule that would have kept the page out.

Neither one is security. robots.txt is a public file that anyone can read, and badly behaved bots ignore it. If something must stay private, put it behind a login.

Why it happens

These are the usual causes, roughly from most to least common:

• Using robots.txt to hide pages. Someone wanted pages out of Google, blocked them in robots.txt, and links found them anyway. This is the textbook case.
• robots.txt and noindex on the same pages. The noindex is there, but Google can't read it. This is the one I found on my own site.
• Blocked URLs that get linked. Internal search (/search?q=), cart and checkout pages, filtered listings and "add to cart" links are often blocked on purpose. When people or your own templates link to them, a few get indexed.
• A rule that's wider than you meant. robots.txt rules match the start of the path, so Disallow: /app blocks /app, /app/settings, and also /apple-touch-icon.png and /application-form. Disallow: /*? blocks every URL with a query string, including ones you wanted indexed.
• Rules left over from staging. A Disallow: / that was right on the test site, copied to the live one. If this is the cause, it hits the whole site at once.
• Old rules nobody remembers. A folder was blocked years ago, the site changed, and the rule now covers pages that matter.

Should you fix it?

Not every entry needs work. Open the report, look at the URLs, and put each one in one of these rows:

The last row is a judgement call. A handful of ?add-to-cart= URLs in this report is common and does little harm, because almost nobody searches for them. If the report holds real pages, or you find them showing up in search with "No information is available", fix them.

How to fix it, step by step

1. Find the URLs. In Search Console, open Indexing → Pages. This is a warning, not a reason for non-indexing, so it's listed in the Improve page experience table rather than under "Why pages aren't indexed". Click it and Export the full list.
2. Find the rule. Inspect a URL. Under Page indexing, Crawl allowed? will say No: blocked by robots.txt. To see which line is responsible, open Settings → robots.txt in Search Console, which shows the file Google last fetched, or run the script below on the whole export. (Google retired the old robots.txt Tester in 2023, so older guides that send you there are out of date.)
3. Decide what each page should be, using the table above.
4. Edit robots.txt. Remove the rule, or narrow it so it stops catching the pages you want crawled. An Allow line can open a single path inside a blocked folder. When two rules match, Google follows the longer one, as in the first example below.
5. Add noindex where the page should stay out. Put the meta tag in the page's <head>, or, for PDFs and other files with no <head>, send it as a response header. Both are shown below.
6. Let Google read the new robots.txt. Google generally caches robots.txt for up to 24 hours. The robots.txt report in Search Console has a Request a recrawl option if you need it sooner.
7. Request indexing for important pages. In URL Inspection, run Test live URL to confirm that crawling is now allowed, then Request indexing for the few pages that matter most.
8. Validate the fix in the report. Validation usually takes a couple of weeks. Pages you've set to noindex will move to Excluded by 'noindex' tag, which is exactly where you want them.

Opening one page inside a blocked folder:

User-agent: *
Disallow: /account/
Allow: /account/help

The noindex meta tag, inside <head>:

<meta name="robots" content="noindex">

The same rule as a response header:

X-Robots-Tag: noindex

Need a page out of search today?

Fixing robots.txt and adding noindex works, but it waits for a recrawl. If a page must disappear now, use Indexing → Removals in Search Console. It hides the URL from results for about six months, which gives the noindex time to take over. A removal on its own isn't permanent. When it expires, the URL can come back if nothing else changed.

Blocked by robots.txt vs Indexed, though blocked

Search Console has two robots.txt statuses, and they sit on opposite sides of the report:

"Blocked by robots.txt" is your rule working. The help page still adds a warning: "there is a very small chance that the page might still be indexed. To ensure that a page is not indexed by Google, remove the robots.txt block and use a 'noindex' directive." In other words, a URL can move from the first column to the second as soon as someone links to it.

The same robots.txt mix-up appears in the canonical tags guide: if a duplicate is blocked, Google can't read its canonical tag either. Any rule inside a page, whether noindex or a canonical, only works on pages Google is allowed to fetch.

What I found on txtnode

Here's txtnode's robots.txt as it is today:

User-Agent: *
Allow: /
Disallow: /notes/
Disallow: /tasks/
Disallow: /invite/
Disallow: /chat/
Disallow: /app/
Disallow: /app$

Sitemap: https://txtnode.com/sitemap.xml

The idea was sound: shared notes, invitations and the signed-in app are for the person holding the link, not for a search engine. But those same pages also send noindex. I ran the checker below on a few of them, and it flagged /notes/…, /app/… and /app as both blocked and noindex. That's the exact pattern Google warns about. If someone posts a link to a shared note on a public page, Google can index that URL, and the noindex that should keep it out is never read.

Google's docs say the fix is to let those pages be crawled so the noindex can do its job. I'm keeping the block anyway. Shared notes are what people write, there are far more of them than articles, and none of them belong in search. I'd rather Googlebot spend its visits on the blog and the tools than on pages it would only be told to drop, and I'd rather it didn't read people's notes at all.

The cost is small and known: a note linked from a public page can show up in this report, and in search as a bare URL with no description. That's the last row of the "Should you fix it?" table, chosen on purpose. The point isn't that robots.txt and noindex must never meet. It's that you should know which one Google can actually see, and pick the trade-off yourself.

There's one more lesson in that file. The last two lines are /app/ and /app$, not a plain /app, because rules match by prefix. A bare Disallow: /app would also have blocked /apple-touch-icon.png, the icon the site's structured data points to as its logo. The checker confirms the icon is crawlable. If you write a short prefix rule, list what else starts with those letters.

Check your blocked URLs with a script

This script reads your robots.txt, checks every URL from a Search Console export against it the way Google does (the longest matching rule wins, and Allow wins a tie), and tells you which rule blocks each one and whether the page also has a hidden noindex. It needs Node.js 18 or later.

// check-blocked.mjs
// Usage: node check-blocked.mjs https://example.com Table.csv
import { readFileSync } from 'node:fs'

const [site, file] = process.argv.slice(2)
const urls = readFileSync(file, 'utf8')
    .split(/\r?\n/)
    .map((line) => line.split(',')[0].replace(/^"|"$/g, '').trim())
    .filter((u) => u.startsWith('http'))

const robots = await (await fetch(`${site}/robots.txt`)).text()

const groups = []
let group = null
for (const raw of robots.split(/\r?\n/)) {
    const [, key, value] = raw.replace(/#.*/, '').trim().match(/^([a-z-]+)\s*:\s*(.*)$/i) ?? []
    if (!key) continue
    const field = key.toLowerCase()
    if (field === 'user-agent') {
        if (!group || group.rules.length) groups.push((group = { agents: [], rules: [] }))
        group.agents.push(value.toLowerCase())
    } else if (group && (field === 'allow' || field === 'disallow') && value) {
        group.rules.push({ allow: field === 'allow', path: value })
    }
}

const rulesFor = (agent) => groups.filter((g) => g.agents.includes(agent)).flatMap((g) => g.rules)
const rules = rulesFor('googlebot').length ? rulesFor('googlebot') : rulesFor('*')

const toRegex = (path) =>
    new RegExp('^' + path.replace(/[.+?^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*').replace(/\\\$$/, '$'))

const ruleFor = (url) => {
    const { pathname, search } = new URL(url)
    const matches = rules.filter((r) => toRegex(r.path).test(pathname + search))
    matches.sort((a, b) => b.path.length - a.path.length || b.allow - a.allow)
    return matches[0]
}

const hasNoindex = async (url) => {
    const res = await fetch(url, { redirect: 'manual' })
    const header = res.headers.get('x-robots-tag') ?? ''
    const head = res.status === 200 ? (await res.text()).split(/<\/head>/i)[0] : ''
    return /(^|,\s*)(googlebot:\s*)?(noindex|none)\b/i.test(header) || /<meta[^>]+name=["'](robots|googlebot)["'][^>]*content=["'][^"']*(noindex|none)/i.test(head)
}

for (const url of urls) {
    const rule = ruleFor(url)
    if (!rule || rule.allow) {
        console.log(`${url}\n    not blocked now, so the warning should clear on the next crawl`)
        continue
    }
    const noindex = await hasNoindex(url)
    console.log(
        `${url}\n    blocked by "Disallow: ${rule.path}"` +
            (noindex ? '\n    and has noindex, which Google cannot see: remove the block' : '\n    no noindex: unblock it, and add noindex if it should stay out'),
    )
}

Export the URL list from the report as CSV and pass it as the second argument. A plain text file with one URL per line works too. The script fetches the blocked pages itself to look for noindex, which is fine on your own site. It follows the same pattern as the checkers in part 1 and part 3.

Platform notes

WordPress

WordPress serves a virtual robots.txt when there's no real file on the server, and SEO plugins such as Yoast SEO and Rank Math let you edit it from the dashboard. Check three things:

• A real robots.txt file in the site's root folder replaces the virtual one completely, so edits in the plugin do nothing. Delete the file or edit it directly.
• Rules added by plugins or old tutorials, such as blocking /tag/, /category/, /?s= or /wp-content/. If you want archives out of search, set them to noindex in your SEO plugin and unblock them. Never block /wp-content/: it holds the images, CSS and scripts Google needs to render your pages.
• WooCommerce cart and "add to cart" URLs. These are the most common entries in this report on shops. They rarely matter. If they bother you, make sure your theme doesn't link to ?add-to-cart= URLs in places Google crawls.

The Settings → Reading → "Discourage search engines from indexing this site" box no longer writes a Disallow: / rule. Since WordPress 5.3 it adds a noindex tag instead. It's still worth checking that it isn't ticked on a live site.

Shopify

Shopify's default robots.txt blocks the cart, checkout, internal search and some filtered collection URLs. Entries for those in this report are normal and usually fine to leave.

To change the rules, add a robots.txt.liquid template to your theme (Online Store → Themes → Edit code → Add a new template → robots). Edit it carefully: Shopify's defaults are sensible, and a broken template affects every page. If you want a page type out of search, noindex it in theme.liquid rather than adding a new block.

Next.js (App Router)

robots.txt comes from app/robots.js, and noindex comes from each page's metadata. Keep a route out of search with noindex alone, not both:

// app/robots.js
const robots = () => ({
    rules: [{ userAgent: '*', allow: '/', disallow: ['/api/'] }],
    sitemap: 'https://example.com/sitemap.xml',
})

export default robots

// app/thank-you/page.js
export const metadata = {
    robots: { index: false, follow: true },
}

That renders <meta name="robots" content="noindex, follow">, and because /thank-you isn't in disallow, Google can actually read it. Blocking /api/ is fine: API responses aren't pages, and nothing there needs a noindex to be seen.

Two things to watch:

• disallow values are prefixes. Write '/app/' and '/app$' rather than '/app' if other files start with the same letters. My robots.txt above does exactly this.
• Metadata is inherited. A robots: { index: false } in a layout applies to every page under it, which is handy for a private section and a disaster in the root layout.

Questions people ask

What does "Indexed, though blocked by robots.txt" mean? Google indexed the URL from links pointing to it, but your robots.txt stops it from crawling the page, so it can't read the content.

How do I fix "Indexed, though blocked by robots.txt"? Decide whether the page should be in search. If yes, remove the robots.txt rule blocking it. If no, remove the rule and add noindex, so Google can crawl the page and see that it should drop it. Then validate the fix in Search Console.

Does robots.txt stop a page from being indexed? No. It stops crawling. Google says robots.txt is "not the correct mechanism to avoid being indexed".

Should I use robots.txt or noindex? Use noindex to keep a page out of search results. Use robots.txt to stop Google spending time on URLs that never need crawling, such as endless filter combinations. Don't use both on the same URL.

Can I add noindex to robots.txt? No. Google stopped supporting noindex in robots.txt in September 2019. Use a meta robots tag or an X-Robots-Tag header.

Is "Indexed, though blocked by robots.txt" bad for SEO? On its own it doesn't hurt the rest of your site. It's bad for the pages involved: either a page you want found is shown without a description, or a page you wanted hidden is in search.

Why are pages I never linked to showing up in the report? Someone else may have linked to them, or they're in your sitemap. Google can find a URL from any page it crawls, not only yours. Also remove blocked URLs from your sitemap. Listing a URL there while blocking it in robots.txt sends Google two different answers.

How long until the warning goes away? Google needs to fetch the updated robots.txt, which it caches for up to a day, and then recrawl each page. Busy pages clear in days; quiet ones can take weeks or more.

Does Bing treat robots.txt the same way? Mostly. Bing also treats robots.txt as a crawling rule, and it can also list a blocked URL it found through links. Bing supports noindex in the meta tag and the header, so the same fix works.

Next in the checklist

This is part 7 of the SEO checklist. Part 1 covers page titles, part 2 covers meta descriptions, part 3 covers canonical tags, part 4 covers H1 tags, part 5 covers image alt text, part 6 covers pages Google hasn't indexed, and the full list is in the SEO audit checklist.

If you're sorting an export of blocked URLs into "should be in search" and "should be out", paste it somewhere you can mark each one as you go. txtnode opens a notepad at any URL you type, without an account, and Files keeps notes in folders when you sign in.