Does robots.txt remove a page from Google?

No. It stops the page being fetched. The URL can still be listed if other sites link to it, typically with no snippet, because the crawler cannot see the content or any noindex directive.

Can I use both robots.txt and noindex on the same page?

You should not. Blocking the fetch prevents the noindex being read, so the two directives cancel out. Choose one: allow the crawl and use noindex, or block the crawl and accept the URL may be listed.

Where should the noindex directive go?

As a meta robots tag in the page head, or as an X-Robots-Tag HTTP header. The header is the option for non-HTML files such as PDFs.

Is robots.txt enforceable?

It is voluntary. Major search engines honour it; scrapers and malicious bots ignore it entirely. Anything confidential needs authentication.

Explainers

robots.txt vs noindex: which one actually keeps a page out of Google

The two directives control different things, and using the wrong one is why blocked pages keep appearing in search results.

By Remi Okafor, Editor, Toolspea · Published 2026-07-15 · Updated 2026-08-06 · 5 min read

Two different controls

robots.txt is a request not to fetch a URL. noindex is an instruction, delivered in the page's HTML or an HTTP header, not to include the page in results. They operate at different stages, which is why combining them incorrectly produces the outcome nobody wants.

The classic mistake

Disallowing a URL in robots.txt and adding a noindex tag to it means the crawler never fetches the page, never sees the tag, and can still list the URL based on external links — usually with no description. The fix is to remove the disallow so the directive can be read.

Choosing the right tool

Match the directive to the goal:

Keep a page out of search results: allow crawling, serve noindex.
Save crawl budget on infinite or worthless URL space (faceted filters, session parameters): disallow in robots.txt.
Protect private data: authentication. Neither directive is a security control.
Consolidate duplicates that should still be reachable: a canonical tag, not a block.

After the page is already indexed

Removal takes a re-crawl. Serve the noindex, confirm the page is crawlable, and request the URL through Search Console's removal tool if the delay matters. Deleting the page instead and returning a 410 also works and is often faster to communicate.

Frequently asked questions

Tools mentioned

Robots.txt Generator — Build a valid robots.txt with crawl rules and a sitemap reference.
Meta Tag Generator — Produce title, description, Open Graph and Twitter tags for any page.
SERP Snippet Preview — See how a title and description will truncate in search results.

Keep reading

Everything referenced in this article, plus the hubs and guides that go with it.

Robots.txt GeneratorToolBuild a valid robots.txt with crawl rules and a sitemap reference.Meta Tag GeneratorToolProduce title, description, Open Graph and Twitter tags for any page.SERP Snippet PreviewToolSee how a title and description will truncate in search results.SEO toolsCategory4 browser-based seo tools that never upload your files.What is Base64 encoding, and when should you actually use it?GuideBase64 turns binary into safe ASCII text. Here is how it works, why it is not encryption, and the three cases where it is the right tool.Title tags and meta descriptions: what still mattersGuideLength limits, rewriting behaviour and a repeatable way to write snippets that survive truncation and earn the click.Using the toolsFAQInput limits, supported formats, browser requirements and what to do when a tool behaves unexpectedly.