Detailed answer
It asks compliant search engines not to index a page they can fetch.
The crawler must be able to retrieve the page to read the tag. Use authentication to protect private material because noindex is not access control. Local and staging content should not accidentally become a public search destination.
Example
<meta name="robots" content="noindex, nofollow">
Understanding the behavior
Noindex asks a crawler that can access the page to exclude it from search results. Blocking the URL in robots.txt can prevent the crawler from seeing that instruction. It is not an access-control mechanism for private data.
Practical implementation notes
Inspect both the robots meta tag and any X-Robots-Tag response header. For private material, enforce authentication and authorization on the server; do not rely on a search directive to keep the content confidential.
How to check it
Check the returned robots metadata while keeping private data behind authentication.
Expected result
A compliant crawler can read noindex; sensitive content still needs access control.
Watch out: Blocking crawling can prevent a search engine from seeing the noindex tag.
Read the reference ↗