SEO

Robots.txt or noindex? A practical guide for small websites

Understand the different jobs of crawl instructions and indexing rules before changing a website’s search settings.

By Graphic Core   /   Published
Monthly guide series: July 2021

A miniature signpost between cards showing a crossed-out eye and branching arrows.

Separate crawling from indexing

Crawling is the process of a search engine requesting a page or resource; indexing is the process of storing and considering eligible content for search. The two are related but not identical. A website owner may want an important service page to be discoverable while choosing not to show a thank-you page as a search result.

A robots.txt file gives participating crawlers instructions about access to paths. A noindex rule asks supporting search engines not to include a page in their results. The right choice depends on the reason for the restriction. Before changing anything, list the exact URLs affected and the people who need to use them.

Understand what robots.txt can do

A robots.txt file is normally placed at the root of a site and can specify rules for participating crawlers. It is used for crawl management, not as a password or a way to erase a URL from search. If another page links to a blocked address, a search engine may still know that the address exists even when it cannot read the page content.

Avoid broad restrictions on an active business website unless you understand their effect. Blocking an entire site or key resources can prevent search engines from seeing content you intended to publish. A developer should check host, protocol, rule syntax and the actual public file before assuming a rule applies as intended.

Understand noindex

A noindex instruction can be delivered in an HTML robots meta tag or, for suitable resources, an HTTP header. Google explains that it must be able to access the page to see the instruction. If a URL is blocked by robots.txt, the crawler may not see the noindex rule, so combining the two without a clear plan can fail to achieve the intended result.

Use noindex only when the specific page should be kept out of supporting search results. It is not a way to protect confidential information and is not a replacement for login or access control. Different search engines and other automated tools may behave differently; check the relevant platform’s documentation.

AT A GLANCE

Different tools, different jobs

  • Robots.txtGuides participating crawlers
  • NoindexRequests exclusion from search
  • AuthenticationRestricts access to private content
  • SitemapLists preferred public URLs
Crawl rules, indexing instructions and access controls are not interchangeable.

Choose the rule for the situation

For a public service page that should be available in search, check that the page loads, can be crawled and does not contain an unintended noindex instruction. A private client document needs proper access controls. A public post-submission confirmation page might use noindex if the owner does not want it in search, provided the crawler can read the instruction.

For development or staging, use appropriate authentication or another deliberate access-control method. Relying on obscure addresses or a robots.txt entry can leave content accessible. Keep a launch checklist so a temporary restriction is removed from intended public pages when the site is released.

Inspect before and after a change

Open the relevant public URL and ask a developer to examine its HTML or response headers. Check whether the robots.txt file blocks the crawler before relying on an in-page noindex rule. Search Console can help inspect a page and report what Google observed, but its indexed record and a live test may describe different points in time.

Record the original setting and the intended outcome, then make one controlled change. Recheck the live response and any relevant site-wide settings. Processing is not instantaneous and no individual tool can promise a particular search result immediately. Keep a route for reversing an incorrect restriction.

An illustrative launch issue

Imagine a business launches a redesigned website and finds that its main service page still carries a noindex instruction from staging. The page is meant to be public, so the team confirms the correct production URL, removes the unintended rule and verifies the live response. They also check that the site’s robots.txt file does not prevent access to it.

A separate internal document remains behind appropriate authentication; it is not made public merely to let a crawler read a tag. This example is hypothetical. The exact fix depends on the site’s hosting and configuration, and existing indexed information may take time to update.

A common conflicting setup

Suppose a public page is blocked in robots.txt and also carries a noindex tag. Because a crawler that follows the block cannot fetch the page to read the tag, the intended removal may not happen as expected. First decide whether the content should be public at all; if it should be excluded from search but remain public, use an approach that allows the supported crawler to see the relevant instruction.

This is a technical example rather than a universal configuration. Check the current search-engine guidance, the actual URL and any response headers with a qualified person. Other tools, including malicious crawlers, do not necessarily follow voluntary instructions.

Prevent a repeat at launch

Give staging and production settings clear names, and document who can change them. Before release, inspect representative page types rather than checking only the homepage: service pages, articles, location pages and important documents may be controlled by different templates or plugins.

After release, record the live robots file and a sample of page responses, then schedule a follow-up using the tools available to the business. If an unexpected exclusion appears, compare the live configuration with the intended page list and make a reversible change. Maintain separate access protection for anything genuinely private.

A safe review checklist

For each important page, document whether it should be public, whether it should appear in search and which URL is canonical. Check the current robots.txt file, page-level directives and any plugin or hosting settings that may add headers. Review staging settings separately from production so a precaution in one environment does not unintentionally affect the other.

Before publishing a restriction, ask who approved it, what evidence will show it works and how to reverse it. If a page contains sensitive material, address access and privacy first rather than relying on indexing instructions. Accurate configuration supports discovery but does not ensure that a search engine will index or rank a page.

YOUR NEXT MOVE / 01

Ready to make
more of your business?

Tell us about your website, your services and where you want to grow.

START A CONVERSATION