Robots.txt Guide: Write, Test, and Troubleshoot Crawl Rules for Modern Websites

Overview

Robots.txt is a plain-text file placed at the root of a website host, such as https://example.com/robots.txt. It communicates crawl preferences to automated clients that choose to follow the file. A rule can allow or disallow access to URL paths, and the file can point crawlers to one or more XML sitemaps.

It is important to separate crawling from indexing. A disallowed URL may still be discovered or referenced without its content being fetched. Conversely, a page that is crawlable is not automatically guaranteed to appear in search results. Robots.txt is therefore best used to manage crawler access to paths, not to hide confidential information or guarantee removal from an index.

The basic directives are:

Support for advanced matching behavior can vary between crawlers. Treat a robots.txt tester as a verification aid, not as permission to assume that every automated client interprets every directive identically.

Checklist by scenario

Starting with a simple public website

Most small sites do not need a complicated rule set. Begin with the smallest file that expresses the real requirement:

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

An empty Disallow value means no path is blocked for that group. Replace the sitemap URL with the canonical sitemap location for the correct host and protocol. If you do not have a sitemap, omit the line rather than publishing a guessed URL.

• Confirm that the file is available at the root of the intended host.
• Use plain text, not HTML, JSON, or a document exported from a word processor.
• Check that the server returns the file successfully and does not redirect it to an unrelated page.
• Keep comments brief and make the file easy to review during deployments.

Blocking an internal or low-value path

To request that crawlers avoid a directory, include its path from the root:

User-agent: *
Disallow: /internal/
Disallow: /tmp/

Sitemap: https://example.com/sitemap.xml

A trailing slash helps communicate that the rule is intended for a directory path. Review actual URLs before adding the rule. For example, blocking /account/ may be reasonable for a private application area, but blocking a parent such as /app/ could also cover public documentation or assets.

Allowing a specific path inside a blocked directory

Some sites need a narrow exception:

User-agent: *
Disallow: /resources/
Allow: /resources/public-guide.html

Because matching and precedence details can differ by crawler, test the exact URL with the crawler or provider that matters to your site. If an exception is essential, a simpler URL structure or a narrower disallow rule is often easier to maintain than a dense set of overlapping rules.

Blocking query-driven crawl paths

Faceted navigation, search results, and tracking parameters can create many URL variations. A rule such as Disallow: /search may cover a path, but it does not automatically solve every parameter pattern or alternate endpoint. Map representative URLs first, including encoded parameters and trailing-slash variants, then test them individually.

Do not use robots.txt to protect passwords, customer records, unpublished files, or administrative actions. Use authentication, authorization, server controls, and appropriate response handling for sensitive resources.

Launching a new section or migrating a site

During a migration, compare the old and new files rather than copying a temporary staging rule into production. A staging file often contains a sitewide block such as:

User-agent: *
Disallow: /

That rule may be appropriate for a private environment but disastrous on a public launch. Before switching DNS or hosting, verify the production file, sitemap references, redirects, canonical URLs, and the response from the final public hostname. For DNS and hostname changes, the DNS records guide provides related implementation checks.

What to double-check

1. Host and location: Robots rules apply to a specific protocol, hostname, and port context. Check each relevant host, including www, non-www, staging, and application subdomains where applicable.
2. Exact syntax: Use one directive per line, a colon after the directive name, and a root-relative path for Disallow and Allow. Remove invisible characters and accidental formatting.
3. User-agent grouping: Make sure rules sit under the intended crawler group. A rule under a named group is not necessarily equivalent to a rule under User-agent: *.
4. Wildcard behavior: Characters such as * and end-of-string markers may be supported differently. If you use them, test both a URL that should match and one that should not.
5. Allow and disallow overlap: Read every matching rule together. A broad block and a narrow exception can produce surprising results if the intended crawler interprets specificity differently than your local tester.
6. Sitemap URL: Use an absolute, publicly reachable URL. Confirm that the sitemap contains the host and URL forms you actually want discovered.
7. HTTP response and caching: Check the status, content type, body, redirects, compression, and CDN or proxy behavior. After publishing a change, verify that the response no longer reflects an older cached file.
8. Related directives: Compare robots rules with canonical tags, meta robots directives, and server responses. A canonical points to a preferred URL; it does not replace a crawl-access decision. A noindex instruction also requires the crawler to access the page to reliably read it, so blocking the same URL can prevent that signal from being seen.

Use a robots.txt tester or a command-line request as part of the review. A browser-based developer tool is useful for quickly checking the published text, while a deployment check can confirm that the file exists in the build output. Keep a small test list containing critical pages, blocked paths, asset URLs, parameter variants, and one deliberately similar path that should remain accessible.

Common mistakes

• Using robots.txt as a security barrier: Disallowed paths remain guessable and may still be requested by clients that ignore the file.
• Blocking CSS, JavaScript, or image resources without checking rendering needs: Search systems may need certain public resources to understand a page. Review resource paths before adding broad asset rules.
• Adding a leading full URL to a path rule: A line such as Disallow: https://example.com/private/ is not the same as the root-relative Disallow: /private/.
• Leaving a temporary sitewide block in production: Treat Disallow: / as a high-risk change and require an explicit launch review.
• Assuming comments change behavior: Comments can document intent, but only supported directives and valid values should be relied upon.
• Testing only the file text: A correct local file can still fail when deployed at the wrong hostname, behind a redirect, or through a stale cache.
• Blocking a URL that must be removed: Crawl control and removal are different tasks. Use the appropriate application, server, or search-platform process for content that must no longer be accessible or indexed.

For a broader release review, combine this checklist with the technical SEO pre-launch audit. Security and accessibility checks also belong in the deployment workflow; see the guides to Content Security Policy and web accessibility testing.

When to revisit

Review robots.txt whenever URL patterns, hosting, or crawler-facing workflows change. At minimum, add it to the checklist for:

• a site launch, redesign, or framework migration;
• a change to domain, subdomain, protocol, CDN, or hosting provider;
• the introduction of faceted navigation, internal search, preview URLs, or a new application area;
• an indexing or crawl-access investigation;
• a seasonal content cycle that creates temporary sections or landing-page patterns;
• a change to sitemap generation, deployment tooling, or crawler-testing tools.

Make the next review practical: record the intended purpose of every block, keep production and staging rules visibly separate, test a fixed set of representative URLs, and inspect the live response after deployment. When crawler guidance or testing interfaces change, confirm the current supported syntax in the relevant provider documentation instead of relying on an old screenshot or cached instruction. A short, deliberate robots.txt file is easier to validate than a collection of emergency exceptions—and much safer to carry into the next release.