Skip to main content
SwiftTools

How to Write a Valid robots.txt (and What It Cannot Do)

Published November 6, 2026Guide for the Robots.txt Generator

robots.txt is the first thing a crawler reads when it visits your site, and getting it wrong can either block pages you want indexed or leave sensitive paths exposed to scrapers. It is a small file, but the syntax has enough subtlety that common mistakes are frequent.

This guide explains how to write valid rules, how to reference your sitemap, and the crucial limit: robots.txt is a request, not a wall.

The basic structure

A robots.txt is a set of records. Each record names a user-agent (the crawler, or "*" for all) and then lists allow and disallow paths. A simple file points crawlers to your sitemap and keeps admin or staging paths out of search results.

Allow versus disallow

disallow tells crawlers not to fetch a path; allow can carve an exception back out of a broader disallow. To expose one page inside a otherwise-hidden folder, disallow the folder and allow that page. Rules are matched by prefix, so ordering and trailing slashes matter.

The limit that catches everyone

robots.txt is a polite request to well-behaved crawlers, not a security control. A malicious scraper ignores it, and major search engines will not index a path you disallow. Never rely on robots.txt to hide sensitive content - that requires authentication. Its real job is guiding crawl budget and keeping non-sensitive paths out of the index.

Frequently asked questions

Where does the file go?

At your site root as /robots.txt, so it is the first file a crawler requests. Reference your sitemap URL there too with a "Sitemap:" line.

Does it block a page from being indexed?

It blocks crawling, which usually prevents indexing, but it is not guaranteed. For firm exclusion, use authentication or a noindex meta tag, not robots.txt alone.

Want to try it?

The Robots.txt Generator is free and runs right in your browser (seo tools). Open it now and see the guide in action.

Try the Robots.txt Generator