What is robots.txt and how does it work?
Author: Domeny.ForSale editorial team Published
In brief: robots.txt is a public text file that tells web crawlers which parts of a website they may fetch and which they should skip. It helps manage how crawlers visit a site, but it does not protect data or guarantee that a URL will be removed from search results. Its rules apply only to crawlers that respect them.
Where is robots.txt located, and how does it work?
The robots.txt file belongs in the website’s root directory, at an address such as https://example.com/robots.txt. A crawler—a program that visits pages automatically—may read it before fetching their content. It uses the rules to determine which paths it may access.
The rules apply to a specific protocol, host and port. A file available at https://example.com does not automatically control crawlers on https://shop.example.com or on the HTTP version of the site. A subdomain that needs its own restrictions requires a separate configuration.
If the file is missing and the server returns HTTP status code 404, meaning “not found”, this generally means there are no restrictions imposed by robots.txt. This is different from a server failure: server errors encountered when fetching the file may temporarily limit crawling of the website. How these errors are handled depends on the crawler.
How do you read and write the rules?
The basic entries form a group: they identify a crawler, then specify allowed or blocked paths. An example file might look like this:
User-agent: *
Disallow: /search/
Disallow: /working/
Allow: /working/public-example.html
Sitemap: https://example.com/sitemap.xml
The individual elements mean:
User-agent— the name of the crawler to which the rules apply; the asterisk*denotes a general group.Disallow— a path the crawler should not fetch.Allow— an allowed path, particularly useful as an exception to a broader block.Sitemap— the full URL of a sitemap, a file listing the URLs you want to bring to the search engine’s attention.
In this example, crawlers should skip URLs beginning with /search/ and /working/, but may fetch the specified public file in the working directory. When Allow and Disallow rules overlap, the standard gives precedence to the more specific match. Paths are case-sensitive.
The entry Disallow: / blocks crawling of the entire website for crawlers covered by that group. An empty Disallow: entry does not impose this block. This small difference is a common cause of serious mistakes when publishing a website.
If you add a group for a specific crawler, check its documentation for the rules governing group selection. Do not assume that it will automatically inherit all the restrictions listed under User-agent: *.
Blocking crawling does not prevent indexing
Fetching a page, also called crawling, and indexing are two different processes. Indexing involves analysing and storing information about a page in the search engine’s database, which is used to generate search results.
A URL blocked in robots.txt may still appear in search results—for example, if the search engine discovers it through links on other websites. A crawler does not need to fetch the content to learn that the URL exists.
If a publicly accessible page should not appear in search results, the usual approach is to use the noindex directive: an instruction placed in the page’s meta tag or in the HTTP header X-Robots-Tag. The crawler must be able to fetch the response to read noindex. Blocking it in robots.txt at the same time may prevent this. Writing Noindex directly in robots.txt is not a valid way to exclude pages from Google.
For example, a thank-you page displayed after a form submission can remain accessible to users and have noindex applied to it. By contrast, a dashboard containing customer data should require login and permission checks—a rule telling crawlers to stay away does not protect its content.
When should you use robots.txt, and when is another solution needed?
Robots.txt can be useful for limiting the crawling of extensive internal search results or numerous filter combinations. First, however, you need to establish whether those URLs should appear in search results and whether crawlers need to read other instructions on them.
Do not block CSS and JavaScript files needed to display the page without first assessing the consequences. If these files are unavailable, the search engine may have difficulty rendering the site’s appearance and content correctly. Reducing the number of crawler visits alone does not improve the site’s search visibility.
The robots.txt file is public, so do not put passwords, tokens or lists of confidential resources in it. Use authentication or network restrictions to protect a test environment. A malicious crawler may completely ignore the published restrictions.
How do you deploy and check the file safely?
Before changing robots.txt, save a copy of the current version. Then carry out a quick check:
- Open
/robots.txton the correct domain and check that the server returns plain text rather than an error page or a login form. - Prepare sample URLs: the home page, a product page, an article and a path intended to be blocked.
- Check how the rules match these URLs for the specific crawler. Confirm the supported syntax in the search engine’s official documentation.
- After publishing, check the file’s status in webmaster tools such as Google Search Console. Crawlers may cache a copy, so changes will not always take effect immediately.
If the website uses a CMS or a plugin that generates robots.txt, edit the configuration source responsible for it. A manually uploaded file may be overwritten or may replace the automatically generated version. Before saving changes, identify your specific goal: limiting crawling, excluding pages from search results or protecting data. Each requires a different tool.
See also: What is 2FA?
Read also
