abbreviation robots.txt
robots exclusion protocol
Robots.txt is a simple text file placed in a website’s root directory to instruct web crawlers (like Googlebot) which pages or files they should or shouldn’t access. It’s part of the Robots Exclusion Protocol, a standard followed by most search engines.
When a crawler visits a site, it checks for a robots.txt file before scanning. The file uses directives like Disallow: to block access to specific paths and Allow: to permit access. For example, a site might block crawlers from accessing a /private/ folder to prevent sensitive data from appearing in search results.
For website owners, robots.txt helps control search engine traffic, saving bandwidth and preventing duplicate content issues. However, it’s not a security tool—malicious bots can ignore it. A poorly configured file might accidentally block important pages, hurting SEO. For instance, a misplaced Disallow: / would block all crawlers from the entire site.
A website’s robots.txt file might include <code>User-agent: * Disallow: /admin/</code> to prevent search engines from indexing login pages.
In other languages
See also: Sitemap