robots.txt

abbreviation robots.txt

robots exclusion protocol

Robots.txt is a simple text file placed in a website’s root directory to instruct web crawlers (like Googlebot) which pages or files they should or shouldn’t access. It’s part of the Robots Exclusion Protocol, a standard followed by most search engines.

When a crawler visits a site, it checks for a robots.txt file before scanning. The file uses directives like Disallow: to block access to specific paths and Allow: to permit access. For example, a site might block crawlers from accessing a /private/ folder to prevent sensitive data from appearing in search results.

For website owners, robots.txt helps control search engine traffic, saving bandwidth and preventing duplicate content issues. However, it’s not a security tool—malicious bots can ignore it. A poorly configured file might accidentally block important pages, hurting SEO. For instance, a misplaced Disallow: / would block all crawlers from the entire site.

A website’s robots.txt file might include <code>User-agent: * Disallow: /admin/</code> to prevent search engines from indexing login pages.

See also: Sitemap

Découvrez nos autres services

ElpisIA, c’est tout un univers — explorez nos autres services.