Robots.txt is a text file that is placed on a website to instruct web crawlers and other automated agents about which pages or sections of the site should not be processed or scanned. This file, when properly configured, can help prevent these agents from accessing sensitive or private information, as well as prevent the website’s servers from becoming overloaded with unnecessary requests.
When a web crawler, such as Googlebot, visits a website, it first checks for the presence of a robots.txt file. If the file is present, the crawler will read the file and follow the instructions provided. The instructions in the file typically include which pages or sections of the website should not be accessed, as well as any specific rules or directives for the crawler to follow.
Any expert seo team should be sure that your website has a robot.txt file.
How Does Robots.txt Work?
When a web crawler, such as Googlebot, visits a website, it first checks for the presence of a robots.txt file. If the file is present, the crawler will read the file and follow the instructions provided. The instructions in the file typically include which pages or sections of the website should not be accessed, as well as any specific rules or directives for the crawler to follow.
Common Mistakes with Robots.txt
Here are some common mistakes that people make when creating and using robots.txt files:
- Not including a robots.txt file: If a website doesn’t have a robots.txt file, web crawlers will assume that they are allowed to access all pages on the site.
- Blocking important pages: If a website blocks important pages, such as the homepage or product pages, from being indexed, it can negatively impact the website’s search engine rankings.
- Not testing the file: Before publishing a robots.txt file, it is important to test it to ensure that it is working as intended.
In summary, Robots.txt is a simple text file that can be used to instruct web crawlers and other automated agents about which pages or sections of a website should not be accessed. This file can help prevent sensitive or private information from being accessed, as well as prevent the website’s servers from becoming overloaded with unnecessary requests. However, it’s important to use robots.txt carefully and not to block important pages from being indexed in order to maintain good search engine rankings. It’s also important to test the file before publishing it.
Robots.txt FAQ
What is a robots.txt file?
robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells search engine crawlers which pages or sections of the site they are allowed to crawl. It follows the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers (Googlebot, Bingbot) respect.
robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it, because Google can index the URL without crawling the content. To keep pages out of the index, use noindex meta tags or password protection instead. See Google’s robots.txt introduction for the full spec.
What should be in my robots.txt file?
At minimum, most sites should include a User-agent: * directive (applying rules to all crawlers), followed by Allow: / (permit crawling of the entire site) or specific Disallow rules for admin areas, staging environments, or duplicate content paths.
Include a Sitemap: line pointing to your XML sitemap URL, which helps crawlers discover all your indexable pages. Common Disallow patterns: /wp-admin/ for WordPress admin, /cart/ or /checkout/ for e-commerce checkout flows, /?filter= or similar for faceted navigation that creates duplicate URLs.
Can I block Google from crawling specific pages?
Yes, but consider whether blocking is actually what you want. Use robots.txt Disallow to prevent Google from crawling a page. Use a noindex meta tag in the page’s HTML to prevent indexing (which is usually what people mean when they say “block from Google”).
A common mistake: blocking a page in robots.txt AND adding a noindex tag. Google can’t see the noindex tag because it can’t crawl the page, so the URL may still get indexed based on inbound links. To reliably remove a page from search, allow crawling in robots.txt and use noindex, then wait for Google to re-crawl and process the noindex directive.
How do I test my robots.txt file?
Use Google Search Console’s robots.txt Tester (under Settings > Crawling) to validate syntax and test whether specific URLs are blocked or allowed. The tester shows exactly how Googlebot interprets your file, catching common mistakes like blocking your homepage or important assets.
Also check that your robots.txt doesn’t accidentally block CSS or JavaScript files. Google needs to render pages to evaluate them, and blocking rendering resources produces broken layout in the SERP snapshot Google uses for indexing decisions. See our GSC glossary entry for the full diagnostic workflow.
Further reading
Authoritative sources: Google Search Central on robots.txt, Google Search Central on canonicalization. Related glossary entries: Google Search Console, canonical tag.