robots.txt: The Complete Guide for Modern SEO
Everything you need to know about robots.txt — syntax, common rules, mistakes to avoid, and when to use noindex instead.
Sarah Mitchell
Senior SEO Strategist
What Is robots.txt?
robots.txt is a plain text file at the root of your domain (/robots.txt) that tells search engine crawlers which parts of your site they can and cannot access.
It's one of the first things Googlebot checks when it visits a domain, before it crawls any individual page.
Syntax Basics
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public-api/
Sitemap: https://example.com/sitemap.xml
Key directives:
User-agent: specifies which crawler the rule applies to (*= all)Disallow: paths the crawler should not accessAllow: overrides a Disallow for specific pathsCrawl-delay: seconds between requests (not supported by Google)Sitemap: the URL of your XML sitemap
What robots.txt Does NOT Do
This is the most misunderstood aspect of robots.txt.
Disallowing a URL does not prevent it from being indexed.
If other pages link to a disallowed URL, Google may still index it — it just won't crawl the page's content. You'll see it appear in search results with "No information is available for this page."
To prevent indexing, use:
<meta name="robots" content="noindex">in the page's<head>- The
X-Robots-TagHTTP header
Common robots.txt Rules
Block admin panels:
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login
Block AI scrapers:
User-agent: GPTBot
Disallow: /
User-agent: Claude-Web
Disallow: /
Block staging content:
User-agent: *
Disallow: /staging/
Disallow: /preview/
Block search result pages:
User-agent: *
Disallow: /search?
Testing Your robots.txt
Use Google Search Console → robots.txt Tester to check whether specific URLs are blocked before deploying changes.
Or use our Robots.txt Generator to build and preview your file visually.
Common Mistakes
- Blocking CSS and JS files — Google needs to render pages to assess quality; blocking assets hurts Core Web Vitals scores in Search Console
- Disallowing your sitemap — common in WordPress setups, makes it harder for crawlers to discover pages
- Thinking it protects sensitive data — robots.txt is publicly readable; never treat it as a security layer
- Missing the trailing slash —
Disallow: /adminmatches the path/admin;/admin/matches the directory and everything inside it
Crawl Budget Considerations
For large sites (10,000+ pages), crawl budget — the number of pages Google crawls per day — matters. robots.txt helps by directing Googlebot away from low-value pages (paginated archives, faceted navigation) so it spends more time on content that should rank.
Signs of a crawl budget problem:
- New pages take weeks to appear in search
- Large portions of your site aren't indexed
- Google Search Console shows crawl errors
Use the robots.txt Disallow directive + noindex meta tags together to manage budget efficiently.