← All posts
SEO Basics·4 min read·July 1, 2026·Updated July 16, 2026

robots.txt Explained: How to Write One That Doesn't Hurt Your Rankings

A wrong robots.txt can block Google from your whole site. Here's what it does, how to write it correctly, and the common mistakes to avoid.

TC
Written by Tyler C., founder

What is robots.txt?

robots.txt is a plain text file at the root of your website (yoursite.com/robots.txt) that tells search engine crawlers which pages they're allowed to visit. Crawlers like Googlebot read this file before crawling your site.

It's one of the most commonly misconfigured files on the web. A single wrong line can accidentally block your entire site from Google indexing — and you won't know until you notice your traffic drop weeks later.

Basic robots.txt syntax

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Breaking this down:

  • User-agent: * — applies to all crawlers (Googlebot, Bingbot, etc.)
  • Disallow: /admin/ — blocks the /admin/ directory
  • Allow: / — explicitly allows everything else (good practice to include)
  • Sitemap: — tells crawlers where to find your sitemap (Google recommends including this)

What you should block

Disallow crawling of low-value URL spaces only when Google does not need to fetch them. Remember that robots.txt is not a reliable way to keep a URL out of search results:

  • /admin/ — CMS admin panels
  • /wp-admin/ — WordPress admin
  • /checkout/, /cart/ — selected e-commerce transaction URLs when your platform recommends it
  • /api/ — public crawl routes that do not serve indexable pages; secure private APIs with authentication
  • /search? — on-site search results pages (create duplicate content)
  • /thank-you — post-form submission pages
See this in action on your own site

Paste your URL for a free, instant SEO check — no signup required.

The most common robots.txt mistakes

Mistake 1: Blocking your entire site

User-agent: *
Disallow: /

This blocks everything. You'd be surprised how often this appears on live sites — often left over from development mode. If your site disappeared from Google overnight, check your robots.txt first.

Mistake 2: Blocking CSS and JavaScript files

Google renders pages like a browser. If you block your CSS or JS, Google can't see your page properly and may not rank it. Never disallow /static/, /assets/, or *.js paths unless you have a specific reason.

Mistake 3: Omitting the sitemap line

Adding Sitemap: https://yoursite.com/sitemap.xml to your robots.txt is a free win — it tells every crawler exactly where to find your page index without you having to submit it separately to each search engine.

Mistake 4: Using robots.txt to hide sensitive content

robots.txt is publicly visible. Anyone can read it by visiting yoursite.com/robots.txt. Never list sensitive paths thinking it will hide them — it actually advertises them. Use authentication or server-level access controls for genuinely private content.

Generate a correct robots.txt automatically

Peak Visibility generates a robots.txt tailored to your specific site structure as part of your full SEO package — along with your sitemap, meta tags, and schema markup. Paste your URL and download it in under 3 minutes. Free for 5 pages.

Sources and further reading

Technical claims in this guide were checked against primary documentation. Search features can change, so verify important implementation details before deploying.

See this in action on your own site

Paste your URL for a free, instant SEO check — no signup required.