What Is Robots.txt in SEO?

Robots.txt is one of those small technical files that most small business owners never touch, until it silently kills their SEO. I once audited a Pune dental clinic that had been invisible in Google for months. Their agency had shipped a website with the line "Disallow: /" in the robots.txt, which told Google to skip the entire site. Removing that single line brought all their pages back into the index within 10 days. Rankings followed a month later.

This guide walks through what robots.txt actually is, how it works, exactly how to write one for a typical small business site, the common mistakes that hurt SEO, and how to test your own file. Real examples throughout so you can copy patterns straight into your site.

What is robots.txt?

Robots.txt is a plain text file, always named exactly robots.txt (lowercase), placed at the root of your website. For any site, the file lives at yoursite.com/robots.txt. It cannot live in a subdirectory. It is not an HTML file. It has no styling. It is just text following a specific set of rules.

The file is part of the Robots Exclusion Protocol, an informal standard that dates back to 1994. When a search engine crawler (Googlebot, Bingbot, DuckDuckBot, and so on) visits your site, one of the first things it does is fetch yoursite.com/robots.txt. It reads the rules inside, understands which parts of the site it is allowed to crawl, and only then requests pages.

The rules in robots.txt speak to individual "user agents" (the technical name for a crawler). You can write rules for all crawlers at once (User-agent: *) or target specific ones (User-agent: Googlebot, User-agent: Bingbot).

Well-behaved crawlers respect the rules. Bad bots ignore them entirely. Which means robots.txt is fine for controlling legitimate search engines, but it is useless for blocking scrapers, spam bots, or hackers.

Why robots.txt matters for SEO

Three practical reasons every business site should have a properly configured robots.txt.

Crawl budget management

Google allocates each site a "crawl budget," the number of pages it will fetch per visit. For most small sites, this is not a constraint. For larger sites (thousands or millions of pages), it very much is. Robots.txt keeps Google from wasting crawl budget on admin pages, cart URLs, faceted search variants, and other low-value URLs so it can spend that budget on your important content instead.

Preventing accidental indexing

Every site has pages that should not appear in search results: staging environments, admin dashboards, thank-you pages after form submissions, internal search results, checkout flows. Robots.txt (combined with noindex where appropriate) keeps these out of Google.

Sitemap discovery

Adding a Sitemap directive to your robots.txt tells search engines exactly where to find your XML sitemap. This speeds up discovery of new pages. Google Search Console also lets you submit sitemaps directly, but declaring them in robots.txt is a lightweight extra signal.

For a Pune restaurant client, their robots.txt was blocking Googlebot from crawling their menu pages by mistake (a bad Disallow rule). Fixing it brought their menu pages back into search within two weeks and lifted overall traffic by 40 percent within a month. This kind of quiet fix is why technical SEO reviews matter. See my technical SEO basics for business owners for the full checklist.

How robots.txt is structured

Robots.txt uses three main directives. Each block starts with a User-agent line, followed by one or more Disallow or Allow lines, and optionally a Sitemap declaration.

User-agent

Specifies which crawler the following rules apply to. Common values:

• User-agent: * - applies to all crawlers
• User-agent: Googlebot - only Google's main crawler
• User-agent: Bingbot - only Bing's crawler
• User-agent: Googlebot-Image - only Google's image crawler

Disallow

Tells the specified crawler not to fetch URLs matching the pattern. Examples:

• Disallow: / blocks the entire site (dangerous)
• Disallow: /admin/ blocks everything in the /admin/ folder
• Disallow: /cart blocks any URL starting with /cart
• Disallow: (empty value) means allow everything

Allow

Explicitly permits crawling of URLs that would otherwise be blocked by a broader Disallow rule. Useful for exceptions. Example: block /wp-admin/ but allow /wp-admin/admin-ajax.php because plugins need it.

Sitemap

Declares the location of your XML sitemap. Full URL required:

Sitemap: https://yoursite.com/sitemap.xml

A typical small business robots.txt

Here is a clean, safe robots.txt I use as a starting template for most small business WordPress sites:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?s=
Disallow: /search
Disallow: /cart
Disallow: /checkout
Disallow: /my-account
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap.xml

What this does:

• Applies to all crawlers (User-agent: *)
• Blocks WordPress admin pages so crawlers do not waste time there
• Blocks internal search result pages (they have no unique content)
• Blocks cart, checkout, and account pages (user-specific, not for public search)
• Explicitly allows admin-ajax.php because plugins and themes need it
• Tells search engines where to find the XML sitemap

Adapt the paths to match your site. If you use WooCommerce, the cart and checkout blocks are already correct. If you use a custom CMS or Shopify, the exact paths differ. Look at your URL structure and block only URLs that should not be indexed.

Common robots.txt mistakes I see in audits

Accidentally blocking the entire site

The classic disaster:

User-agent: *
Disallow: /

This tells every crawler to skip everything. If your site launches with this rule (often left over from staging), your pages will not appear in search until you fix it. I have seen this exact mistake on 4 different Pune client sites over the past 2 years. Always check yoursite.com/robots.txt after a new site launch.

Blocking CSS and JavaScript files

Some outdated tutorials suggest blocking /wp-content/ or /wp-includes/ entirely. This prevents Google from rendering your pages properly, because Google needs to load your CSS and JS to understand how the page actually looks. Rendering issues hurt your rankings, especially on mobile. Modern Google guidance is to allow crawling of all resources needed to render the page.

Using robots.txt to hide private content

Robots.txt is public. Anyone can read yoursite.com/robots.txt. Listing your private admin login path there advertises its existence to attackers. For real security, use authentication, not robots.txt. Use robots.txt only for crawl control on genuinely low-value URLs.

Trying to noindex via Disallow

Common misconception: "If I want a page removed from Google, I will Disallow it in robots.txt." This does not work as expected. If Google has already indexed the page and you then Disallow it, Google cannot crawl the page to see any noindex tag. The page can stay in the index as a bare URL with no description. To fully remove a page, use a noindex meta tag on the page (not a Disallow in robots.txt).

Using wildcards incorrectly

Robots.txt supports two wildcards: * (matches any characters) and $ (matches end of URL). Using them wrong can accidentally block way more than you meant. Example: Disallow: /*.pdf$ blocks all PDF files. Disallow: /*.pdf without the $ would also block PDFs but might match other URLs containing .pdf in them (like /report.pdf/details).

Case sensitivity errors

Robots.txt is case-sensitive on the URL side. Disallow: /Admin does not block /admin. Match your actual URL structure exactly, including case.

Forgetting the Sitemap directive

Not fatal, but a missed opportunity. Every robots.txt should end with a Sitemap line pointing to your XML sitemap. This makes discovery instant for any crawler that reads the file.

Blocking after a redesign or migration

When you migrate a site or launch a redesign, staging environments often use Disallow: / to prevent early indexing. If that file gets deployed to production by mistake, your live site becomes invisible. Always confirm the production robots.txt is correct on the day of launch.

Robots.txt vs noindex vs canonical: when to use which

These three tools all deal with what appears in search results, but they work differently.

Robots.txt (crawl control)

Use robots.txt to prevent crawlers from fetching pages at all. Useful for saving crawl budget on truly low-value URLs. The page can still be indexed if it is linked from elsewhere, but Google will not have the page content to display, so it will show a bare URL.

Noindex meta tag (indexing control)

Use noindex on a page you want removed from search results but you still want crawlers to see. Google fetches the page, reads the noindex directive, and drops it from the index. Add it in the head:

<meta name="robots" content="noindex, follow">

Important: for noindex to work, Google must be able to crawl the page. Do not combine noindex with Disallow. If you Disallow a page in robots.txt, Google never sees the noindex.

Canonical tag (duplicate content control)

Use rel="canonical" when the same content is accessible via multiple URLs and you want to tell Google which URL is the primary version. This consolidates ranking signals to one URL instead of splitting them across duplicates. Add it in the head:

<link rel="canonical" href="https://yoursite.com/preferred-url">

Decision framework:

• Want to save crawl budget on low-value pages? Use robots.txt Disallow.
• Want a page fully removed from search? Use noindex meta tag.
• Have duplicate URLs and want to consolidate them? Use canonical.
• Want to hide private content from anyone? Use authentication, not any of these.

How to create or edit your robots.txt

WordPress with an SEO plugin

Yoast SEO, Rank Math, and All in One SEO all have robots.txt editors built in. In Yoast, go to SEO > Tools > File editor. Rank Math has it under Rank Math > General Settings > Edit robots.txt. These editors save the file to the right location automatically.

WordPress without a plugin

Create a plain text file called robots.txt in the root of your WordPress install (same folder as wp-config.php). Upload it via FTP or your hosting file manager. WordPress will serve this static file instead of its default virtual one.

Custom or static sites

Create a plain text file called robots.txt in the same folder as your homepage HTML. Upload via FTP or Git. Confirm it is accessible at yoursite.com/robots.txt.

Shopify, Wix, Squarespace

These platforms manage robots.txt for you. Shopify recently added a robots.txt.liquid file for editing. Wix and Squarespace have limited robots.txt customization inside their SEO settings.

How to test your robots.txt

1. Visit yoursite.com/robots.txt in your browser. Read the actual content. Does it match what you expect? Any lines you do not understand?
2. Use Google Search Console's URL Inspection tool. Enter a specific URL. If robots.txt is blocking it, the report will say so under "Crawl allowed."
3. Use the robots.txt Tester in Search Console (available in older versions of the tool). Paste your file and test whether specific URLs would be blocked or allowed.
4. Check with third-party tools like technicalseo.com/tools/robots-txt/ for a syntax check.
5. Run my free SEO checker on your homepage. If your site is being blocked by robots.txt, the checker will flag major crawlability issues immediately.
6. Compare against local competitors using my website authority checker to spot whether crawl issues are dragging your rankings behind theirs.

Robots.txt for specific business situations

E-commerce sites

Block user-specific URLs (cart, checkout, my-account, wishlist) and internal search result pages. Consider blocking filter and sort URL variations that create thousands of near-duplicate pages.

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /my-account
Disallow: /*?filter=
Disallow: /*?sort=
Disallow: /*?search=
Sitemap: https://yoursite.com/sitemap.xml

Local service business

Small clean robots.txt. Just block admin and internal search, allow everything else.

User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xml

Blog or content site

Block admin, internal search, tag archives if they are thin, and any staging environments.

Multi-region site

If you have separate sections for different countries or languages, do not use robots.txt to control language targeting. Use hreflang tags instead, and only block truly non-public sections.

Related technical SEO guides

• Technical SEO basics for business owners — the full technical checklist.
• What is crawl budget — how Google decides how many of your pages to fetch.
• Schema markup guide — structured data that helps search engines understand your content.
• Free website SEO audit guide — how to evaluate your full site including robots.txt.
• Free on-page SEO checklist for 2026 — the full on-page fundamentals.

Frequently asked questions

What is robots.txt in SEO?

Robots.txt is a plain text file at the root of your website (like yoursite.com/robots.txt) that tells search engine crawlers which pages or sections of your site they can and cannot crawl. It is one of the first files search engines request when they visit your site. A well-written robots.txt keeps crawlers focused on your important pages and prevents them from wasting time on admin panels, checkout flows, or duplicate URLs. A misconfigured one can accidentally block your entire site from Google.

Does every website need a robots.txt file?

Not strictly, but every serious site should have one. Without it, search engines assume they can crawl everything, which is usually fine for small sites but wastes crawl budget on larger ones. A basic robots.txt with at minimum a Sitemap directive helps search engines discover your XML sitemap faster. For a Pune clinic client whose robots.txt accidentally had Disallow: / (blocking everything), removing that single line brought their pages back into Google's index within 10 days after months of invisibility.

What is the difference between robots.txt and noindex?

Robots.txt controls crawling, noindex controls indexing. Robots.txt tells crawlers not to fetch a page at all. Noindex (a meta tag on the page itself or an HTTP header) tells search engines they can crawl the page but should not include it in search results. These are not interchangeable. If you Disallow a page in robots.txt, Google cannot see the noindex tag on it, and the page may still appear in search results as a bare URL with no description. Use noindex for pages you want out of the index.

Can robots.txt hide sensitive content?

No. Robots.txt is a public file that anyone can read at yoursite.com/robots.txt. Listing a private page there advertises its existence to the world. If you need to keep content private, use proper authentication (login required, password protection, IP restriction) or noindex meta tags. Robots.txt only politely asks well-behaved bots not to crawl. Bad bots ignore it entirely. For real security, robots.txt is the wrong tool. Use it for crawl control, not privacy.

How do I test my robots.txt file?

Visit yoursite.com/robots.txt in your browser to see the current file. Then use Google Search Console's URL Inspection tool to test whether specific pages are blocked. Google also provides a robots.txt tester inside older versions of Search Console. Run your homepage through my free SEO checker to catch any accidental blocking. For a full picture including how competitors set up crawl access, check domain authority at my website authority checker.

Your next step

Open a new browser tab and visit yoursite.com/robots.txt. Read what is there. If you see Disallow: / on its own, that is your entire site being blocked. Fix it today. Then run my free SEO checker to confirm your pages are crawlable and to catch any other technical issues holding back your rankings.