SEOLyzerPro

Robots.txt Explained: What It Does and 5 Mistakes to Avoid

By Safdar Waraich | Last updated: 30 October, 2026

Robots.txt is one of the smallest files on your website, and one of the easiest to get wrong. A single line in it can stop search engines from visiting your whole site. A wrong rule can hide important pages, or give a false sense of security about pages that should stay private.

This guide explains what robots.txt is, how it works, how to write it, and the five mistakes that cause the most trouble. You will also see a WordPress example and learn how to test your file before it causes damage.

What is robots.txt?

Robots.txt is a plain text file placed in the root folder of your website. It gives instructions to web crawlers, the automated programs (also called bots or spiders) that visit pages for search engines and other services. The instructions tell them which parts of your site they may request and which they should stay away from.

For a site at https://www.example.com, the file must be available at:

https://www.example.com/robots.txt

Anyone can open this address in a browser and read it, including competitors and people with bad intentions. Remember this, because it matters for one of the mistakes below.

What robots.txt does, and what it does not do

This is the most important idea in the article, so read it carefully.

Robots.txt controls crawling. It asks well-behaved bots not to request certain addresses.

It does not control indexing. Indexing means that a search engine stores a page and can show it in the results. A page that is blocked in robots.txt can still appear in results if other websites link to it. The listing may have no description, because the search engine could not read the page, but the address can still be shown.

It is not security. Reputable search engines follow the file, but it is only a request. Bad bots can ignore it, and listing a private folder in robots.txt actually tells everyone where it is. Private content needs a password or server permissions.

To keep a page out of search results, you normally use a noindex instruction in the page itself, and you must let the page be crawled so that the instruction can be seen.

The basic syntax

A robots.txt file is made of groups of rules. Each group starts with the crawler it applies to, followed by what that crawler may or may not request.

Two useful pattern characters are supported by Google: the asterisk * matches any sequence of characters, and the dollar sign $ marks the end of an address. For example, Disallow: /*.pdf$ blocks addresses that end in .pdf. Lines that start with # are comments.

Example files

1. Allow everything (a good starting point for most small sites):

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

An empty Disallow: means that nothing is blocked.

2. Block one folder:

User-agent: *
Disallow: /private-test/

Sitemap: https://www.example.com/sitemap.xml

3. A typical WordPress file:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap.xml

The admin area is blocked, but the file admin-ajax.php stays available, because some themes and plugins use it on the public side of the site. Many WordPress sites create a similar file automatically.

4. Rules for one specific crawler:

User-agent: Googlebot-Image
Disallow: /private-images/

You can build files like these with the Robots.txt Generator, which helps you avoid typing errors.

A few rules that trip people up

• Paths are case sensitive. /Photos/ and /photos/ are different.
• The file applies to one host. A robots.txt file at www.example.com does not cover blog.example.com. Each subdomain needs its own file. The protocol (http or https) and the port also count.
• The most specific rule wins. When rules conflict, Google follows the one that matches the longest path. If an Allow and a Disallow match equally, the less restrictive one (Allow) is used.
• File size matters. Google reads only the first 500 KiB of the file. Most sites never come near this limit.
• Crawl-delay is ignored by Google. Some other search engines read it, but Google does not.

Now let us look at the five mistakes.

Mistake 1: leaving “Disallow: /” on a live site

This is the most dangerous line in the file:

User-agent: *
Disallow: /

It tells all crawlers not to visit any page. Developers often use it on a test or staging site to keep it out of search, and then forget to remove it when the site goes live. Some WordPress settings can also produce a similar effect (for example the option to discourage search engines from indexing the site). The result can be a sharp fall in visits, as new pages are not discovered and existing pages may be dropped over time.

How to avoid it: check your robots.txt on launch day, after a migration, and after any redesign. If your traffic suddenly drops. look at this file first. First check robots.txt, examine carefully and fix problems accordingly.

Mistake 2: blocking CSS, JavaScript and images

Years ago, it was common to block the folders that contain styles and scripts. Today, search engines render pages in a way similar to a browser. If they cannot load your CSS and JavaScript, they may not see your layout correctly, and may judge a page as hard to use on mobile. Blocking image folders can also keep your pictures out of image search.

How to avoid it: do not block /wp-content/, /assets/, /css/, /js/ or your image folders unless you have a clear reason. If one part of a blocked folder is needed, add an Allow rule for it. You can check how Google sees a page with the URL Inspection tool in Google Search Console.

Mistake 3: using robots.txt to hide pages or to “noindex” them

There are two versions of this mistake.

Hiding private content. Because anyone can read robots.txt, a line such as Disallow: /secret-reports/ is like putting a sign on a door that says “do not look here”. Use proper protection instead: a login, a password, or server access rules.

Trying to remove pages from search. Blocking a page does not remove it from the index. If the page is already indexed, or others link to it, it can stay in the results as a bare listing. A Noindex line inside robots.txt does not work either, because Google stopped supporting it in 2019. The right method is to place a noindex meta tag in the page (or send it in the HTTP header) and keep the page crawlable until Google has seen it and dropped the page. After that, you can decide whether blocking is still needed.

How to avoid it: ask yourself whether you want the page not to be crawled, or not to be shown in search. They are different goals with different tools.

Mistake 4: a wrong name, place or format

Small details break the file silently:

• Wrong name or case. It must be exactly robots.txt, in lower case. Robots.txt or robots.TXT may not be found.
• Wrong location. It must be in the root, not in a subfolder such as /files/robots.txt.
• Wrong format. It should be a plain text file, saved in UTF-8. A file saved from a word processor can contain hidden characters or formatting. Use a plain text editor.
• Wrong host. If your site uses https://www.example.com, make sure the file loads at that exact address, and that the other versions (http, non-www) redirect to it.
• Typing errors. Dissallow or a missing colon means that the rule is ignored.

Also keep in mind how crawlers treat the server response. If the file does not exist (a 404 error), crawlers usually assume that there are no restrictions. If the server returns repeated errors in the 500 range, Google may pause crawling your site for a while, because it cannot tell what is allowed. Use the HTTP Status Code Checker to confirm that your robots.txt returns a normal 200 status.

How to avoid it: open yourdomain.com/robots.txt in a browser after every change, and read it line by line.

Mistake 5: outdated or conflicting rules after changes

A site changes over time: you move to a new domain, switch from http to https, change your folder structure or install a new plugin. The robots.txt file does not update itself, so old rules can stay behind.

Typical problems:

• A rule that blocks a folder which now contains important pages.
• A Sitemap line that points to an old address.
• Two rules that conflict, so that the result is not what you expected.
• A pattern that is too wide. For example, Disallow: /tools also blocks /tools-and-guides/ and /tools.html, because it matches everything that begins with those letters. If you mean only the folder, write Disallow: /tools/.

How to avoid it: review the file every few months and after every major change, and keep a short note of why each rule exists. If you cannot explain a rule, find out before you delete or keep it.

Robots.txt and ads

If you plan to use Google AdSense, remember that Google needs to be able to read your pages to show relevant ads. Google’s AdSense crawler is called Mediapartners-Google. Do not add a rule that blocks it, and make sure that your important pages are not blocked for it. Check Google’s current AdSense documentation if you are unsure, because crawler rules can change.

How to create a robots.txt file

1. Decide what you really need to block. For many small sites, the answer is only the admin area, or nothing.
2. Open the Robots.txt Generator, choose your rules and add your sitemap address.
3. Copy the text, and save it in a plain text editor as robots.txt.
4. Upload it to the root folder of your site, using your file manager or FTP. If you use WordPress with an SEO plugin, the plugin may let you edit the file from the dashboard, so check there first to avoid creating two versions.
5. Open the address in your browser and confirm that it shows what you expect.

How to test your robots.txt

Never publish a change and hope for the best. Test it:

1. Open the file in a browser. It should load as plain text.
2. Check the status. Use the HTTP Status Code Checker to confirm a 200 response.
3. Use Google Search Console. The robots.txt report shows whether Google can fetch your file and whether it found any problems. The tools in Search Console change from time to time, so look for the “robots.txt” report in Settings.
4. Inspect important URLs. The URL Inspection tool tells you if a page is “blocked by robots.txt”. Test your homepage, your main tools and a few articles.
5. Check again after a few days. Look at the crawling and indexing reports for any new blocked pages.

A short checklist

• Is Disallow: / absent from the live file?
• Are CSS, JavaScript and image folders accessible?
• Am I using noindex (not robots.txt) to keep pages out of search?
• Is the file named robots.txt, in the root, in plain text?
• Does the Sitemap line show the correct, current address?
• Are my patterns specific enough (with a trailing slash for folders)?
• Have I tested the file and a few important URLs?

Common questions

Do I need a robots.txt file?
It is not required. Without one, crawlers assume they may visit everything. A file is useful when you want to guide them or point to your sitemap.

Will robots.txt improve my rankings?
Not directly. It helps crawlers spend time on useful pages, which supports good site management.

Should I block my login page?
You may, but robots.txt does not secure it. Use a strong password and other protection.

Can I block a single page?
Yes, with a rule such as Disallow: /page-name/. Remember that this does not remove it from search results.

How long do changes take to work?
Search engines keep a copy of the file for a while, so allow some time, often up to a day, for a change to be picked up.

Final thoughts

Robots.txt is simple, but it deserves attention. Keep it short, block only what you truly need to block, never use it as a security tool, and test every change. If you remember one thing, remember this: robots.txt controls crawling, not indexing.

Take two minutes today to open your own /robots.txt, read it line by line, and compare it with the checklist above. If you need a clean starting point, create one with the Robots.txt Generator.