{"id":7040,"date":"2022-08-31T12:32:11","date_gmt":"2022-08-31T12:32:11","guid":{"rendered":"https:\/\/5minutesseo.com\/?p=7040"},"modified":"2022-09-07T09:27:51","modified_gmt":"2022-09-07T09:27:51","slug":"how-to-use-robots-txt-to-block-subdomain","status":"publish","type":"post","link":"https:\/\/5minutesseo.com\/seo\/how-to-use-robots-txt-to-block-subdomain\/","title":{"rendered":"How to Disallow Sub-Domain From robots.txt So That It Doesn\u2019t Appear in Search Results?"},"content":{"rendered":"\n

Want to block a sub-domain from robots.txt so that search engines don’t crawl it?<\/strong><\/p>\n\n\n\n

Today I’ll show you how to use robots.txt to block sub-domain. <\/p>\n\n\n\n

There is a simple way to remove the entire sub-domain or selected directories on the subdomain from search results.<\/strong><\/p>\n\n\n\n

I have also used these techniques to disallow multilevel subdomains. But for multilevel subdomains, the trick is a bit different!<\/p>\n\n\n\n

So, whether you want to disallow one of your entire subdomains or a particular folder, you can follow this guide. <\/p>\n\n\n\n

What Is a robots.txt File?<\/strong><\/h2>\n\n\n\n

A robots.txt file is a unique text file within your website’s root directory that instructs search engine crawlers on which pages of the website they should crawl and which pages they should not crawl.<\/strong><\/p>\n\n\n\n

These crawl instructions are defined by “allowing” or “disallowing”<\/strong><\/em> specific web crawling behavior. It is used to avoid overloading your website with requests.<\/p>\n\n\n\n

And, making sure only relevant pages are indexed & confidential pages (like pages that have login credentials of the website) are not indexed in SERPs<\/p>\n\n\n\n

Now let’s see…<\/p>\n\n\n\n

How to Use robots.txt to Block Search Engine Crawlers<\/strong> from Accessing the Subdomain<\/h2>\n\n\n\n

If you don’t want bots to index certain subdomains, you can block them using the robots.txt file. <\/p>\n\n\n\n

Step I: <\/strong>Open Notepad in your system & add the following code:<\/p>\n\n\n\n

User-agent: *
Disallow: \/<\/code><\/pre>\n\n\n\n
Step II: <\/strong>Save the file on your system with the name \u201crobots.txt\u201d.<\/p>\n\n\n\n
Step III: <\/strong>Now upload the saved robots.txt file to the root directory of each sub-domain you want to protect from search engine crawlers. <\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Ensure that you exclude the domains that you want to be crawled.<\/p>\n\n\n\n
Pro tip: You must create a robots.txt file for each subdomain you want to block from search engines. Google crawlers will look for the robots.txt file in each of the subdomains.<\/em><\/strong><\/p><\/blockquote>\n\n\n\n
FAQs<\/strong><\/h2>\n\n\n\n
1. How Do I Block a Domain With robots.txt<\/strong><\/h3>\n\n\n\n
If you want to block a domain with robots.txt use this code:<\/p>\n\n\n\n
User-agent: *
Disallow: \/<\/code><\/pre>\n\n\n\n
With this code, you can block the entire site.<\/p>\n\n\n\n
2. What Should You Block in the robots.txt File<\/strong><\/h3>\n\n\n\n
Are you wondering what should be disallowed in robots.txt? <\/p>\n\n\n\n
You can block unimportant style files, images, or scripts<\/strong> in the robots.txt file. You can also block URLs in robots txt to prevent Google from indexing expired special offers, private photos, or other pages that are not ready for users to access.<\/p>\n\n\n\n
3. What Does Block by Robots.txt Mean?<\/strong><\/h3>\n\n\n\n
Blocked by robots.txt means you have certain pages on your site that Google indexed but can’t crawl. Google can view these pages, but it won’t show them as part of search engine results pages.<\/p>\n\n\n\n
4. How Do I Remove the Subdomain From the Main Domain?<\/strong><\/h3>\n\n\n\n