Robots.txt uploading tips for blogger

The robots.txt file is used to control the crawlers activity on a website/blog. It will help you to keep some directories away from crawling while allowing some. For example if yu have two folders 1.Articles and 2.Javascripts – and if you wish to exclude Javascripts from crawling by robots, then you can command it on the robots.txt file.

A few basics about what the robots.txt file is -
- It is found in the root folder, Ex:-www.google.com/robots.txt
- It’s a text file and can be edited
- It is used to command the robots what to crawl and what not
- It is used to help the crawlers locate the sitemap on your site

If you are on blogger platform, then you can’t upload the robots.txt file. Panic not – there is another option which you can utilize. I’ll discuss it towards the end of this article. First let’s discuss a normal robots.txt implementation on a hosted site.

Specifying a sitemap with the Robots.txt file

Due to the recent agreement with the major search engines, they have come up with a common command that they will follow to detect sitemaps from robots.txt file. The command is –

Sitemap: Sitemap url here

Robots.txt for Blogger users.

Blogger users cannot upload the robots.txt file instead, they can use the robots meta tag to control the crawling of bots on particular files.

These codes should be included in the HEAD section of the particular page template.(Enclosed in arrow brackets)

META NAME=”ROBOTS” CONTENT=”NOINDEX”

This command will not index the current page in which this code is included.

META NAME=”ROBOTS” CONTENT=”NOFOLLOW”

This command will not follow/parse the links present on the particular page where this code is present in the head section.Blogger users can use this option to their advantage when making posts.If you want every new page to be crawled by the bots, include the following code to head section of your blogger template.