Online Earning

How To Use Robots.TXT File

Thursday, 3 September 2015 by muntazir ali

How To Use Robots.TXT File 

Recently I published many tutorials on SEO’s topic on this blog as well as on My Youtube Channel. Many guys emailed Me & contacted me. Some of them appreciated my SEO efforts & many of them asked many questions about SEO. Few of them asked about Robots.txt file; that what is this file and what does it do?. Therefore today I created about Robots.txt file and describe it with examples. You’ll learn how to create this file and how to carefully use this file in English, if anything went wrong then it might very harmful for your website, so you should be doing things very carefully. However, in case something goes wrong, you can instantly remove this file from your website to avoid bad experience.

What is Robot.txt File?

Robot.txt file is a text file which instructs the search engines robots/crawlers/spiders which folder or page of any a website should not be accessed by them. You can disallow any search robot from some of your website’s specific folders or pages. They will not be able to access those pages which you’ve specified in the robots.txt file.

How to create a robots.txt file?

The syntax of creating this file is quite simple, just open notepad & save the file, give it a name as robots.txt and write

following two lines:

User-agent: *

Disallow: /

The above example is the simplest syntax for creating a robots.txt file, and the the first line “User-agent*” means that all crawlers/robots are allowed to access all the pages/folders of the website, however you can specify a folder/page in the below line “Disallow:/” you just need to write that page/folder after the / backslash.

if you wanna disallow search robots from any of your pages or folders then just add that page address after the back slash of Disallow line. Like this: Disallow: /kaka.html. Upload that txt file to your root directory of your website and you’re all done! in Blogger you can find this file by default by visiting setting page & then search preferences. Also if you want to disallow many pages or folders in the txt file then just do like this :

User-agent:*

Disallow:/images

Disallow:/plugins

Disallow:/wp-admin.php

Disallow:/login.php

In above txt file we disallowed some pages and folders of our site, so now Search robots/crawlers can not access them, and they will neither be indexed nor be crawled.

Hopefully, you have learned about the robots.txt file which is now a days very important in SEO, because correct usage of this file can improve your site/blog’s overall SEO, however, using this file in bad way can result your site/blog ignored by search engines.

Let me know if you got questions. Have a Nice time!

- See more at: http://popularearning.blogspot.com/2015/09/how-to-use-robotstxt-file.html#sthash.GIvRqGFS.dpuf