Block search engines google, yahoo indexing specific folder using robot.txt
by rajesh[ Edit ] 2012-08-16 13:03:35
To block the search engines including google and yahoo can be done using a single file named robot.txt
create a filenamed robot.txt in the websites home folder. The file should be accessible like http://hiox.org/robot.txt
Inside the file add the following code to block the folder
User-agent: *
Disallow: /folder-name/
you can replace the folder-name with your folder name that should not be indexed by the search engines.
we can have multiple such blocks as
User-agent: *
Disallow: /folder-name1/
Disallow: /folder-name2/