Blocking search engine bots from caching specific webpages

Every search engine before accessing/caching your page will look for a file named robots.txt file in your home directory. Example: http://hiox.org/robots.txt

If the file is not present the bot will cache every page it can scroll through..

We can add the robots.txt file and set specific like allowing only certain bot, disallwing certain file folders etc..

Some Example Follow:

First create a file named robots.txt in your home directory and proceed

To allow all robots complete access: Add the following code in robots.txt file


To exclude all robots from the server: Note we have used a forward Slash


To disallow all robots from accessing/caching a folder of a server:


When you use the above code, no robot will access/cache any file under http://sitename/foldername/ & http://sitename/example/ folders

To disallow all robots from accessing/caching a file of a server:


When you use the above code, no robot will access/cache http://sitename/foldername/filename.html file


Blocking a specific search engine or bot


The above code block Google bot. Just replace Googlebot with whatever search robot to block.

Example: If you want to block access to all bots other than the Googlebot, you can use the following syntax: