3 ways to stop your web pages being indexed

Here are 3 ways to stop a web page from being indexed:

1. Robots.txt

Before a search engine bot crawls your website, they will look for a text file at the root of your domain. The robots.txt should tell them what pages they are allowed to index when they crawl the site.

However, some robots may ignore the robots.txt. And it’s not a good idea to block ‘secret’ pages using robots.txt as anyone can access it at www.yourdomain.com/robots.txt and see what pages you’ve blocked.

2. ‘Noindex’ metatag

The noindex metatag looks like this:

<meta name=”robots” content=”noindex”>

Simply add this code in the section on any pages you don’t want indexed. However, you can only use this on html pages – you won’t be able to use the noindex tag to block out PDFs, Flash files or Word Documents etc.

Be aware that some bots will continue to index pages with the noindex metatag. And it’s not a reliable way to make links on a page ‘nofollow’.

3. Password protect pages

If you have pages that you only want certain people to access, password protecting them will restrict access to unwanted visitors and the search engine bots.

This is a good solution for restricting sections of your site that are in development.