Bans in robots.txt
by Geethalakshmi[ Edit ] 2010-02-27 21:23:16
Bans in robots.txt
If indexing of a considerable portion of the site is banned, this is likely to affect the nonbanned part as well because spiders will come less frequently to a “noindex” site.