What are the max. page limits a Search Engine can crawl?
by Geethalakshmi[ Edit ] 2010-07-18 12:59:35
What are the max. page limits a Search Engine can crawl?
One of the most asked question in forums and in message boards what is the maximum depth a search engine can crawl for a page. What is the maximum size Of html /pdf that can indexed by top search engines.
1. For google: As per our latest research Google has a maximum crawl and cache depth of 1 MB only ( excluding images / graphics ) . It Used to be 100 kb then they increased to 250 kb then to 500kb and the latest update is 1 MB per file.
2. Yahoo overtakes Google by a long way, Their indexing and caching limit is 5 MB
3. MSN Search engine: Its very unpredictable for MSN but from our experiment MSN can cache upto 3 MB, we never tested about that probably someone in their search quality team can answer that.