Check your website for If-Modified-Since HTTP header

What is The If-Modified-Since HTTP header?

This header function tells a search engine crawler whether the webpage has been changed or not with a 304 Status code.

You may see this status code in your statistics report.

The 304 status code in a log file tells the search engine spider that webpage has not been changed and does not need to be indexed again.

If you cannot see the status code 304, you may inquire your web host. All modern web servers supports this header.

The If-Modified-Since HTTP header is used with the GET method.

When a search spider request for a webpage that supports the header, whose variants has been changed since its last visit, the response will be as follows,

If-Modified-Since = "If-Modified-Since" ":" HTTP-date

i.e., If-Modified-Since: Mon, 02 Jul 2010 08:24:42 GMT

If these is no change in the webpage, when the spider crawls there will be no response from the server, instead a 304 status code will be returned.

In static pages the header is normally set up by the server itself, but, in the dynamic pages you may have to use some scripting language like PHP and set the header of your own.

Advantages:

The two major advantages of using this if-modified-since header is that,

The site owners can save a lot of bandwidth usage.

The search spiders may not waste its time crawling the pages that are not modified.

You may also make use of online tool to check whether your web servers supports the if-modified-since header.