Nginx block bad bots

I really love to learn and work with NGINX as it makes tough things very easier. I was struggling to provide 100% uptime for website hosted in my server, this is because of bot request which eats bandwidth and downs the websites by pushing crawling requests very frequently.

Its easier to block these bad bots with nginx. To identify the bad bots just analyze your nginx log file of the website. You might find the bots like Baiduspider,proximic,JikeSpider which request for webpages, some of those bots don't obey robots.txt file in which delays the request frequency of the bots.

To block these bots you can add the entry like below inside 'server { }' tag

if ($http_user_agent ~ "JikeSpider|proximic|Sosospider|Baiduspider|Sogou web spider") {
return 400;
}


Make a restart of nginx after adding the entry. You can view the logs of nginx and can find that the above bots get 400 response. This might also reduce load & reduce the bandwidth usage of your server. I was able to add this scrap as the above entry in nginx stopped load in my server and gave me free time to relax, you will also enjoy the same.