+3 votes
in Web & Google by (75.0k points)
In the log file of my website, I see many bad bots and crawlers accessing my website. I have disallowed some of them in the robots.txt, but it seems that they are not obeying robots.txt. How can I block them using the .htaccess file?

1 Answer

+3 votes
by (353k points)
selected by
 
Best answer

You can use the HTTP_USER_AGENT to check those bots or spiders and block them. Here I am blocking some of the well-known bad bots. When these bots will try to access the page, 403 will be returned.

## block these bots
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|SemrushBot|Barkrowler|MJ12bot|DotBot|Java|DataForSeoBot|BLEXBot) [NC]
RewriteRule .* - [F,L]


...