hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wil - <>
Subject Re: filtering out crawlers
Date Thu, 10 Feb 2011 02:04:19 GMT

There are quite a few databases online with known robots. 
and comes to mind. The 
hardest part is figuring out the suspect robots which do not identify 

From: Cam Bazz <>
Sent: Tue, February 8, 2011 7:57:53 PM
Subject: filtering out crawlers


Is there a practical way to filter the logs left by crawlers like google?

They usually have user-agent strings like

Mozilla/5.0 (compatible; Googlebot/2.1; +
Mozilla/5.0 (compatible; bingbot/2.0; +

is there a database for these?

Best Regards,


View raw message