tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Crowther <>
Subject RE: howto stop crawler and bots according to their user agent string
Date Tue, 15 Jul 2008 10:13:14 GMT
> From: Mathias Walter []
> How can I prevent crawler and bots according to their user agent?
> I've put a robots.txt in webapps/ROOT, but this file is not
> read again.

So, to check, the crawlers are not reading your robots.txt and are crawling your site anyway?

> I'd like to stop crawlers by their useragent string.

What do you mean by "stop"?  Do you want to return 404s or similar when a request with a particular
user agent string is received?  If so, the obvious approach would be to write a Filter that
is placed in front of your webapp, or a Valve that is placed in the request processing chain,
that examines the user agent string in the request and returns an appropriate response if
you don't like the agent.

                - Peter

To start a new topic, e-mail:
To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message