httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hilton <jeremy-l...@adtcs.com>
Subject Re: [users@httpd] mod_throttle relaxing restrictions for crawlers
Date Thu, 27 Jan 2005 23:25:21 GMT
On 1/27/05 5:34 PM, "Manuel Lemos" <mlemos@acm.org> wrote:

> Hello,
> 
> on 01/27/2005 08:02 PM Jeremy Hilton said the following:
>>> I want to use mod_throttle to limit the rate of requests that can be
>>> done from each client IP.
>>> 
>>> However, I also need to allow the internal crawler, which is HT:Dig, to
>>> crawl the site without retrictions.
>>> 
>>> My problem is that I could not find a way to specify a different
>>> throttle policy for clients accessing from a given IP nor even from a
>>> given user agent.
>>> 
>>> Anybody has any ideas on how to solve this problem?
>> 
>> http://www.snert.com/Software/mod_throttle/
> 
> If you read me again, you may notice that I want to use mod_throttle but
> AFAIK there is no way to tell it to not throttle connections that come
> from a given IP. That is the problem.
> 
> I was wondering if there is a way to configure mod throttle in
> combination with mod_setenvif like I already do to split my logs, but I
> could not find a way to do it.

Since the throttle rules can be declared in the context of <VirtualHost>,
couldn't you setup a second virt host that has no throttle rules, and
instruct your crawler to use the "other" virt host?




---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message