httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hilton <>
Subject Re: [users@httpd] mod_throttle relaxing restrictions for crawlers
Date Thu, 27 Jan 2005 23:25:21 GMT
On 1/27/05 5:34 PM, "Manuel Lemos" <> wrote:

> Hello,
> on 01/27/2005 08:02 PM Jeremy Hilton said the following:
>>> I want to use mod_throttle to limit the rate of requests that can be
>>> done from each client IP.
>>> However, I also need to allow the internal crawler, which is HT:Dig, to
>>> crawl the site without retrictions.
>>> My problem is that I could not find a way to specify a different
>>> throttle policy for clients accessing from a given IP nor even from a
>>> given user agent.
>>> Anybody has any ideas on how to solve this problem?
> If you read me again, you may notice that I want to use mod_throttle but
> AFAIK there is no way to tell it to not throttle connections that come
> from a given IP. That is the problem.
> I was wondering if there is a way to configure mod throttle in
> combination with mod_setenvif like I already do to split my logs, but I
> could not find a way to do it.

Since the throttle rules can be declared in the context of <VirtualHost>,
couldn't you setup a second virt host that has no throttle rules, and
instruct your crawler to use the "other" virt host?

The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:> for more info.
To unsubscribe, e-mail:
   "   from the digest:
For additional commands, e-mail:

View raw message