Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (nike.apache.org: domain of prvs=083691ffa1=micha@lenk.info
 designates 188.40.85.232 as permitted sender)
Message-ID: <51855730.2010100@lenk.info>
Date: Sat, 04 May 2013 20:45:04 +0200
From: Micha Lenk <micha@lenk.info>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:10.0.12) Gecko/20130116 Icedove/10.0.12
MIME-Version: 1.0
To: dev@httpd.apache.org
Subject: Re: URL scanning by bots
References: <517F96F0.3090004@ice-sa.com>
 <0028CAB3-663E-4FEF-B52C-0643D493167C@sharp.fm> <51804F41.90707@ice-sa.com>
 <CADkdwvS1TQ8aCpP428SYOCqmCaF2KcsyNg=Ofqeeqk5edtPU7g@mail.gmail.com>
 <5180606F.30105@ice-sa.com>
 <CADkdwvS8fu=U2MTtM3M31goYZ7ig=7YovDq_gtOfwtvGMZDYJg@mail.gmail.com>
 <51837B65.1030309@yuhu.biz> <A2C3AB3F-EF56-4331-AFA7-C20A919B1832@bbc.co.uk>
In-Reply-To: <A2C3AB3F-EF56-4331-AFA7-C20A919B1832@bbc.co.uk>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

Am 03.05.2013 11:27, schrieb Dirk-Willem van Gulik:
> FWIIW - the same sentiments where expressed when 'greylisting[1]' in
> SMTP came in vogue. For small relays (speaking just from personal
> experience and from the vantage of my own private tiny MTA's) that
> has however not been the case. Greylisting did dampen things
> significantly - and the effect lasts to this day.

The main difference I see here is, that a SMTP server that uses
greylisting really can close the client connection almost immediately
with keeping minimal state, usually on cheap disk. So, until the client
retries, neither the kernel nor any processes have to deal with the
greylisting during the delay period.

In HTTP this is totally different. You can't just return a temporary
error code and assume that web browser will retry some reasonable
moments later. For this reason you would have to delay the real HTTP
response. And this has a substantial resource usage impact, as you have
to maintain state across all operating system layers. The network stack
needs to maintain the TCP connection open, the kernel needs to maintain
an open socket for the server process, the server process needs to
maintain an (some kind of) active HTTP request -- for every single
delayed request. These resources would just wait for the delay timer to
expire, so essentially hanging around without doing anything useful, and
without changing any outcome of the actual HTTP transaction. As others
already pointed out, this opens the doors for denial of service attacks
by excessive resource usage. From a security point of view, you
definitely don't want such HTTP response delays.

Regards,
Micha