httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Reser <>
Subject Re: URL scanning by bots
Date Fri, 03 May 2013 04:24:43 GMT
On Tue, Apr 30, 2013 at 5:23 PM, André Warnier <> wrote:
> Alternatives :
> 1) if you were running such a site (which I would still suppose is a
> minority of the 600 Million websites which exist), you could easily disable
> the feature.
> 2) you could instead return a redirect response, to a page saying "that one
> was sold, but look at these".
> That may be even more friendly to search engines, and to customers.

My point isn't that there aren't alternatives, but that 404's are
legitimate responses that legitimate users can be expected to receive.
 As such you'll find it nearly impossible in my opinion to convince
people to degrade performance for them as a default.  If it isn't a
default you're hardly any better off than you are today since it will
not be widely deployed.

If you want to see a case where server behavior has been tweaked in
order to combat miscreants go take a look at SMTP.  SMTP is no longer
simple, largely because of the various schemes people have undertaken
to stop spam.  Despite all these schemes, spam still exists and the
only effective counters has been:
1) Securing open-relays.
2) Removing the bot-nets that are sending the spam.
3) Ultimately improving the security of the vulnerable systems that
are sending the spam.

All the effort towards black lists, SPF, domainkeys, etc... has been
IMHO a waste of time.  At best it has been a temporarily road block.

View raw message