httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reindl Harald <>
Subject Re: URL scanning by bots
Date Fri, 03 May 2013 09:52:49 GMT

Am 03.05.2013 11:38, schrieb André Warnier:
> I agree that 404's are legitimate responses.
> And I agree that legitimate clients/users can expect to receive them.
> But if they do receive them when appropriate, but receive them slower than other kinds
of responses, this is not
> really "breaking the rules"

maybe you have not much expierience and not watching error-logs
on servers with some hundret domains

on our machines 99% of any web-apps is *carefully* written inhouse
there are always mistakes resulting in a lot of 404 which people
building templates and inclduing modules mostly not realize

this starts with no "favicon.ico" in the docroot while most
"modern" browsers try to access them without a link-tag and
so your first delay is on the homepage itself

looking at the crap-quality of the most common webapps the
situation is much more worse - well, this all does affect you
only if you have load and traffic on your machine but then
it hurts really

i had a large project 2 years ago where some hundret people
where in front of their machine with a webcam, the application
generated thumbnails which where shown on the page and cleanup
routines to get rid of the thumbs of no longer active users

we worked hard to optimize all the code to get as less as possible
404 errors while bot fill the disks, the overall connection count
was very very high while the braodcast show connected to the app
was on air, the load was very high but all ran smooth with 500
apache workers

with the proposal of this thread the server would not have
survived 10 minutes by have all wroker-processes in this
useless wait-state for zero benefit

if someone is paranoid enough he may setup such nonsense but
do not believe you heal the world this way

View raw message