tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Eggers <its_toas...@yahoo.com.INVALID>
Subject Re: Apache/Tomcat vulnerability
Date Wed, 30 Nov 2016 17:58:21 GMT
Artur,

On 11/30/2016 9:02 AM, Jaaz Portal wrote:
> hi,
> sorry, there was two open connection on port 80
> from 194.135.88.32 that is somwhere on epix.net.pl
> an association of internet traffick exchange (some pirate hub)
> 
> best,
> artur

194.135.88.32 appears to be your web site, no?

. . . just my two cents
/mde/

> 
> 2016-11-30 17:52 GMT+01:00 Jaaz Portal <jaazportal@gmail.com>:
> 
>> hi,
>> i looked at the logs but there are no strange things,
>> traffic as usual, no errors despite this one:
>> [Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid
>> 139906329666752] AH00484: server reached MaxRequestWorkers setting,
>> consider raising the MaxRequestWorkers setting
>>
>> any idea what to do next?
>>
>> best,
>> artur
>>
>> 2016-11-30 17:36 GMT+01:00 Jaaz Portal <jaazportal@gmail.com>:
>>
>>> hi,
>>> they has tried again with success despite setting connection_timeout and
>>> limiting number of clients by mod_bw
>>> the tomcat has frozen again.
>>>
>>> netstat does not showed any connections on port 80 but plenty of
>>> connections from apache to localhost:8009
>>> so it was not an attack that you has described (no slowlaris)
>>>
>>> im looking into debug files of mod_jk and forensic for some hints. If you
>>> want i can share them (they have 4mb compressed)
>>>
>>> best wishes
>>> artur
>>>
>>> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com>:
>>>
>>>> On 28.11.2016 22:04, Jaaz Portal wrote:
>>>>
>>>>> hi Andre,
>>>>> you are wrong. This vulnerability is not only causing memory leaks, it
>>>>> makes also apache workers to hang
>>>>>
>>>>
>>>> Maybe for the last time here :
>>>>
>>>> - what do you call "apache workers" ?
>>>>
>>>> , making it easy to exhaust the pool.
>>>>
>>>> - what do you call "the pool" ?
>>>>
>>>> what i have in my log files. But it is true also that such exhaustion can
>>>>> be made by other forms of dos attacks described in this thread.
>>>>>
>>>>> regarding you suggestion on our application, it does not dos bind server
>>>>> nether does not scan for various vulnerabilities in apache, what i have
>>>>> also in the logs
>>>>>
>>>>
>>>> For your information : I run about 25 Internet-facing Apache webservers
>>>> (some with a back-end tomcat, some not).
>>>> On every single one of those webservers, there are *hundreds* of such
>>>> "scans" every day, as shown by the Apache access logs.  That is just a fact
>>>> of life on the Internet.
>>>> They are annoying, but most of them are harmless (from an "attack" point
>>>> of view), because they are scanning for things that we do not run
>>>> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and
>>>> thus are responded to by Apache as "404 Not found".
>>>> What is annoying with those scans, is
>>>> a) that they fill the logfile, and thus make it more difficult to find
>>>> really significant things
>>>> b) that each of those requires some bandwidth and system resource, if
>>>> only to return a "404 Not found" (or a "401 Unauthorised"), and that we pay
>>>> for that bandwidth and resources.
>>>>
>>>> If I could find a way to charge 0.1 cent per access to my servers, from
>>>> the people who wrote or run the programs who are doing this, I could retire
>>>> in luxury.
>>>>
>>>> But they are not a real problem, because they are caught as "invalid" by
>>>> Apache, and rejected quickly, so they cannot do anything really nasty
>>>> (except if they were sending several thousand such requests per second to
>>>> one of my servers for a long time).
>>>>
>>>> The ones that are worrying, are the ones
>>>> - a) which do /not/ end up as a "404 Not found", because they have found
>>>> an application which responds, and they are not coming from our legitimate
>>>> customers
>>>> - b) /the ones which we do not see/, because they either do not send a
>>>> valid HTTP request, or they have found a way to trigger one of our
>>>> applications, in such a way that the application misbehaves and, perhaps,
>>>> even if they do not crash our servers, they may provide the attacker with
>>>> some entry point to do other things which we do not know and do not control
>>>>
>>>> What I am trying to say here, is /do not jump to premature conclusions/.
>>>> Such "scans" as you mention, happen to everyone, all the time, from
>>>> ever-changing IP addresses located all over the world. Some of those
>>>> "scans" may come from the infected PC of your grandmother, and she does not
>>>> even know about it.
>>>>
>>>> There is no guarantee, and no indication or proof so far, that /these/
>>>> scans are even related to "the other thing" which happens on your
>>>> webserver, which looked much more focused.
>>>>
>>>> So do not just bundle them together as being the same thing, until you
>>>> have some real data that shows for example that these different things all
>>>> come from the same IP addresses.
>>>>
>>>> And one more thing, also finally until you come back with some real data
>>>> : I am not saying that your application "scans your server".  What I am
>>>> saying is that, maybe, by chance or by design, the attackers have found a
>>>> URL which goes to your application, and which causes your application to
>>>> keep tomcat and/or Apache busy for a long time.
>>>> And that maybe /that/ is the problem you should be looking for, and not
>>>> some hypothetical bug in Apache httpd or tomcat.
>>>>
>>>>
>>>>
>>>>> kindly regards,
>>>>> artur
>>>>>
>>>>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com>:
>>>>>
>>>>> On 28.11.2016 20:34, Jaaz Portal wrote:
>>>>>>
>>>>>> hi mark,
>>>>>>> yes, i understand now what slowloris attack is.
>>>>>>> maybe it was this maybe *this one based on * * mod_proxy denial of
>>>>>>> service *
>>>>>>> CVE-2014-0117 <http://cve.mitre.org/cgi-bin/
>>>>>>> cvename.cgi?name=CVE-2014-0117>
>>>>>>>
>>>>>>>
>>>>>> You keep on saying this, but the description of that vulnerability of
>>>>>> *Apache httpd*, and the symptoms which you described, *do not match*.
>>>>>> You described the problem as ocurring in Apache tomcat, which in your
>>>>>> case
>>>>>> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat
>>>>>> cured the problem.
>>>>>>
>>>>>> The CVE above applies to Apache httpd, and describes how an attacker
>>>>>> could
>>>>>> attack Apache httpd and cause *its children* processes to crash (the
>>>>>> children processes of Apache httpd), by leading them to consume a lot
>>>>>> of
>>>>>> memory and crash with an out-of-memory error.
>>>>>> Granted, the problem occurred in the mod_proxy module of Apache httpd;
>>>>>> but
>>>>>> it was httpd which crashed, not tomcat.
>>>>>> And tomcat processes are not "Apache httpd children processes" in any
>>>>>> understanding of the term.
>>>>>>
>>>>>> As far as I remember, you never mentioned Apache httpd crashing. You
>>>>>> mentioned "the pool" getting full or satured or something like that,
>>>>>> without ever describing properly what you meant by "the pool".
>>>>>>
>>>>>> As far as I am concerned, according to all the relatively unspecific
>>>>>> information which you have previously provided :
>>>>>> 1) the attack - if that is what it is/was - is definitely NOT related
>>>>>> to
>>>>>> the CVE which you have repeatedly mentioned
>>>>>> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed
>>>>>> at
>>>>>> your front-end Apache. Instead, it seems that the requests are properly
>>>>>> received by Apache, properly decoded by Apache, and then whatever
>>>>>> Apache
>>>>>> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) is
>>>>>> properly forwarding these requests to a back-end tomcat; and it is at
>>>>>> the
>>>>>> level of that back-end tomcat that the requests never seem to end, and
>>>>>> in
>>>>>> the end paralyse your tomcat server (and later on maybe your Apache
>>>>>> httpd
>>>>>> server too, because it is also waiting for tomcat to respond).
>>>>>>
>>>>>> So your very way of describing the problem, in terms of "first we used
>>>>>> this proxy module, and then they exploited the vulnerability so and so;
>>>>>> then we changed the proxy module, and they exploited that too; etc.."
>>>>>> seems to not have anything to do with the problem per se, and (I
>>>>>> believe)
>>>>>> confuses everyone, including yourself.
>>>>>>
>>>>>> It is not that "they" exploited different vulnerabilities of various
>>>>>> httpd
>>>>>> proxy modules, one after the other. Each of these proxy modules was
>>>>>> doing
>>>>>> its job properly, and forwarding valid HTTP requests properly to
>>>>>> tomcat.
>>>>>> When you changed from one proxy module to another, you did not really
>>>>>> change anything in that respect, because any proxy module would do the
>>>>>> same.
>>>>>>
>>>>>> But in all cases, what did not change, was the tomcat behind the
>>>>>> front-end, and the application running on that tomcat.  So the presumed
>>>>>> attackers did not have to change anything, they just kept on sending
>>>>>> the
>>>>>> same requests, because they were really targeting your back-end tomcat
>>>>>> or
>>>>>> the tomcat application in it, no matter /how/ you were forwarding
>>>>>> requests
>>>>>> from Apache httpd to tomcat.
>>>>>>
>>>>>> So either it is tomcat itself, which has a problem with some request
>>>>>> URLs
>>>>>> which do not bother Apache httpd (possible, but statistically
>>>>>> unlikely), or
>>>>>> it is the application which runs in tomcat that has such a problem
>>>>>> (statistically, much more likely).
>>>>>>
>>>>>> we do not know yet
>>>>>>
>>>>>>>
>>>>>>> we have setup more logging and are waiting for them to attack once
>>>>>>> again
>>>>>>>
>>>>>>>
>>>>>> Yes, that is the right thing to do.  Before deciding what the problem
>>>>>> may
>>>>>> be, and what you can do about it, the first thing you need is *data*.
>>>>>> You
>>>>>> need to know
>>>>>> - which request URL(s?) cause that problem
>>>>>> - which IPs these requests come from (always the same ? multiple IPs
>>>>>> that
>>>>>> change all the time ? how many ? can these IPs be valid/expected
>>>>>> clients or
>>>>>> not ? do these IPs look like some "coordinated group" ?)
>>>>>> - how many such requests there may be during some period of time (10,
>>>>>> 100,
>>>>>> 1000, more ?)
>>>>>> - if these URLs result in passing the request to tomcat
>>>>>> - what tomcat application (if any) they are directed to
>>>>>> - if so, when that application receives such a request, what is it
>>>>>> supposed to do ? does it do it properly ? how long does it need, to
>>>>>> respond
>>>>>> to such a request ?
>>>>>>
>>>>>> You also need to ask yourself a question : is the application which you
>>>>>> run inside tomcat something that you designed yourself (and which
>>>>>> hackers
>>>>>> are unlikely to know well-enough to find such a URL which paralyses
>>>>>> your
>>>>>> server) ? or is it some well-known third-party java application which
>>>>>> you
>>>>>> are running (and for which would-be attackers would be much more
>>>>>> likely to
>>>>>> know exactly such a bug) ?
>>>>>>
>>>>>>
>>>>>> anyway, thank you for all informations, it was very useful and
>>>>>>> educational
>>>>>>> reading for all of us
>>>>>>>
>>>>>>> best wishes,
>>>>>>> artur
>>>>>>>
>>>>>>> 2016-11-28 19:46 GMT+01:00 Mark Eggers <its_toasted@yahoo.com.invalid
>>>>>>>> :
>>>>>>>
>>>>>>> Jaaz,
>>>>>>>
>>>>>>>>
>>>>>>>> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:
>>>>>>>>
>>>>>>>> On 27.11.2016 19:03, Jaaz Portal wrote:
>>>>>>>>>
>>>>>>>>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com>:
>>>>>>>>>>
>>>>>>>>>> On 27.11.2016 14:26, Jaaz Portal wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> hi,
>>>>>>>>>>>
>>>>>>>>>>>> everything i know so far is just this single log line that
>>>>>>>>>>>> appeared
>>>>>>>>>>>> in
>>>>>>>>>>>> apache error.log
>>>>>>>>>>>>
>>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid
>>>>>>>>>>>> 13385:tid
>>>>>>>>>>>> 1397934896385
>>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, consider
>>>>>>>>>>>>
>>>>>>>>>>>> raising
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> MaxR
>>>>>>>>>>>> equestWorkers setting
>>>>>>>>>>>>
>>>>>>>>>>>> there was nothing else, just this strange line
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is not a "strange" line. It is telling you something.
>>>>>>>>>>> One problem is that you seem convinced in advance, without serious
>>>>>>>>>>> proof,
>>>>>>>>>>> that there is a "bug" or a vulnerability in httpd or tomcat.
>>>>>>>>>>> Read the explanation of the httpd parameter, here :
>>>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequ
>>>>>>>>>>> estworkers
>>>>>>>>>>> and try to understand it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I understand it very well. We are serving no more that 500
>>>>>>>>>> clients per
>>>>>>>>>> day
>>>>>>>>>> so there was no other option that some kind of attack.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> About the "bug" or "vulnerability" : a webserver is supposed to
>>>>>>>>>> serve
>>>>>>>>>> HTTP
>>>>>>>>>>
>>>>>>>>>> requests from clients.  That is what you expect of it. It has no
>>>>>>>>>>> choice but
>>>>>>>>>>> to accept any client connection and request, up to the maximum it
>>>>>>>>>>> can
>>>>>>>>>>> handle considering its configuration and the capacity of the
>>>>>>>>>>> system on
>>>>>>>>>>> which it runs. That is not a bug, it is a feature.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We have some weeks ago come under attack from some Polish group.
>>>>>>>>>>> It
>>>>>>>>>>>
>>>>>>>>>> was
>>>>>>>>>> first bind that was DoS'ed, we was running on stable Debian so i
>>>>>>>>>> updated
>>>>>>>>>> bind to latest version. It did not helped. They has dos'ed it so we
>>>>>>>>>> switched to other dns provider. That has helped.
>>>>>>>>>>
>>>>>>>>>> Then they exploited some well know vulnerability in mod_proxy. We
>>>>>>>>>> have
>>>>>>>>>> updated apache to the latest but again they has exploited it, so we
>>>>>>>>>> have
>>>>>>>>>> switched to mod_jk. And then guess what. They exploited it too so i
>>>>>>>>>> decided
>>>>>>>>>> to write to this list looking for help before trying jetty.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The normal Apache httpd access log, will log a request only when
>>>>>>>>>>> it is
>>>>>>>>>>> finished.  If the request never finishes, it will not get logged.
>>>>>>>>>>> That may be why you do not see these requests in the log.
>>>>>>>>>>> But have a look at this Apache httpd module :
>>>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ok, thanks, will take care
>>>>>>>>>>
>>>>>>>>>> Note : that is also why I was telling you to enable the mod_jk log,
>>>>>>>>>> and to
>>>>>>>>>>
>>>>>>>>>> examine it.
>>>>>>>>>>> Because mod_jk will also log information before the request
>>>>>>>>>>> produces a
>>>>>>>>>>> response.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> and server hanged.
>>>>>>>>>>>
>>>>>>>>>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Apache was accepting connection but not processed it. After
>>>>>>>>>> restart of
>>>>>>>>>> tomcat server it worked again.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also in
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> access logs there are no clues that it was under any heavy load.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> after around hour after discovering that our server hanged-out we
>>>>>>>>>>>> have
>>>>>>>>>>>> restarted tomcat server
>>>>>>>>>>>> and it worked again
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, because that will close all connections between Apache
>>>>>>>>>>> httpd and
>>>>>>>>>>> tomcat, and abort all requests that are in the process of being
>>>>>>>>>>> processed
>>>>>>>>>>> by tomcat. So mod_jk will get an error from tomcat, and will
>>>>>>>>>>> report an
>>>>>>>>>>> error to httpd, and httpd will communicate that error to the
>>>>>>>>>>> clients,
>>>>>>>>>>> and
>>>>>>>>>>> close their connection.
>>>>>>>>>>> It still does not tell you what the problem was.
>>>>>>>>>>> The only thing that it suggests, is that the "bad" requests
>>>>>>>>>>> actually
>>>>>>>>>>> make
>>>>>>>>>>> it all the way to tomcat.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> correct
>>>>>>>>>>
>>>>>>>>>> i will enable logs that you has pointed out and we will look what i
>>>>>>>>>> will
>>>>>>>>>> catch
>>>>>>>>>> however i think we have only one chance, as if the solution we has
>>>>>>>>>> found
>>>>>>>>>> out (connection_timeout + mod_bn)
>>>>>>>>>> will work they will stop exploiting it
>>>>>>>>>>
>>>>>>>>>> thank you very much for all the help and explanations
>>>>>>>>>> i will report to the list new facts once they will attack us again
>>>>>>>>>>
>>>>>>>>>> best regards,
>>>>>>>>>> artur
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ok, but also read this e.g. :
>>>>>>>>> https://www.corero.com/blog/695-going-after-the-people-
>>>>>>>>>
>>>>>>>>> behind-ddos-attacks.html
>>>>>>>>
>>>>>>>>
>>>>>>>>> Attempts to bring down a site by DoS attacks is a crime, in most
>>>>>>>>> places..
>>>>>>>>>
>>>>>>>>> You can report it, while at the same time trying to defend yourself
>>>>>>>>> against it.
>>>>>>>>>
>>>>>>>>> It is also relatively easy, and quite inexpensive in terms of system
>>>>>>>>> resources, to run a small shell script which takes a list every few
>>>>>>>>> seconds of the connections to the port of your webserver, and which
>>>>>>>>> IPs
>>>>>>>>> they are coming *from*.
>>>>>>>>> E.g.
>>>>>>>>> First try the netstat command, to see what it lists, like :
>>>>>>>>> # netstat -n --tcp | more
>>>>>>>>>
>>>>>>>>> Then you will want to filter this a bit, to only consider
>>>>>>>>> established
>>>>>>>>> connections to your webserver, for example :
>>>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED"
>>>>>>>>>
>>>>>>>>> Then you will want to send this to a logfile, regularly, like this :
>>>>>>>>>
>>>>>>>>> # date >> some_file.log
>>>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >>
>>>>>>>>> some_file.log
>>>>>>>>> (repeat every 3 seconds)
>>>>>>>>>
>>>>>>>>> This will not generate GB of logfiles, and it will tell you, when
>>>>>>>>> the
>>>>>>>>> problem happens, how many connections there are exactly to your
>>>>>>>>> webserver, and where they are coming from.
>>>>>>>>> Then later you can further analyse this information..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> i think that setting connection-timeout and limiting the number of
>>>>>>>>>>
>>>>>>>>>>> clients
>>>>>>>>>>>> by mod_bd i will
>>>>>>>>>>>> have effect that next time somebody will try this exploit it will
>>>>>>>>>>>>
>>>>>>>>>>>> block
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> his
>>>>>>>>>
>>>>>>>>>> access to the site
>>>>>>>>>>>> for minute or two, pretty good holistic solution i would say
>>>>>>>>>>>>
>>>>>>>>>>>> still, it seems that there is serious vulnerability somewhere in
>>>>>>>>>>>> apache,
>>>>>>>>>>>> mod_jk or tomcat
>>>>>>>>>>>> i would like to help find it out but need some hints which debug
>>>>>>>>>>>> options
>>>>>>>>>>>> enable to catch the bad guys
>>>>>>>>>>>> when they will try next time
>>>>>>>>>>>>
>>>>>>>>>>>> best regards,
>>>>>>>>>>>> artur
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com
>>>>>>>>>>>>> :
>>>>>>>>>>>>
>>>>>>>>>>>> On 27.11.2016 13:23, Jaaz Portal wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> hi Andre,
>>>>>>>>>>>>>
>>>>>>>>>>>>> thank you very much this was very educative but in my case it is
>>>>>>>>>>>>>> little
>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>> different.
>>>>>>>>>>>>>> The server is no flooded, there is maybe dozen of very
>>>>>>>>>>>>>> sophisticated
>>>>>>>>>>>>>> connections that somehow hangs apache workers threads
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you be a bit more specific ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>> When you say "apache workers threads", do you mean threads in
>>>>>>>>>>>>> Apache
>>>>>>>>>>>>> httpd, or threads in Apache Tomcat ? (both are Apache
>>>>>>>>>>>>> webservers,
>>>>>>>>>>>>> so it
>>>>>>>>>>>>> is
>>>>>>>>>>>>> difficult to tell what you are talking about, unless you are
>>>>>>>>>>>>> very
>>>>>>>>>>>>> precise).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me give you some additional explanations, and maybe you can
>>>>>>>>>>>>>
>>>>>>>>>>>>> figure
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> out
>>>>>>>>>
>>>>>>>>>> exactly where it "hangs".
>>>>>>>>>>>>>
>>>>>>>>>>>>>     From the Apache httpd front-end point of view, mod_jk (the
>>>>>>>>>>>>> connector to
>>>>>>>>>>>>> Apache Tomcat) is basically one among other "response
>>>>>>>>>>>>> generators".
>>>>>>>>>>>>> Apache
>>>>>>>>>>>>> httpd does not "know" that behind mod_jk, there is one or more
>>>>>>>>>>>>> Tomcat
>>>>>>>>>>>>> servers.  Apache httpd receives the original client request, and
>>>>>>>>>>>>> depending
>>>>>>>>>>>>> on the URL of the request, it will pass it to mod_jk or to
>>>>>>>>>>>>> another
>>>>>>>>>>>>> response
>>>>>>>>>>>>> generator, to generate the response to the request.
>>>>>>>>>>>>> That mod_jk in the background is using a Tomcat server to
>>>>>>>>>>>>> actually
>>>>>>>>>>>>> generate the response, is none of the business of Apache httpd,
>>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>> it
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> does
>>>>>>>>>
>>>>>>>>>> not care. All it cares about, is to actually receive the response
>>>>>>>>>>>>>
>>>>>>>>>>>>> from
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> mod_jk, and pass it back to the client.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> If httpd passes a request to mod_jk, it is because you have
>>>>>>>>>>>>> specified in
>>>>>>>>>>>>> the Apache configuration, the type of URL that it should pass to
>>>>>>>>>>>>> mod_jk..
>>>>>>>>>>>>> That happens via your "JkMount (URL pattern)" directives in
>>>>>>>>>>>>> Apache
>>>>>>>>>>>>> httpd.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Of course Apache httpd will not pass a request to mod_jk,
>>>>>>>>>>>>> before it
>>>>>>>>>>>>> has
>>>>>>>>>>>>> received at least the URL of the request, and analysed it to
>>>>>>>>>>>>>
>>>>>>>>>>>>> determine
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> *if*
>>>>>>>>>
>>>>>>>>>> it should pass it to mod_jk (*).
>>>>>>>>>>>>>
>>>>>>>>>>>>> If the mod_jk logging is enabled, you can see in it, exactly
>>>>>>>>>>>>> *which*
>>>>>>>>>>>>> requests are passed to mod_jk and to Tomcat.
>>>>>>>>>>>>> Do you know *which* requests, from which clients, cause this
>>>>>>>>>>>>> "thread
>>>>>>>>>>>>> hanging" symptom ?
>>>>>>>>>>>>> Once you would know this, maybe you can design a strategy to
>>>>>>>>>>>>> block
>>>>>>>>>>>>> specifically these requests.
>>>>>>>>>>>>>
>>>>>>>>>>>>> and the effect is permanent. Quickly the pool is exhausted
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> which pool exactly ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> and the only
>>>>>>>>>>>>>
>>>>>>>>>>>>> solution that works is to restart tomcat.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think it is a bug similar to this one from mod_proxy:
>>>>>>>>>>>>>> https://tools.cisco.com/security/center/viewAlert.x?alertId=
>>>>>>>>>>>>>> 34971
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe, maybe not. As long as we do not know what the requests
>>>>>>>>>>>>>> are,
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> block things, we do not know this.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think also that your solution with setting connectionTimeout
>>>>>>>>>>>>> will
>>>>>>>>>>>>> solve
>>>>>>>>>>>>>
>>>>>>>>>>>>> the problem, at least partially. THANK YOU.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Same thing, we do not know this yet.  It was only giving this
>>>>>>>>>>>>>>
>>>>>>>>>>>>> explanation,
>>>>>>>>>>>>> to help you think about where the problem may be.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would like to help you further investigate this issue, as our
>>>>>>>>>>>>>
>>>>>>>>>>>>> server
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> comes under such attack once or twice in a week.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Other than giving you hints, there is not much I or anyone else
>>>>>>>>>>>>>> can do
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>> help. You are the one with control of your servers and
>>>>>>>>>>>>> logfiles, so
>>>>>>>>>>>>> you
>>>>>>>>>>>>> have to investigate and provide more precise information.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> (*) actually, to be precise, Apache httpd passes *all* requests
>>>>>>>>>>>>> to
>>>>>>>>>>>>> mod_jk,
>>>>>>>>>>>>> to ask it "if it wants that request". mod_jk then accepts or
>>>>>>>>>>>>>
>>>>>>>>>>>>> declines,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> depending on the JkMount instructions. If mod_jk declines, then
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> Apache
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> httpd will ask the next response generator if it wants this request,
>>>>>>>>>
>>>>>>>>>> etc...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> best regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> artur
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) <
>>>>>>>>>>>>>> aw@ice-sa.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have a look that the indicated parameters in the two pages
>>>>>>>>>>>>>>> below..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You may be the target of such a variant of DDoS attack : many
>>>>>>>>>>>>>>> clients
>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>> a TCP connection to your server (front-end), but then never
>>>>>>>>>>>>>>> sends
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> HTTP
>>>>>>>>>>>>>>> request on that connection.  In the meantime, the server
>>>>>>>>>>>>>>> accepts
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>> TCP
>>>>>>>>>
>>>>>>>>>> connection, and passes it on to a "child" process or thread for
>>>>>>>>>>>>>>> processing.  The child then waits for the HTTP request line to
>>>>>>>>>>>>>>> arrive
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> the connection (during a certain time), but it never arrives.
>>>>>>>>>>>>>>> After a
>>>>>>>>>>>>>>> while, this triggers a timeout (see below), but the standard
>>>>>>>>>>>>>>> value of
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> timeout may be such that in the meantime, a lot of other
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> connections
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>> have
>>>>>>>>>
>>>>>>>>>> been established by other such nefarious clients, so a lot of
>>>>>>>>>>>>>>> resources
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the webserver are tied up, waiting for something that will
>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>> come..
>>>>>>>>>>>>>>> Since there is never any real request sent on the connection,
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>> (probably) not see this in the logs either.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The above is the basic mechanism of such an attack.  There
>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>> variations, such as the client not "not sending" a request
>>>>>>>>>>>>>>> line,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>> sending it extremely slowly, thus achieving perhaps similar kinds
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>> effects.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>>> As someone pointed out, it is quite difficult to do something
>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> at the level of the webserver itself, because by the time you
>>>>>>>>>>>>>>> would do
>>>>>>>>>>>>>>> something about it, the resources have already been consumed
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> server is probably already overloaded.
>>>>>>>>>>>>>>> There are specialised front-end devices and software
>>>>>>>>>>>>>>> available, to
>>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>>> and protect against this kind of attack.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You may want to have a look at the following parameters, but
>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> read the caveats (side-effects, interlocking timeouts etc.),
>>>>>>>>>>>>>>> otherwise
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> may do more harm than good.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Another thing : the settings below are for Apache Tomcat,
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>> in your
>>>>>>>>>>>>>>> case is the back-end. It would of course be much better to
>>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> eliminate this at the front-end, or even before.  I had a
>>>>>>>>>>>>>>> look at
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> Apache httpd documentation, and could not find a corresponding
>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>> But I am sure that it must exist. You may want to post this
>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>> question
>>>>>>>>>>>>>>> on the Apache httpd user's list for a better response.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Tomcat configuration settings :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AJP Connector : (http://tomcat.apache.org/tomc
>>>>>>>>>>>>>>> at-8.5-doc/config/ajp.html#
>>>>>>>>>>>>>>> Standard_Implementations)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> connectionTimeout
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after
>>>>>>>>>>>>>>> accepting a
>>>>>>>>>>>>>>> connection, for the request URI line to be presented. The
>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> AJP protocol connectors is -1 (i.e. infinite).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (You could for example try to set this to 3000 (milliseconds)
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>> lower. That should be more than enough for any legitimate
>>>>>>>>>>>>>>> client
>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>> the HTTP request line.  Note however that by doing this at the
>>>>>>>>>>>>>>> Tomcat
>>>>>>>>>>>>>>> level, you will probably move the problem to the Apache
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> httpd/mod_jk
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>> level.  But at least it might confirm that this is the problem
>>>>>>>>>
>>>>>>>>>> that you
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> seeing.  The mod_jk logfile at the httpd level may give you
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> hints
>>>>>>>>>>>>>>> there.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HTTP Connector : (http://tomcat.apache.org/tomc
>>>>>>>>>>>>>>> at-8.5-doc/config/http.html#Standard_Implementation)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> connectionTimeout
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after
>>>>>>>>>>>>>>> accepting a
>>>>>>>>>>>>>>> connection, for the request URI line to be presented. Use a
>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>> of -1
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> indicate no (i.e. infinite) timeout. The default value is
>>>>>>>>>>>>>>> 60000
>>>>>>>>>>>>>>> (i.e.
>>>>>>>>>>>>>>> 60
>>>>>>>>>>>>>>> seconds) but note that the standard server.xml that ships with
>>>>>>>>>>>>>>> Tomcat
>>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>>> this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> set to
>>>>>>>>>>>>>>> false, this timeout will also be used when reading the request
>>>>>>>>>>>>>>> body (if
>>>>>>>>>>>>>>> any).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 26.11.2016 09:57, Jaaz Portal wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> sorry, its mod_jk no jk2, my typo. All at latest versions. We
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>> with
>>>>>>>>>
>>>>>>>>>> mod proxy too.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is no flood of the server. Nobody is flooding us, they
>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>> specific connections after which pool of apache workers is
>>>>>>>>>>>>>>>> exhausted
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> blocked
>>>>>>>>>>>>>>>> and we need to restart tomcat server.
>>>>>>>>>>>>>>>> It is some kind of exploit but do not know how to log it to
>>>>>>>>>>>>>>>> obtain
>>>>>>>>>>>>>>>> details.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> i had put a limit on connections per client with hope that
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>> but once again, it is not a flood.
>>>>>>>>>>>>>>>> They open several connections that are not dropped by apache
>>>>>>>>>>>>>>>> when they
>>>>>>>>>>>>>>>> disconnect. This way whole pool is quickly exhausted and the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>> broken.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>>>> i would like to help you to figure details of this attack but
>>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>>> production server so it is impossible to much debugging
>>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>>>> artur
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu <
>>>>>>>>>>>>>>>> niranjan.bommu@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> you can find who is flooding site in apache access.log and
>>>>>>>>>>>>>>>>> block
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> firewall.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ex to find the IP:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -c|sort
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>> -gr
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal
>>>>>>>>>>>>>>>>> <jaazportal@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> we are from some weeks struggling with some Polish hackers
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> bringing our server down. After updating apache to latest
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>>> (2.4.23)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> and tomcat (8.0.38) available for debian systems we still
>>>>>>>>>>>>>>>>> cannot
>>>>>>>>>>>>>>>>> secure
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> server.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Today it has stopped to respond again and we needed to
>>>>>>>>>>>>>>>>>> restart
>>>>>>>>>>>>>>>>>> tomcat
>>>>>>>>>>>>>>>>>> process to get it back alive.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> There is no too much clues in the logs. The apache
>>>>>>>>>>>>>>>>>> error.log
>>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>> this line:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid
>>>>>>>>>>>>>>>>>> 13385:tid
>>>>>>>>>>>>>>>>>> 1397934896385
>>>>>>>>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting,
>>>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>> raising
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> MaxR
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> equestWorkers setting
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> seems that somehow tomcat, mod-jk2 or even apache is
>>>>>>>>>>>>>>>>>> vulnerable to
>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> exploit, as we certainly does not have such traffic that
>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> block
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> server otherwise
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for now we have increased MaxRequestWorkers and we have
>>>>>>>>>>>>>>>>>> limited
>>>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> connections from one client to 5 by mod_bw and limited
>>>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> simultaneous connections from one ip by iptables but does
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> will help
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> best regards,
>>>>>>>>>>>>>>>>>> artur
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Thanks*
>>>>>>>>>>>>>>>>> *Niranjan*
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This sounds like a variant of the slowloris attack.
>>>>>>>>
>>>>>>>> This type of attack doesn't take a large number of clients or
>>>>>>>> consume a
>>>>>>>> large amount of bandwidth.
>>>>>>>>
>>>>>>>> Basically, the maximum number of connections are made to the server,
>>>>>>>> and
>>>>>>>> just enough data is sent to each connection in order to not trigger
>>>>>>>> the
>>>>>>>> timeout. André has explained this in more detail earlier in the
>>>>>>>> thread.
>>>>>>>> Search for "slowloris attack" for more information.
>>>>>>>>
>>>>>>>> There are several ways of mitigating this type of attack.
>>>>>>>>
>>>>>>>> As André has mentioned, placing a dedicated device in front of your
>>>>>>>> systems is often the best way. Lots of benefits (platform neutral, no
>>>>>>>> stress on your current servers), and some issues (cost, placement /
>>>>>>>> access may be an issue with hosted solutions).
>>>>>>>>
>>>>>>>> However, there are Apache HTTPD modules that can help mitigate these
>>>>>>>> types of attacks. Some of them are:
>>>>>>>>
>>>>>>>> mod_reqtimeout (should be included by default in your Apache HTRPD
>>>>>>>> 2.4)
>>>>>>>> mod_qos (quality of service module)
>>>>>>>> mod_security (application firewall with lots of security rules)
>>>>>>>>
>>>>>>>> Do a quick search on "slowloris attack apache httpd 2.4" to get some
>>>>>>>> ideas.
>>>>>>>>
>>>>>>>> All of them will probably place additional load on your Apache HTTPD
>>>>>>>> server, so make sure that the platform is robust enough to manage the
>>>>>>>> additional load.
>>>>>>>>
>>>>>>>> There is also a beta version of the mod_security module written as a
>>>>>>>> servlet filter. It should be possible to build this and configure the
>>>>>>>> filter in Tomcat's default web.xml ($CATALINA_BASE/conf/web.xml).
>>>>>>>> I've
>>>>>>>> not tried this. Also, the code base hasn't seen any activity for 3
>>>>>>>> years.
>>>>>>>>
>>>>>>>> Do a quick search on "modsecurity servlet filter" to find out more
>>>>>>>> about
>>>>>>>> the servlet filter version of mod_security.
>>>>>>>>
>>>>>>>> In short, there appear to be some ways to mitigate the attack.
>>>>>>>>
>>>>>>>> . . . just my two cents
>>>>>>>> /mde/




Mime
View raw message