tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaaz Portal <jaazpor...@gmail.com>
Subject Re: Apache/Tomcat vulnerability
Date Wed, 30 Nov 2016 17:02:37 GMT
hi,
sorry, there was two open connection on port 80
from 194.135.88.32 that is somwhere on epix.net.pl
an association of internet traffick exchange (some pirate hub)

best,
artur

2016-11-30 17:52 GMT+01:00 Jaaz Portal <jaazportal@gmail.com>:

> hi,
> i looked at the logs but there are no strange things,
> traffic as usual, no errors despite this one:
> [Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid
> 139906329666752] AH00484: server reached MaxRequestWorkers setting,
> consider raising the MaxRequestWorkers setting
>
> any idea what to do next?
>
> best,
> artur
>
> 2016-11-30 17:36 GMT+01:00 Jaaz Portal <jaazportal@gmail.com>:
>
>> hi,
>> they has tried again with success despite setting connection_timeout and
>> limiting number of clients by mod_bw
>> the tomcat has frozen again.
>>
>> netstat does not showed any connections on port 80 but plenty of
>> connections from apache to localhost:8009
>> so it was not an attack that you has described (no slowlaris)
>>
>> im looking into debug files of mod_jk and forensic for some hints. If you
>> want i can share them (they have 4mb compressed)
>>
>> best wishes
>> artur
>>
>> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com>:
>>
>>> On 28.11.2016 22:04, Jaaz Portal wrote:
>>>
>>>> hi Andre,
>>>> you are wrong. This vulnerability is not only causing memory leaks, it
>>>> makes also apache workers to hang
>>>>
>>>
>>> Maybe for the last time here :
>>>
>>> - what do you call "apache workers" ?
>>>
>>> , making it easy to exhaust the pool.
>>>
>>> - what do you call "the pool" ?
>>>
>>> what i have in my log files. But it is true also that such exhaustion can
>>>> be made by other forms of dos attacks described in this thread.
>>>>
>>>> regarding you suggestion on our application, it does not dos bind server
>>>> nether does not scan for various vulnerabilities in apache, what i have
>>>> also in the logs
>>>>
>>>
>>> For your information : I run about 25 Internet-facing Apache webservers
>>> (some with a back-end tomcat, some not).
>>> On every single one of those webservers, there are *hundreds* of such
>>> "scans" every day, as shown by the Apache access logs.  That is just a fact
>>> of life on the Internet.
>>> They are annoying, but most of them are harmless (from an "attack" point
>>> of view), because they are scanning for things that we do not run
>>> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and
>>> thus are responded to by Apache as "404 Not found".
>>> What is annoying with those scans, is
>>> a) that they fill the logfile, and thus make it more difficult to find
>>> really significant things
>>> b) that each of those requires some bandwidth and system resource, if
>>> only to return a "404 Not found" (or a "401 Unauthorised"), and that we pay
>>> for that bandwidth and resources.
>>>
>>> If I could find a way to charge 0.1 cent per access to my servers, from
>>> the people who wrote or run the programs who are doing this, I could retire
>>> in luxury.
>>>
>>> But they are not a real problem, because they are caught as "invalid" by
>>> Apache, and rejected quickly, so they cannot do anything really nasty
>>> (except if they were sending several thousand such requests per second to
>>> one of my servers for a long time).
>>>
>>> The ones that are worrying, are the ones
>>> - a) which do /not/ end up as a "404 Not found", because they have found
>>> an application which responds, and they are not coming from our legitimate
>>> customers
>>> - b) /the ones which we do not see/, because they either do not send a
>>> valid HTTP request, or they have found a way to trigger one of our
>>> applications, in such a way that the application misbehaves and, perhaps,
>>> even if they do not crash our servers, they may provide the attacker with
>>> some entry point to do other things which we do not know and do not control
>>>
>>> What I am trying to say here, is /do not jump to premature conclusions/.
>>> Such "scans" as you mention, happen to everyone, all the time, from
>>> ever-changing IP addresses located all over the world. Some of those
>>> "scans" may come from the infected PC of your grandmother, and she does not
>>> even know about it.
>>>
>>> There is no guarantee, and no indication or proof so far, that /these/
>>> scans are even related to "the other thing" which happens on your
>>> webserver, which looked much more focused.
>>>
>>> So do not just bundle them together as being the same thing, until you
>>> have some real data that shows for example that these different things all
>>> come from the same IP addresses.
>>>
>>> And one more thing, also finally until you come back with some real data
>>> : I am not saying that your application "scans your server".  What I am
>>> saying is that, maybe, by chance or by design, the attackers have found a
>>> URL which goes to your application, and which causes your application to
>>> keep tomcat and/or Apache busy for a long time.
>>> And that maybe /that/ is the problem you should be looking for, and not
>>> some hypothetical bug in Apache httpd or tomcat.
>>>
>>>
>>>
>>>> kindly regards,
>>>> artur
>>>>
>>>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com>:
>>>>
>>>> On 28.11.2016 20:34, Jaaz Portal wrote:
>>>>>
>>>>> hi mark,
>>>>>> yes, i understand now what slowloris attack is.
>>>>>> maybe it was this maybe *this one based on * * mod_proxy denial of
>>>>>> service *
>>>>>> CVE-2014-0117 <http://cve.mitre.org/cgi-bin/
>>>>>> cvename.cgi?name=CVE-2014-0117>
>>>>>>
>>>>>>
>>>>> You keep on saying this, but the description of that vulnerability of
>>>>> *Apache httpd*, and the symptoms which you described, *do not match*.
>>>>> You described the problem as ocurring in Apache tomcat, which in your
>>>>> case
>>>>> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat
>>>>> cured the problem.
>>>>>
>>>>> The CVE above applies to Apache httpd, and describes how an attacker
>>>>> could
>>>>> attack Apache httpd and cause *its children* processes to crash (the
>>>>> children processes of Apache httpd), by leading them to consume a lot
>>>>> of
>>>>> memory and crash with an out-of-memory error.
>>>>> Granted, the problem occurred in the mod_proxy module of Apache httpd;
>>>>> but
>>>>> it was httpd which crashed, not tomcat.
>>>>> And tomcat processes are not "Apache httpd children processes" in any
>>>>> understanding of the term.
>>>>>
>>>>> As far as I remember, you never mentioned Apache httpd crashing. You
>>>>> mentioned "the pool" getting full or satured or something like that,
>>>>> without ever describing properly what you meant by "the pool".
>>>>>
>>>>> As far as I am concerned, according to all the relatively unspecific
>>>>> information which you have previously provided :
>>>>> 1) the attack - if that is what it is/was - is definitely NOT related
>>>>> to
>>>>> the CVE which you have repeatedly mentioned
>>>>> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed
>>>>> at
>>>>> your front-end Apache. Instead, it seems that the requests are properly
>>>>> received by Apache, properly decoded by Apache, and then whatever
>>>>> Apache
>>>>> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) is
>>>>> properly forwarding these requests to a back-end tomcat; and it is at
>>>>> the
>>>>> level of that back-end tomcat that the requests never seem to end, and
>>>>> in
>>>>> the end paralyse your tomcat server (and later on maybe your Apache
>>>>> httpd
>>>>> server too, because it is also waiting for tomcat to respond).
>>>>>
>>>>> So your very way of describing the problem, in terms of "first we used
>>>>> this proxy module, and then they exploited the vulnerability so and so;
>>>>> then we changed the proxy module, and they exploited that too; etc.."
>>>>> seems to not have anything to do with the problem per se, and (I
>>>>> believe)
>>>>> confuses everyone, including yourself.
>>>>>
>>>>> It is not that "they" exploited different vulnerabilities of various
>>>>> httpd
>>>>> proxy modules, one after the other. Each of these proxy modules was
>>>>> doing
>>>>> its job properly, and forwarding valid HTTP requests properly to
>>>>> tomcat.
>>>>> When you changed from one proxy module to another, you did not really
>>>>> change anything in that respect, because any proxy module would do the
>>>>> same.
>>>>>
>>>>> But in all cases, what did not change, was the tomcat behind the
>>>>> front-end, and the application running on that tomcat.  So the presumed
>>>>> attackers did not have to change anything, they just kept on sending
>>>>> the
>>>>> same requests, because they were really targeting your back-end tomcat
>>>>> or
>>>>> the tomcat application in it, no matter /how/ you were forwarding
>>>>> requests
>>>>> from Apache httpd to tomcat.
>>>>>
>>>>> So either it is tomcat itself, which has a problem with some request
>>>>> URLs
>>>>> which do not bother Apache httpd (possible, but statistically
>>>>> unlikely), or
>>>>> it is the application which runs in tomcat that has such a problem
>>>>> (statistically, much more likely).
>>>>>
>>>>> we do not know yet
>>>>>
>>>>>>
>>>>>> we have setup more logging and are waiting for them to attack once
>>>>>> again
>>>>>>
>>>>>>
>>>>> Yes, that is the right thing to do.  Before deciding what the problem
>>>>> may
>>>>> be, and what you can do about it, the first thing you need is *data*.
>>>>> You
>>>>> need to know
>>>>> - which request URL(s?) cause that problem
>>>>> - which IPs these requests come from (always the same ? multiple IPs
>>>>> that
>>>>> change all the time ? how many ? can these IPs be valid/expected
>>>>> clients or
>>>>> not ? do these IPs look like some "coordinated group" ?)
>>>>> - how many such requests there may be during some period of time (10,
>>>>> 100,
>>>>> 1000, more ?)
>>>>> - if these URLs result in passing the request to tomcat
>>>>> - what tomcat application (if any) they are directed to
>>>>> - if so, when that application receives such a request, what is it
>>>>> supposed to do ? does it do it properly ? how long does it need, to
>>>>> respond
>>>>> to such a request ?
>>>>>
>>>>> You also need to ask yourself a question : is the application which you
>>>>> run inside tomcat something that you designed yourself (and which
>>>>> hackers
>>>>> are unlikely to know well-enough to find such a URL which paralyses
>>>>> your
>>>>> server) ? or is it some well-known third-party java application which
>>>>> you
>>>>> are running (and for which would-be attackers would be much more
>>>>> likely to
>>>>> know exactly such a bug) ?
>>>>>
>>>>>
>>>>> anyway, thank you for all informations, it was very useful and
>>>>>> educational
>>>>>> reading for all of us
>>>>>>
>>>>>> best wishes,
>>>>>> artur
>>>>>>
>>>>>> 2016-11-28 19:46 GMT+01:00 Mark Eggers <its_toasted@yahoo.com.invalid
>>>>>> >:
>>>>>>
>>>>>> Jaaz,
>>>>>>
>>>>>>>
>>>>>>> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote:
>>>>>>>
>>>>>>> On 27.11.2016 19:03, Jaaz Portal wrote:
>>>>>>>>
>>>>>>>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com>:
>>>>>>>>>
>>>>>>>>> On 27.11.2016 14:26, Jaaz Portal wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> hi,
>>>>>>>>>>
>>>>>>>>>>> everything i know so far is just this single log line that
>>>>>>>>>>> appeared
>>>>>>>>>>> in
>>>>>>>>>>> apache error.log
>>>>>>>>>>>
>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid
>>>>>>>>>>> 13385:tid
>>>>>>>>>>> 1397934896385
>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, consider
>>>>>>>>>>>
>>>>>>>>>>> raising
>>>>>>>>>>
>>>>>>>>>
>>>>>>> the
>>>>>>>>
>>>>>>>>> MaxR
>>>>>>>>>>> equestWorkers setting
>>>>>>>>>>>
>>>>>>>>>>> there was nothing else, just this strange line
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is not a "strange" line. It is telling you something.
>>>>>>>>>> One problem is that you seem convinced in advance, without serious
>>>>>>>>>> proof,
>>>>>>>>>> that there is a "bug" or a vulnerability in httpd or tomcat.
>>>>>>>>>> Read the explanation of the httpd parameter, here :
>>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequ
>>>>>>>>>> estworkers
>>>>>>>>>> and try to understand it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I understand it very well. We are serving no more that 500
>>>>>>>>> clients per
>>>>>>>>> day
>>>>>>>>> so there was no other option that some kind of attack.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> About the "bug" or "vulnerability" : a webserver is supposed to
>>>>>>>>> serve
>>>>>>>>> HTTP
>>>>>>>>>
>>>>>>>>> requests from clients.  That is what you expect of it. It has no
>>>>>>>>>> choice but
>>>>>>>>>> to accept any client connection and request, up to the maximum it
>>>>>>>>>> can
>>>>>>>>>> handle considering its configuration and the capacity of the
>>>>>>>>>> system on
>>>>>>>>>> which it runs. That is not a bug, it is a feature.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We have some weeks ago come under attack from some Polish group.
>>>>>>>>>> It
>>>>>>>>>>
>>>>>>>>> was
>>>>>>>>> first bind that was DoS'ed, we was running on stable Debian so i
>>>>>>>>> updated
>>>>>>>>> bind to latest version. It did not helped. They has dos'ed it so we
>>>>>>>>> switched to other dns provider. That has helped.
>>>>>>>>>
>>>>>>>>> Then they exploited some well know vulnerability in mod_proxy. We
>>>>>>>>> have
>>>>>>>>> updated apache to the latest but again they has exploited it, so we
>>>>>>>>> have
>>>>>>>>> switched to mod_jk. And then guess what. They exploited it too so i
>>>>>>>>> decided
>>>>>>>>> to write to this list looking for help before trying jetty.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The normal Apache httpd access log, will log a request only when
>>>>>>>>>> it is
>>>>>>>>>> finished.  If the request never finishes, it will not get logged.
>>>>>>>>>> That may be why you do not see these requests in the log.
>>>>>>>>>> But have a look at this Apache httpd module :
>>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ok, thanks, will take care
>>>>>>>>>
>>>>>>>>> Note : that is also why I was telling you to enable the mod_jk log,
>>>>>>>>> and to
>>>>>>>>>
>>>>>>>>> examine it.
>>>>>>>>>> Because mod_jk will also log information before the request
>>>>>>>>>> produces a
>>>>>>>>>> response.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and server hanged.
>>>>>>>>>>
>>>>>>>>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apache was accepting connection but not processed it. After
>>>>>>>>> restart of
>>>>>>>>> tomcat server it worked again.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also in
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> access logs there are no clues that it was under any heavy load.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> after around hour after discovering that our server hanged-out we
>>>>>>>>>>> have
>>>>>>>>>>> restarted tomcat server
>>>>>>>>>>> and it worked again
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes, because that will close all connections between Apache
>>>>>>>>>> httpd and
>>>>>>>>>> tomcat, and abort all requests that are in the process of being
>>>>>>>>>> processed
>>>>>>>>>> by tomcat. So mod_jk will get an error from tomcat, and will
>>>>>>>>>> report an
>>>>>>>>>> error to httpd, and httpd will communicate that error to the
>>>>>>>>>> clients,
>>>>>>>>>> and
>>>>>>>>>> close their connection.
>>>>>>>>>> It still does not tell you what the problem was.
>>>>>>>>>> The only thing that it suggests, is that the "bad" requests
>>>>>>>>>> actually
>>>>>>>>>> make
>>>>>>>>>> it all the way to tomcat.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> correct
>>>>>>>>>
>>>>>>>>> i will enable logs that you has pointed out and we will look what i
>>>>>>>>> will
>>>>>>>>> catch
>>>>>>>>> however i think we have only one chance, as if the solution we has
>>>>>>>>> found
>>>>>>>>> out (connection_timeout + mod_bn)
>>>>>>>>> will work they will stop exploiting it
>>>>>>>>>
>>>>>>>>> thank you very much for all the help and explanations
>>>>>>>>> i will report to the list new facts once they will attack us again
>>>>>>>>>
>>>>>>>>> best regards,
>>>>>>>>> artur
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok, but also read this e.g. :
>>>>>>>> https://www.corero.com/blog/695-going-after-the-people-
>>>>>>>>
>>>>>>>> behind-ddos-attacks.html
>>>>>>>
>>>>>>>
>>>>>>>> Attempts to bring down a site by DoS attacks is a crime, in most
>>>>>>>> places..
>>>>>>>>
>>>>>>>> You can report it, while at the same time trying to defend yourself
>>>>>>>> against it.
>>>>>>>>
>>>>>>>> It is also relatively easy, and quite inexpensive in terms of system
>>>>>>>> resources, to run a small shell script which takes a list every few
>>>>>>>> seconds of the connections to the port of your webserver, and which
>>>>>>>> IPs
>>>>>>>> they are coming *from*.
>>>>>>>> E.g.
>>>>>>>> First try the netstat command, to see what it lists, like :
>>>>>>>> # netstat -n --tcp | more
>>>>>>>>
>>>>>>>> Then you will want to filter this a bit, to only consider
>>>>>>>> established
>>>>>>>> connections to your webserver, for example :
>>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED"
>>>>>>>>
>>>>>>>> Then you will want to send this to a logfile, regularly, like this :
>>>>>>>>
>>>>>>>> # date >> some_file.log
>>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >>
>>>>>>>> some_file.log
>>>>>>>> (repeat every 3 seconds)
>>>>>>>>
>>>>>>>> This will not generate GB of logfiles, and it will tell you, when
>>>>>>>> the
>>>>>>>> problem happens, how many connections there are exactly to your
>>>>>>>> webserver, and where they are coming from.
>>>>>>>> Then later you can further analyse this information..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> i think that setting connection-timeout and limiting the number of
>>>>>>>>>
>>>>>>>>>> clients
>>>>>>>>>>> by mod_bd i will
>>>>>>>>>>> have effect that next time somebody will try this exploit it will
>>>>>>>>>>>
>>>>>>>>>>> block
>>>>>>>>>>
>>>>>>>>>
>>>>>>> his
>>>>>>>>
>>>>>>>>> access to the site
>>>>>>>>>>> for minute or two, pretty good holistic solution i would say
>>>>>>>>>>>
>>>>>>>>>>> still, it seems that there is serious vulnerability somewhere in
>>>>>>>>>>> apache,
>>>>>>>>>>> mod_jk or tomcat
>>>>>>>>>>> i would like to help find it out but need some hints which debug
>>>>>>>>>>> options
>>>>>>>>>>> enable to catch the bad guys
>>>>>>>>>>> when they will try next time
>>>>>>>>>>>
>>>>>>>>>>> best regards,
>>>>>>>>>>> artur
>>>>>>>>>>>
>>>>>>>>>>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) <aw@ice-sa.com
>>>>>>>>>>> >:
>>>>>>>>>>>
>>>>>>>>>>> On 27.11.2016 13:23, Jaaz Portal wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> hi Andre,
>>>>>>>>>>>>
>>>>>>>>>>>> thank you very much this was very educative but in my case it is
>>>>>>>>>>>>> little
>>>>>>>>>>>>> bit
>>>>>>>>>>>>> different.
>>>>>>>>>>>>> The server is no flooded, there is maybe dozen of very
>>>>>>>>>>>>> sophisticated
>>>>>>>>>>>>> connections that somehow hangs apache workers threads
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you be a bit more specific ?
>>>>>>>>>>>>>
>>>>>>>>>>>> When you say "apache workers threads", do you mean threads in
>>>>>>>>>>>> Apache
>>>>>>>>>>>> httpd, or threads in Apache Tomcat ? (both are Apache
>>>>>>>>>>>> webservers,
>>>>>>>>>>>> so it
>>>>>>>>>>>> is
>>>>>>>>>>>> difficult to tell what you are talking about, unless you are
>>>>>>>>>>>> very
>>>>>>>>>>>> precise).
>>>>>>>>>>>>
>>>>>>>>>>>> Let me give you some additional explanations, and maybe you can
>>>>>>>>>>>>
>>>>>>>>>>>> figure
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> out
>>>>>>>>
>>>>>>>>> exactly where it "hangs".
>>>>>>>>>>>>
>>>>>>>>>>>>     From the Apache httpd front-end point of view, mod_jk (the
>>>>>>>>>>>> connector to
>>>>>>>>>>>> Apache Tomcat) is basically one among other "response
>>>>>>>>>>>> generators".
>>>>>>>>>>>> Apache
>>>>>>>>>>>> httpd does not "know" that behind mod_jk, there is one or more
>>>>>>>>>>>> Tomcat
>>>>>>>>>>>> servers.  Apache httpd receives the original client request, and
>>>>>>>>>>>> depending
>>>>>>>>>>>> on the URL of the request, it will pass it to mod_jk or to
>>>>>>>>>>>> another
>>>>>>>>>>>> response
>>>>>>>>>>>> generator, to generate the response to the request.
>>>>>>>>>>>> That mod_jk in the background is using a Tomcat server to
>>>>>>>>>>>> actually
>>>>>>>>>>>> generate the response, is none of the business of Apache httpd,
>>>>>>>>>>>> and
>>>>>>>>>>>>
>>>>>>>>>>>> it
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> does
>>>>>>>>
>>>>>>>>> not care. All it cares about, is to actually receive the response
>>>>>>>>>>>>
>>>>>>>>>>>> from
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> mod_jk, and pass it back to the client.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> If httpd passes a request to mod_jk, it is because you have
>>>>>>>>>>>> specified in
>>>>>>>>>>>> the Apache configuration, the type of URL that it should pass to
>>>>>>>>>>>> mod_jk..
>>>>>>>>>>>> That happens via your "JkMount (URL pattern)" directives in
>>>>>>>>>>>> Apache
>>>>>>>>>>>> httpd.
>>>>>>>>>>>>
>>>>>>>>>>>> Of course Apache httpd will not pass a request to mod_jk,
>>>>>>>>>>>> before it
>>>>>>>>>>>> has
>>>>>>>>>>>> received at least the URL of the request, and analysed it to
>>>>>>>>>>>>
>>>>>>>>>>>> determine
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> *if*
>>>>>>>>
>>>>>>>>> it should pass it to mod_jk (*).
>>>>>>>>>>>>
>>>>>>>>>>>> If the mod_jk logging is enabled, you can see in it, exactly
>>>>>>>>>>>> *which*
>>>>>>>>>>>> requests are passed to mod_jk and to Tomcat.
>>>>>>>>>>>> Do you know *which* requests, from which clients, cause this
>>>>>>>>>>>> "thread
>>>>>>>>>>>> hanging" symptom ?
>>>>>>>>>>>> Once you would know this, maybe you can design a strategy to
>>>>>>>>>>>> block
>>>>>>>>>>>> specifically these requests.
>>>>>>>>>>>>
>>>>>>>>>>>> and the effect is permanent. Quickly the pool is exhausted
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> which pool exactly ?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and the only
>>>>>>>>>>>>
>>>>>>>>>>>> solution that works is to restart tomcat.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I think it is a bug similar to this one from mod_proxy:
>>>>>>>>>>>>> https://tools.cisco.com/security/center/viewAlert.x?alertId=
>>>>>>>>>>>>> 34971
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe, maybe not. As long as we do not know what the requests
>>>>>>>>>>>>> are,
>>>>>>>>>>>>> that
>>>>>>>>>>>>>
>>>>>>>>>>>>> block things, we do not know this.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I think also that your solution with setting connectionTimeout
>>>>>>>>>>>> will
>>>>>>>>>>>> solve
>>>>>>>>>>>>
>>>>>>>>>>>> the problem, at least partially. THANK YOU.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Same thing, we do not know this yet.  It was only giving this
>>>>>>>>>>>>>
>>>>>>>>>>>> explanation,
>>>>>>>>>>>> to help you think about where the problem may be.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to help you further investigate this issue, as our
>>>>>>>>>>>>
>>>>>>>>>>>> server
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> comes under such attack once or twice in a week.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other than giving you hints, there is not much I or anyone else
>>>>>>>>>>>>> can do
>>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>> help. You are the one with control of your servers and
>>>>>>>>>>>> logfiles, so
>>>>>>>>>>>> you
>>>>>>>>>>>> have to investigate and provide more precise information.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> (*) actually, to be precise, Apache httpd passes *all* requests
>>>>>>>>>>>> to
>>>>>>>>>>>> mod_jk,
>>>>>>>>>>>> to ask it "if it wants that request". mod_jk then accepts or
>>>>>>>>>>>>
>>>>>>>>>>>> declines,
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> depending on the JkMount instructions. If mod_jk declines, then
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> Apache
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> httpd will ask the next response generator if it wants this request,
>>>>>>>>
>>>>>>>>> etc...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> best regards,
>>>>>>>>>>>>
>>>>>>>>>>>> artur
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) <
>>>>>>>>>>>>> aw@ice-sa.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Have a look that the indicated parameters in the two pages
>>>>>>>>>>>>>> below..
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You may be the target of such a variant of DDoS attack : many
>>>>>>>>>>>>>> clients
>>>>>>>>>>>>>> open
>>>>>>>>>>>>>> a TCP connection to your server (front-end), but then never
>>>>>>>>>>>>>> sends
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> HTTP
>>>>>>>>>>>>>> request on that connection.  In the meantime, the server
>>>>>>>>>>>>>> accepts
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>> TCP
>>>>>>>>
>>>>>>>>> connection, and passes it on to a "child" process or thread for
>>>>>>>>>>>>>> processing.  The child then waits for the HTTP request line to
>>>>>>>>>>>>>> arrive
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> the connection (during a certain time), but it never arrives.
>>>>>>>>>>>>>> After a
>>>>>>>>>>>>>> while, this triggers a timeout (see below), but the standard
>>>>>>>>>>>>>> value of
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> timeout may be such that in the meantime, a lot of other
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> connections
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>> have
>>>>>>>>
>>>>>>>>> been established by other such nefarious clients, so a lot of
>>>>>>>>>>>>>> resources
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> the webserver are tied up, waiting for something that will
>>>>>>>>>>>>>> never
>>>>>>>>>>>>>> come..
>>>>>>>>>>>>>> Since there is never any real request sent on the connection,
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> (probably) not see this in the logs either.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The above is the basic mechanism of such an attack.  There
>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>> variations, such as the client not "not sending" a request
>>>>>>>>>>>>>> line,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>> sending it extremely slowly, thus achieving perhaps similar kinds
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>> effects.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> As someone pointed out, it is quite difficult to do something
>>>>>>>>>>>>>> about
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> at the level of the webserver itself, because by the time you
>>>>>>>>>>>>>> would do
>>>>>>>>>>>>>> something about it, the resources have already been consumed
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> server is probably already overloaded.
>>>>>>>>>>>>>> There are specialised front-end devices and software
>>>>>>>>>>>>>> available, to
>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>> and protect against this kind of attack.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You may want to have a look at the following parameters, but
>>>>>>>>>>>>>> make
>>>>>>>>>>>>>> sure
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> read the caveats (side-effects, interlocking timeouts etc.),
>>>>>>>>>>>>>> otherwise
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> may do more harm than good.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another thing : the settings below are for Apache Tomcat,
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>> in your
>>>>>>>>>>>>>> case is the back-end. It would of course be much better to
>>>>>>>>>>>>>> detect
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> eliminate this at the front-end, or even before.  I had a
>>>>>>>>>>>>>> look at
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> Apache httpd documentation, and could not find a corresponding
>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>> But I am sure that it must exist. You may want to post this
>>>>>>>>>>>>>> same
>>>>>>>>>>>>>> question
>>>>>>>>>>>>>> on the Apache httpd user's list for a better response.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tomcat configuration settings :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AJP Connector : (http://tomcat.apache.org/tomc
>>>>>>>>>>>>>> at-8.5-doc/config/ajp.html#
>>>>>>>>>>>>>> Standard_Implementations)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> connectionTimeout
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after
>>>>>>>>>>>>>> accepting a
>>>>>>>>>>>>>> connection, for the request URI line to be presented. The
>>>>>>>>>>>>>> default
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> AJP protocol connectors is -1 (i.e. infinite).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (You could for example try to set this to 3000 (milliseconds)
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> even
>>>>>>>>>>>>>> lower. That should be more than enough for any legitimate
>>>>>>>>>>>>>> client
>>>>>>>>>>>>>> so
>>>>>>>>>>>>>> send
>>>>>>>>>>>>>> the HTTP request line.  Note however that by doing this at the
>>>>>>>>>>>>>> Tomcat
>>>>>>>>>>>>>> level, you will probably move the problem to the Apache
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> httpd/mod_jk
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>> level.  But at least it might confirm that this is the problem
>>>>>>>>
>>>>>>>>> that you
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>> seeing.  The mod_jk logfile at the httpd level may give you
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>> hints
>>>>>>>>>>>>>> there.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HTTP Connector : (http://tomcat.apache.org/tomc
>>>>>>>>>>>>>> at-8.5-doc/config/http.html#Standard_Implementation)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> connectionTimeout
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after
>>>>>>>>>>>>>> accepting a
>>>>>>>>>>>>>> connection, for the request URI line to be presented. Use a
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>> of -1
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> indicate no (i.e. infinite) timeout. The default value is
>>>>>>>>>>>>>> 60000
>>>>>>>>>>>>>> (i.e.
>>>>>>>>>>>>>> 60
>>>>>>>>>>>>>> seconds) but note that the standard server.xml that ships with
>>>>>>>>>>>>>> Tomcat
>>>>>>>>>>>>>> sets
>>>>>>>>>>>>>> this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> set to
>>>>>>>>>>>>>> false, this timeout will also be used when reading the request
>>>>>>>>>>>>>> body (if
>>>>>>>>>>>>>> any).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 26.11.2016 09:57, Jaaz Portal wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> sorry, its mod_jk no jk2, my typo. All at latest versions. We
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> tried
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>> with
>>>>>>>>
>>>>>>>>> mod proxy too.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is no flood of the server. Nobody is flooding us, they
>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> specific connections after which pool of apache workers is
>>>>>>>>>>>>>>> exhausted
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> blocked
>>>>>>>>>>>>>>> and we need to restart tomcat server.
>>>>>>>>>>>>>>> It is some kind of exploit but do not know how to log it to
>>>>>>>>>>>>>>> obtain
>>>>>>>>>>>>>>> details.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> i had put a limit on connections per client with hope that
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>> but once again, it is not a flood.
>>>>>>>>>>>>>>> They open several connections that are not dropped by apache
>>>>>>>>>>>>>>> when they
>>>>>>>>>>>>>>> disconnect. This way whole pool is quickly exhausted and the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>> broken.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>>> i would like to help you to figure details of this attack but
>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>> production server so it is impossible to much debugging
>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>>> artur
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu <
>>>>>>>>>>>>>>> niranjan.bommu@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> you can find who is flooding site in apache access.log and
>>>>>>>>>>>>>>>> block
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> firewall.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ex to find the IP:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -c|sort
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>> -gr
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal
>>>>>>>>>>>>>>>> <jaazportal@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> we are from some weeks struggling with some Polish hackers
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> bringing our server down. After updating apache to latest
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>> (2.4.23)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and tomcat (8.0.38) available for debian systems we still
>>>>>>>>>>>>>>>> cannot
>>>>>>>>>>>>>>>> secure
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> server.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Today it has stopped to respond again and we needed to
>>>>>>>>>>>>>>>>> restart
>>>>>>>>>>>>>>>>> tomcat
>>>>>>>>>>>>>>>>> process to get it back alive.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There is no too much clues in the logs. The apache
>>>>>>>>>>>>>>>>> error.log
>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>> this line:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid
>>>>>>>>>>>>>>>>> 13385:tid
>>>>>>>>>>>>>>>>> 1397934896385
>>>>>>>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting,
>>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>> raising
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> MaxR
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> equestWorkers setting
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> seems that somehow tomcat, mod-jk2 or even apache is
>>>>>>>>>>>>>>>>> vulnerable to
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> exploit, as we certainly does not have such traffic that
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> block
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> server otherwise
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for now we have increased MaxRequestWorkers and we have
>>>>>>>>>>>>>>>>> limited
>>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> connections from one client to 5 by mod_bw and limited
>>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> simultaneous connections from one ip by iptables but does
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> will help
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> best regards,
>>>>>>>>>>>>>>>>> artur
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Thanks*
>>>>>>>>>>>>>>>> *Niranjan*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This sounds like a variant of the slowloris attack.
>>>>>>>
>>>>>>> This type of attack doesn't take a large number of clients or
>>>>>>> consume a
>>>>>>> large amount of bandwidth.
>>>>>>>
>>>>>>> Basically, the maximum number of connections are made to the server,
>>>>>>> and
>>>>>>> just enough data is sent to each connection in order to not trigger
>>>>>>> the
>>>>>>> timeout. André has explained this in more detail earlier in the
>>>>>>> thread.
>>>>>>> Search for "slowloris attack" for more information.
>>>>>>>
>>>>>>> There are several ways of mitigating this type of attack.
>>>>>>>
>>>>>>> As André has mentioned, placing a dedicated device in front of your
>>>>>>> systems is often the best way. Lots of benefits (platform neutral, no
>>>>>>> stress on your current servers), and some issues (cost, placement /
>>>>>>> access may be an issue with hosted solutions).
>>>>>>>
>>>>>>> However, there are Apache HTTPD modules that can help mitigate these
>>>>>>> types of attacks. Some of them are:
>>>>>>>
>>>>>>> mod_reqtimeout (should be included by default in your Apache HTRPD
>>>>>>> 2.4)
>>>>>>> mod_qos (quality of service module)
>>>>>>> mod_security (application firewall with lots of security rules)
>>>>>>>
>>>>>>> Do a quick search on "slowloris attack apache httpd 2.4" to get some
>>>>>>> ideas.
>>>>>>>
>>>>>>> All of them will probably place additional load on your Apache HTTPD
>>>>>>> server, so make sure that the platform is robust enough to manage the
>>>>>>> additional load.
>>>>>>>
>>>>>>> There is also a beta version of the mod_security module written as a
>>>>>>> servlet filter. It should be possible to build this and configure the
>>>>>>> filter in Tomcat's default web.xml ($CATALINA_BASE/conf/web.xml).
>>>>>>> I've
>>>>>>> not tried this. Also, the code base hasn't seen any activity for 3
>>>>>>> years.
>>>>>>>
>>>>>>> Do a quick search on "modsecurity servlet filter" to find out more
>>>>>>> about
>>>>>>> the servlet filter version of mod_security.
>>>>>>>
>>>>>>> In short, there appear to be some ways to mitigate the attack.
>>>>>>>
>>>>>>> . . . just my two cents
>>>>>>> /mde/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message