tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: tomcat thread incurring CPU load
Date Mon, 18 Nov 2019 12:00:59 GMT
OK, it looks like I can reproduce this.

Steps to reproduce:

- Windows 2016 Server fully patched
- Java 1.8.0u144
- Install Tomcat 8.5.45 from windows installer
- Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23
- Modify server.xml to use Http11AprProtocol on port 8080
- Make a single request

I then see 1 core running at 100% until the connection times out after
20s. Make another request and a core goes back up to 100% for 20s (the
default keep-alive time out).

Next steps are to try and track down the root cause.

Mark



> Mark and M,
> 
> On 11/13/19 19:31, Mark Thomas wrote:
>> On November 13, 2019 11:42:34 PM UTC, "M. Manna"
>> <manmedia@gmail.com> wrote:
>>> I see this update on Windows which may have been responsible
>>> (suspicion only, haven’t rolled it back yet)
>>>
>>>
>>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr
> ocode-updates
>>>
>>>
>>>
> Was 8.5.45 built on Windows 10 in presence of this update ?
> 
>> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully 
>> patched at the time of the build Windows 7 64-bit VM.
> Also it doesn't matter because binaries don't include CPU microcode.
> 
> It's more likely that the target system has microcode updates such as
> these that may negatively impact performance.
> 
> -chris
> 
>>>
>>> Thanks,
>>>
>>> On Wed, 13 Nov 2019 at 17:55, M. Manna <manmedia@gmail.com>
>>> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < 
>>>> chris@christopherschultz.net> wrote:
>>>>
>> On 11/13/19 11:20, M. Manna wrote:
>>>>>>> HI Mark,
>>>>>>>
>>>>>>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas
>>>>>>> <markt@apache.org> wrote:
>>>>>>>
>>>>>>>> On 12/11/2019 19:11, M. Manna wrote:
>>>>>>>>> HI Mark,
>>>>>>>>>
>>>>>>>>> following my previous reply, we have now confirmed
>>>>>>>>> that it's indeed
>>>>>>>> 8.5.45
>>>>>>>>> with APR 1.2.23 that's causing such high JVM CPU
>>>>>>>>> usage. We used took out 2 out of 50 servers from the
>>>>>>>>> load balancer config, reverted tomcat, and
>>>>>>>>> redeployed. With near to identical user traffic, the
>>>>>>>>> two servers are responding normally without/without
>>>>>>>>> traffic with 8.5.41. The JVM dump looks a lot better
>>>>>>>>> with 8.5.41.
>>>>>>>>>
>>>>>>>>> We do think that the recent changes in APR and some
>>>>>>>>> other tomcat jar may have caused compatibility issue
>>>>>>>>> on Windows server 2016 (64-bit) platform. But
>>>>>>>>> unfortunately, we cannot pinpoint exactly what change
>>>>>>>>> may have caused this (i.e. actual OS vs Security
>>>>>>>>> Updates). With this in mind, we are also being wary
>>>>>>>>> to move to 8.5.47 as we don't know if the same issue
>>>>>>>>> will
>>>>>>>> occur
>>>>>>>>> again. Since 8.5.41 has been packaged with previously
>>>>>>>>> accepted
>>>>>>>> application
>>>>>>>>> installer, we are more comfortable rolling back.
>>>>>>>>
>>>>>>>> Just to confirm, you see this high CPU usage with a
>>>>>>>> clean install (no additional web applications deployed,
>>>>>>>> no configuration changes) on Windows 2016 DataCenter
>>>>>>>> (64-bit)?
>>>>>>>>
>>>>>>>> If this is the case, it should be fairly easy to
>>>>>>>> reproduce.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>> We do not deploy multiple applications. In fact, Under
>>>>>>>> tomcat
>>>>>>> webapps/ROOT we only have one application (ours). Each
>>>>>>> tomcat instance is hosted on a VM (total 50) and all of
>>>>>>> them are identically configured (server.xml, web.xml,
>>>>>>> logging, CPU/RAM). We have not made any other
>>>>>>> configuration change between 8.5.41 and 8.5.45. And yes,
>>>>>>> I agree with you that it's fairly easy to reproduce.
> 
>> I think the question is whether or not your application is
>> required
>>>> to
>> be deployed. Can you reproduce this issue with just the stock 
>> applications bundled with Tomcat?
> 
>>>>>
>>>>> My apologies, but our application needs to be deployed. We
>>>>> have not
>>>> (or
>>>>> didn't try in the past) to simply deploy tomcat with stock
>>>> application (in
>>>>> other words, simply starting the tomcat OOB) on our prod
>>>>> servers. This is the first time it has hit us with such
>>>>> disparity. I’ll try to investigate and get a stock
>>>>> application data. But we may not be able
>>>> to do
>>>>> that quite easily as it’s in our production.
>>>>>
>>>>> What I can see is that 3 Windows updates may have been
>>>>> responsible
>>>> for
>>>>> this, but we aren’t sure about that. I’ll let you know if we
>>>>> can get anything with the stock application instance.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> - -chris
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>>>>
>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail:
>>>>> users-help@tomcat.apache.org
>>>>>
>>>>>
> 
> 
>> ---------------------------------------------------------------------
> 
> 
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message