tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: tomcat thread incurring CPU load
Date Wed, 13 Nov 2019 15:38:09 GMT
On 12/11/2019 19:11, M. Manna wrote:
> HI Mark,
> 
> following my previous reply, we have now confirmed that it's indeed 8.5.45
> with APR 1.2.23 that's causing such high JVM CPU usage.
> We used took out 2 out of 50 servers from the load balancer config,
> reverted tomcat, and redeployed. With near to identical user traffic, the
> two servers are responding normally without/without traffic with 8.5.41.
> The JVM dump looks a lot better with 8.5.41.
> 
> We do think that the recent changes in APR and some other tomcat jar may
> have caused compatibility issue on Windows server 2016 (64-bit) platform.
> But unfortunately, we cannot pinpoint exactly what change may have caused
> this (i.e. actual OS vs Security Updates). With this in mind, we are also
> being wary to move to 8.5.47 as we don't know if the same issue will occur
> again. Since 8.5.41 has been packaged with previously accepted application
> installer, we are more comfortable rolling back.

Just to confirm, you see this high CPU usage with a clean install (no
additional web applications deployed, no configuration changes) on
Windows 2016 DataCenter (64-bit)?

If this is the case, it should be fairly easy to reproduce.

Mark


> 
> I would appreciate if this can be looked into.
> 
> On Tue, 12 Nov 2019 at 11:27, M. Manna <manmedia@gmail.com> wrote:
> 
>> Hey Mark (appreciate your response in US holiday time)
>>
>> On Tue, 12 Nov 2019 at 07:51, Mark Thomas <markt@apache.org> wrote:
>>
>>> On November 12, 2019 12:54:53 AM UTC, "M. Manna" <manmedia@gmail.com>
>>> wrote:
>>>> Just to give an update again:
>>>>
>>>> 1) We reverted the APR to 1.2.21 - but observed no difference.
>>>> 2) We took 3 thread dumps over 1 min interval (without any user
>>>> sessions) -
>>>> All threads are tomcat's internal pool threads.
>>>>
>>>> When we checked the thread details (using fasthread.io) - we didn't see
>>>> any
>>>> of our application stack. Since there is no user traffic, this is
>>>> coming
>>> >from tomcat internally. At this stage, we cannot really figure out
>>>> what's
>>>> the root cause.
>>>>
>>>> Any help is appreciated.
>>>
>>> Migrated from what (full version info please)?
>>>
>>  from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June)
>>
>>>
>>> Operating system exact version?
>>>
>>  Microsoft Windows Server 2016 DataCentre (64-bit)
>>
>>>
>>> JRE vendor and exact version?
>>>
>>  C:\jdk1.8.0\bin>java.exe -version
>> java version "1.8.0_162"
>> Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
>> Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
>>
>>
>>> Do you see the same behavior with the latest 8.5.x and latest Tomcat
>>> Native?
>>>
>>   We are using APR 1.2.23 which I can also see in latest tomcat. Due to
>> production due diligence we cannot roll to a different version that easily.
>> Normally, we lag behind by 2 monthly releases of tomcat. We also reverted
>> the APR to 1.2.21 (but no difference).
>>
>>>
>>> What triggers this behaviour?
>>>
>>  That is quite strange. Due to US holidays, we had a low traffic on our
>> servers, and nothing has crept in to suggest that it's application-driven.
>> We took one tomcat instance out of 50 instances and removed all user
>> sessions (i.e. no application activities or threads). Upon restart of
>> tomcat, the CPU spike lingered past the initial servlet startup period. We
>> monitored that over 1-2 hours but there was no difference.
>>
>>>
>>> How often do you see this behaviour?
>>>
>> We took 2 sets of data
>> 1) 3 Jstack dump based on 10 seconds interval.
>> 2) 3 jstack dump based on 1 min interval.
>>
>> Both the above reveals that all background threads (http, pool etc.) were
>> from tomcat. We didn't have any application threads lingered in those 3
>> samples. So yes we see this almost all the time if we take samples.
>> However, when we compared with pre-production instances (with Windows
>> server R2 x64 bit), we don't see such abnormal spike. In fact, the
>> application instance doesn't incur such a big CPU spike. Whilst composing
>> this email, I am now thinking if the APR is indeed incompatible with
>> WIndows Server R2 (or the presence of any Windows Updates) which blocks the
>> native poll() call longer than usual.
>>
>> An example is that on Windows Server 2012 - APR poll() call takes about
>> 30% CPU time - but with Windows Server 2016 it's almost always 95%.
>>
>>
>>>
>>> And anything else you think might be relevant.
>>>
>>
>> We are using end-2-end encryption using APR (with Certificate and
>> SSLConfig resource setup in server.xml). But it's survived past 3 tomcat
>> upgrades without any issue.
>> Except OS we don't have any obvious culprit identified at the moment.
>>
>> Thanks,
>>
>>>
>>> Mark
>>>
>>>>
>>>> Thanks,
>>>>
>>>> On Mon, 11 Nov 2019 at 20:57, M. Manna <manmedia@gmail.com> wrote:
>>>>
>>>>> Hello All,
>>>>>
>>>>> Any thoughts regarding this? Slightly clueless at this point, so any
>>>>> direction will be appreciated.
>>>>>
>>>>> We are seeing the poll taking all the CPU time. We are using
>>>>> OperatingSystemMXBean.getProcessCpuLoad() and
>>>>> OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then
>>>> x100 to
>>>>> get the pct).
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> On Mon, 11 Nov 2019 at 17:46, M. Manna <manmedia@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> after migrating to 8.5.45, we are seeing a lot of cpu load by
>>>> following
>>>>>> JVM thread dump:
>>>>>>
>>>>>> "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE :
>>>>>> cpu=172902703125000 : cpuLoad= 74.181015
>>>>>>
>>>>>> BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1
>>>>>> LockOwnerName:null
>>>>>>
>>>>>> WaitedCount:5397 WaitedTime:0 InNative:false IsSuspended:false at
>>>>>> org.apache.tomcat.jni.Poll.poll(Poll.java:-2)
>>>>>>
>>>>>>             at
>>>>>>
>>>> org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1547)
>>>>>>
>>>>>>             at java.lang.Thread.run(Thread.java:748)
>>>>>>
>>>>>>
>>>>>> These are coming after 2-3 successful jvm dump. Is this something
>>>>>> familiar to anybody?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message