tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Re: tomcat thread incurring CPU load
Date Mon, 18 Nov 2019 12:06:17 GMT
Mark and others,

On Mon, 18 Nov 2019 at 12:01, Mark Thomas <markt@apache.org> wrote:

> OK, it looks like I can reproduce this.
>
> Steps to reproduce:
>
> - Windows 2016 Server fully patched
> - Java 1.8.0u144
> - Install Tomcat 8.5.45 from windows installer
> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23
> - Modify server.xml to use Http11AprProtocol on port 8080
> - Make a single request
>
> I then see 1 core running at 100% until the connection times out after
> 20s. Make another request and a core goes back up to 100% for 20s (the
> default keep-alive time out).
>

 I have also successfully reproduced this with making a single request
(sorry for not replying in the weekend). Not sure how your graph looked
like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as
the request hit the server. and it didn't go down at all.

Thanks,

>
> Next steps are to try and track down the root cause.
>
> Mark
>
>
>
> > Mark and M,
> >
> > On 11/13/19 19:31, Mark Thomas wrote:
> >> On November 13, 2019 11:42:34 PM UTC, "M. Manna"
> >> <manmedia@gmail.com> wrote:
> >>> I see this update on Windows which may have been responsible
> >>> (suspicion only, haven’t rolled it back yet)
> >>>
> >>>
> >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr
> > ocode-updates
> >>>
> >>>
> >>>
> > Was 8.5.45 built on Windows 10 in presence of this update ?
> >
> >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully
> >> patched at the time of the build Windows 7 64-bit VM.
> > Also it doesn't matter because binaries don't include CPU microcode.
> >
> > It's more likely that the target system has microcode updates such as
> > these that may negatively impact performance.
> >
> > -chris
> >
> >>>
> >>> Thanks,
> >>>
> >>> On Wed, 13 Nov 2019 at 17:55, M. Manna <manmedia@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz <
> >>>> chris@christopherschultz.net> wrote:
> >>>>
> >> On 11/13/19 11:20, M. Manna wrote:
> >>>>>>> HI Mark,
> >>>>>>>
> >>>>>>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas
> >>>>>>> <markt@apache.org> wrote:
> >>>>>>>
> >>>>>>>> On 12/11/2019 19:11, M. Manna wrote:
> >>>>>>>>> HI Mark,
> >>>>>>>>>
> >>>>>>>>> following my previous reply, we have now confirmed
> >>>>>>>>> that it's indeed
> >>>>>>>> 8.5.45
> >>>>>>>>> with APR 1.2.23 that's causing such high JVM CPU
> >>>>>>>>> usage. We used took out 2 out of 50 servers from
the
> >>>>>>>>> load balancer config, reverted tomcat, and
> >>>>>>>>> redeployed. With near to identical user traffic,
the
> >>>>>>>>> two servers are responding normally without/without
> >>>>>>>>> traffic with 8.5.41. The JVM dump looks a lot better
> >>>>>>>>> with 8.5.41.
> >>>>>>>>>
> >>>>>>>>> We do think that the recent changes in APR and some
> >>>>>>>>> other tomcat jar may have caused compatibility issue
> >>>>>>>>> on Windows server 2016 (64-bit) platform. But
> >>>>>>>>> unfortunately, we cannot pinpoint exactly what change
> >>>>>>>>> may have caused this (i.e. actual OS vs Security
> >>>>>>>>> Updates). With this in mind, we are also being wary
> >>>>>>>>> to move to 8.5.47 as we don't know if the same issue
> >>>>>>>>> will
> >>>>>>>> occur
> >>>>>>>>> again. Since 8.5.41 has been packaged with previously
> >>>>>>>>> accepted
> >>>>>>>> application
> >>>>>>>>> installer, we are more comfortable rolling back.
> >>>>>>>>
> >>>>>>>> Just to confirm, you see this high CPU usage with a
> >>>>>>>> clean install (no additional web applications deployed,
> >>>>>>>> no configuration changes) on Windows 2016 DataCenter
> >>>>>>>> (64-bit)?
> >>>>>>>>
> >>>>>>>> If this is the case, it should be fairly easy to
> >>>>>>>> reproduce.
> >>>>>>>>
> >>>>>>>> Mark
> >>>>>>>>
> >>>>>>>> We do not deploy multiple applications. In fact, Under
> >>>>>>>> tomcat
> >>>>>>> webapps/ROOT we only have one application (ours). Each
> >>>>>>> tomcat instance is hosted on a VM (total 50) and all of
> >>>>>>> them are identically configured (server.xml, web.xml,
> >>>>>>> logging, CPU/RAM). We have not made any other
> >>>>>>> configuration change between 8.5.41 and 8.5.45. And yes,
> >>>>>>> I agree with you that it's fairly easy to reproduce.
> >
> >> I think the question is whether or not your application is
> >> required
> >>>> to
> >> be deployed. Can you reproduce this issue with just the stock
> >> applications bundled with Tomcat?
> >
> >>>>>
> >>>>> My apologies, but our application needs to be deployed. We
> >>>>> have not
> >>>> (or
> >>>>> didn't try in the past) to simply deploy tomcat with stock
> >>>> application (in
> >>>>> other words, simply starting the tomcat OOB) on our prod
> >>>>> servers. This is the first time it has hit us with such
> >>>>> disparity. I’ll try to investigate and get a stock
> >>>>> application data. But we may not be able
> >>>> to do
> >>>>> that quite easily as it’s in our production.
> >>>>>
> >>>>> What I can see is that 3 Windows updates may have been
> >>>>> responsible
> >>>> for
> >>>>> this, but we aren’t sure about that. I’ll let you know if we
> >>>>> can get anything with the stock application instance.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> - -chris
> >>>>>
> >>>>>
> >>>>>
> >>> ---------------------------------------------------------------------
> >>>>>
> >>>
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>> For additional commands, e-mail:
> >>>>> users-help@tomcat.apache.org
> >>>>>
> >>>>>
> >
> >
> >> ---------------------------------------------------------------------
> >
> >
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message