tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Castilla | thin(k)design" ...@thin-k-design.com>
Subject Re: Crash in http connector random once a day
Date Fri, 15 Mar 2019 11:56:59 GMT
>
> In each of the four thread dumps, there are (the same) 4 threads
> connected to and waiting for an answer from a PHP FCGI backend behind
> Tomcat. You need to investigate, why that backend isn't sending a
> response (or takes so long for it).
>
> Regards,
>
> Rainer
>

Hi Rainer,

I have debugged PHP, the process ends normally and the response is
sent 100% of the time, so I don't know why the tomcat thread keeps
listening to it.

There are always a maximum of 4 threads that keeps reading from the
PHP Input Stream. As long as there are 4 "hanging" threads all other
requests to tomcat work perfectly, which is very strange.

The thread dumps always show this:

       java.lang.Thread.State: RUNNABLE
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(Unknown Source)
       at java.net.SocketInputStream.read(Unknown Source)
       at php.java.fastcgi.FCGIInputStream.read(FCGIInputStream.java:39)
       ...

At some point (up to 24 hours) tomcat "cleans" the threads and they
stop. And after a while it begins with the failures until there are
again 4 "hanging" threads.

If I look in VisualVM in the tab MBeans at the PhpCGIServlet of those
processes, there is a maxTime of 3406, so PHP should not be the
problem.

A heapdump shows nothing unusual I think:

       Total Bytes: 35.381.250
       Total Classes: 6.655
       Total Instances: 454.066
       Classloaders: 178
       GC Roots: 3.318
       Number of Objects Pending for Finalization: 0

I am really out of ideas...





> Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:
> >>> From: Daniel Castilla | thin(k)design [mailto:dc@thin-k-design.com]
> >>> Sent: Thursday, March 14, 2019 12:37 PM
> >>> To: Jäkel, Guido <G.Jaekel@dnb.de>; users@tomcat.apache.org
> >>> Subject: Re: Crash in http connector random once a day
> >>>
> >>> Dear Guido,
> >>>
> >>> thanks for the reply. The requests are reaching tomcat, and a thread
> >>> is always started, if I look at the current threads on the tomcat
> >>> manager I see the following, there are 4 threads that are processing
> >>> since 2+ hours:
> >>>
> >>> R ? ? ? ? ? ?
> >>> S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET /manager/status
HTTP/1.1
> >>> S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST /cloudworx/?method=words&id=17385
HTTP/1.1
> >>> S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST /cloudworx/?method=words&id=18986
HTTP/1.1
> >>> S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST /cloudworx/?method=words&id=10560
HTTP/1.1
> >>> R ? ? ? ? ? ?
> >>> S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST >/cloudworx/?method=words&id=10560
HTTP/1.1
> >>>
> >>> I'm not sure what other metrics would be helpful, but your Unix script
> >>> wouldn't help much, as I am on a Windows Server 2012 and I would like
> >>> to avoid installing Cygwin or something similar.
> >>
> >> Dear Daniel,
> >>
> >> the script just read-out the same core data and does some pretty print. You
may do it by your own by doing a HTTP-Request against
> >>
> >>     URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
> >>
> >>
> >> What is interesting from your snippet that there are POST Request "in Service"
(i.e. Progress) since more than 2h. And you told that the ID is unique, but there are two
times a '10560'.
> >>
> >> Is there a database service involved in the backend?
> >>
> >> BTW: In addition to pull static thread dumps you may use JVisualVM to get a
live view to the threads.
> >
> > Dear Guido and Mark,
> >
> > the same Id '10560' appears twice in the URL as the same request was
> > retried for two times, the third time it worked.
> > The other two URLs had only one retry, the second time it worked.
> >
> > You can find a complete thread dump here (I hope the link gets
> > through): https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing
> >
> > I had to restart tomcat now in order to access the jmxproxy as it
> > wasn't active in the user roles, so the active "problem" processes are
> > gone for now. But it will happen again, at latest tomorrow.
> >
> > I tried JVisualVM, but I am no Java Expert and don't know what to search for...
> >
> > Thanks
> > Daniel

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message