tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Upgrade to 8.5.8/9
Date Thu, 22 Dec 2016 16:24:15 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Greg,

On 12/22/16 5:18 AM, Greg Huber wrote:
> Here is the first dump and I cannot send all three.  Its my prod
> server which is locked up at the moment.  I need to get it back on
> line so is this enough info?  I can emile the whole file which has
> the three dumps if needed.
> 
> [snip]
> 
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> mode):
> 
> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
> tid=0x000000000cb10000 nid=0x1960 in Object.wait()
> [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f2879c10> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Waiting on a database connection.


> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
> tid=0x000000000c74f800 nid=0x195d in Object.wait()
> [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f2852128> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Another thread waiting for a db connection.

> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
> tid=0x000000000dd3b000 nid=0x1959 in Object.wait()
> [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f282a818> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

I'm starting to see a pattern, here.

> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
> tid=0x000000000cbe6000 nid=0x1953 in Object.wait()
> [0x00002ba658988000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f28030f8> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Hmm. Could it be...

> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
> tid=0x000000000e797800 nid=0x194f in Object.wait()
> [0x00002ba659472000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f27daf28> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

... just possibly ...

> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0
> tid=0x000000000e874000 nid=0x194d in Object.wait()
> [0x00002ba656c67000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f27b3d20> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

... that you have run out of ...

> "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55
> prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting on
> condition [0x00002ba658c50000] java.lang.Thread.State:
> TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) 
> at 
> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobSto
reSupport.java:3992)

(Quartz
> 
waiting around for some event. Ignore this one in erms of my
overarching narrative.)

> "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0
> tid=0x000000000e44f000 nid=0x194a in Object.wait()
> [0x00002ba658948000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f1f541b8> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Database connections?

> "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0
> tid=0x000000000cc2e800 nid=0x1940 in Object.wait()
> [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at
> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)

Ooh!
> 
This one is doing something different. I suspect any ms it will
ask for another db connection.

> "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0
> tid=0x000000000cc7b800 nid=0x1933 in Object.wait()
> [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f1d766c0> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

That's more like it.

> "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0
> tid=0x000000000cb83000 nid=0x1930 in Object.wait()
> [0x00002ba658885000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0x00000000f1d4e918> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

I'm reminded of a Far Side cartoon.
[http://grahammercer.com.au/humour/GodMakesTheSnake.jpg]

> "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0
> tid=0x000000000ea8b000 nid=0x192e in Object.wait()
> [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at
> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)

Another
> 
thread caught in the filter. Filters are usually quick, so I
think this is a transient state. Probably about to request a database
connection.

> "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0
> tid=0x000000000c92a000 nid=0x1928 in Object.wait()
> [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at
> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)

Here,
> 
too.


Okay, I'm done reading the thread dump.

Have a look at your database pool's size and "active" counts[1]. You
may find that your application is locked-up waiting on db connections.
you may have a resource leak (db connections).

- -chris

[1]
http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20Apa
che%20Tomcat%20with%20JMX.pdf,
slide 16
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYW/4vAAoJEBzwKT+lPKRYj8QP/jB4YkmoyLwS6z1RZ4XnV9Gw
43b2aoSdhDihonAao+Zu23+PQCELkzhv06hA2xZ33fW+HFem7cMw3isVh1NTxtVO
1fKsS4ZVftMnjHrFSMN9dzWnClf2vnDgN78to/tsEJGkinP7wJL55qZkdETJF4FO
L3vMabMT1LeOUo8bdJfYKJXnA2vrjoU7W8vLP883LZy3YMUCdVclFOG10v/kr7vB
oSWtSXBlq83fFJasXCSXEH7YvLteY1+jqrOx87JTeKgnYIkBF1KujMKuqw0ADRig
brYegISWAdjs9ovyeTh4PKyWc29lNp7jwRcEMOwh1XF7WgV1msW3awqdIdPnNrtt
1kKLNXJv/aKILwMRCWQyR9/2jCsQygOE3Q5qXAjkjMe8qfORZYDdPQlZzpEuGt3U
IMrRbpUN+jhMLKtfs/PWsuCker5rdGIIQu/SkCiYqpSDPiRpXLTteOW82SjxvLN/
QrasunCCz3N3Y0XEFJbkuE49lNYRruYuhEpmxXYuB+Xv6GOQzic9FYfztwq9z41S
AvOjCiTCSacpScPoB9k3q8YGdxZniKQG6pgm0p+NoQYTO8mid/gznftr35GO+Dw8
c1zREZd5qv858CON1ELthxTFSn43Bte7bxfFl9ZvZ8DOWXEYi8P7fDJcuoBDoxuK
Ag1tPGX3fMKaeC5Bwyw9
=ogwm
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message