tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Upgrade to 8.5.8/9
Date Thu, 22 Dec 2016 18:47:01 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Greg,

On 12/22/16 12:50 PM, Greg Huber wrote:
> Thanks, I have been running this setup for ages, 8 plus years and
> various tomcat versions.  It has never failed on  <= 8.0.x

Interesting. Usually, things don't just stop working like that.

Did you change anything besides replacing Tomcat 8.0.x with 8.5.x?

> I had to stop/start a couple of times to get it to fail and had to 
> run a parallel maven build to slow things down.

Does that "slow down" cause your database to slow-down as well?
Long-running queries can pile-up on top of one another sometimes.

In dev, I always run with pool size=1 so I can catch problem early. Is
it possible that you have a very small pool size and even a little bit
of load can bring everything to a crawl?

> Do not read much on dumps, but what would this -locked be?
> 
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoi
nt.java:1437)
>
> 
at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase
.java:49)
>
> 
- - locked <0x00000000f1c54ed8> (a
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
ava:1142)
>
> 
at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:617)
>
> 
at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThr
ead.java:61)
>
> 
at java.lang.Thread.run(Thread.java:745)

"Locked" means that the thread holds a lock on a monitor. Basically,
the thread has synchronized on something and it's telling you what the
object id is (I think that's just a pointer, actually) and its type
(an NioSocketWrapper).

You'll see the same thing for the datasource: a thread has locked the
data source's latch object while it waits for a connection to become
available.

- -chris

> On 22 December 2016 at 16:24, Christopher Schultz < 
> chris@christopherschultz.net> wrote:
> 
> Greg,
> 
> On 12/22/16 5:18 AM, Greg Huber wrote:
>>>> Here is the first dump and I cannot send all three.  Its my
>>>> prod server which is locked up at the moment.  I need to get
>>>> it back on line so is this enough info?  I can emile the
>>>> whole file which has the three dumps if needed.
>>>> 
>>>> [snip]
>>>> 
>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25
>>>> mixed mode):
>>>> 
>>>> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0 
>>>> tid=0x000000000cb10000 nid=0x1960 in Object.wait() 
>>>> [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f2879c10> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Waiting on a database connection.
> 
> 
>>>> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0 
>>>> tid=0x000000000c74f800 nid=0x195d in Object.wait() 
>>>> [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f2852128> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Another thread waiting for a db connection.
> 
>>>> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0 
>>>> tid=0x000000000dd3b000 nid=0x1959 in Object.wait() 
>>>> [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f282a818> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> I'm starting to see a pattern, here.
> 
>>>> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0 
>>>> tid=0x000000000cbe6000 nid=0x1953 in Object.wait() 
>>>> [0x00002ba658988000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f28030f8> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Hmm. Could it be...
> 
>>>> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0 
>>>> tid=0x000000000e797800 nid=0x194f in Object.wait() 
>>>> [0x00002ba659472000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f27daf28> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> ... just possibly ...
> 
>>>> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0 
>>>> tid=0x000000000e874000 nid=0x194d in Object.wait() 
>>>> [0x00002ba656c67000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f27b3d20> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> ... that you have run out of ...
> 
>>>> "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55 
>>>> prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting
>>>> on condition [0x00002ba658c50000] java.lang.Thread.State: 
>>>> TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native
>>>> Method) at 
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(Job
Sto
>
>>>> 
reSupport.java:3992)
> 
> (Quartz
>>>> 
> waiting around for some event. Ignore this one in erms of my 
> overarching narrative.)
> 
>>>> "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0 
>>>> tid=0x000000000e44f000 nid=0x194a in Object.wait() 
>>>> [0x00002ba658948000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f1f541b8> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Database connections?
> 
>>>> "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0 
>>>> tid=0x000000000cc2e800 nid=0x1940 in Object.wait() 
>>>> [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at 
>>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36
)
>
>>>> 
> Ooh!
>>>> 
> This one is doing something different. I suspect any ms it will ask
> for another db connection.
> 
>>>> "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0 
>>>> tid=0x000000000cc7b800 nid=0x1933 in Object.wait() 
>>>> [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f1d766c0> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> That's more like it.
> 
>>>> "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0 
>>>> tid=0x000000000cb83000 nid=0x1930 in Object.wait() 
>>>> [0x00002ba658885000] java.lang.Thread.State: WAITING (on
>>>> object monitor) at java.lang.Object.wait(Native Method) at 
>>>> java.lang.Object.wait(Object.java:502) at 
>>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
>>>> 
ectPool.java:1104)
>>>> 
>>>> 
> - locked <0x00000000f1d4e918> (a
>>>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> I'm reminded of a Far Side cartoon. 
> [http://grahammercer.com.au/humour/GodMakesTheSnake.jpg]
> 
>>>> "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0 
>>>> tid=0x000000000ea8b000 nid=0x192e in Object.wait() 
>>>> [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at 
>>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36
)
>
>>>> 
> Another
>>>> 
> thread caught in the filter. Filters are usually quick, so I think
> this is a transient state. Probably about to request a database 
> connection.
> 
>>>> "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0 
>>>> tid=0x000000000c92a000 nid=0x1928 in Object.wait() 
>>>> [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at 
>>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36
)
>
>>>> 
> Here,
>>>> 
> too.
> 
> 
> Okay, I'm done reading the thread dump.
> 
> Have a look at your database pool's size and "active" counts[1].
> You may find that your application is locked-up waiting on db
> connections. you may have a resource leak (db connections).
> 
> -chris
> 
> [1] 
> http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20A
pa
>
> 
che%20Tomcat%20with%20JMX.pdf,
> slide 16
>> 
>> ---------------------------------------------------------------------
>>
>> 
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>> 
>> 
> 
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYXB+lAAoJEBzwKT+lPKRYwx8QALOUcVOjkG9HM34b+JYjpCtP
pqNolieQShhcyqCIBHmtfX/ftdtbbhfEvJq2VlOXr/Bos94sA5dVFG1K5FMepJ5b
EZKON1sQp8FPRqahs9nsOoQVYivV/+6DO4gPKbVtRedy+qbOlqFvnLJGpyXTew7s
3ljMxRfgqppy12B2KvXnOCG07Zbr8l9GNWPJja7fxVTax9UA4CE2LHmsx9W7ZdqW
WCmo3L+ELuF4ZLwPEaYat6sSrrrMNx47xFv5oXxjE92whyNnILGzULpnNZKIFki6
HLVj/TTbyff0pygUeztIJV9+DuWHw05LGi0WB7G6YZ6bxy7bFliPBLmG8HvjPWd0
AiC5OtQ13FYauRPeW7jn/4HeUjvrRWTmwRNDn+AbUlHvSVrDeET5pPENltC5KzF2
ja/1MAFvyQECY4jGIqHTn5IWICmjHwnP2PFPqTN6AUiDHqheKT2oemWAWPxmesdd
5zn5igU8AI6N9dZrXpniDOiiK+YaECmBdPfrPnBx/p5Ni4Ne+HTJNE4YywVtRNb2
2fz9DaOlFTlD+FkbyVqRBgJQT1m6ygfyQ0+PUf6FS80mOwmPE9C9QkKAFgOui046
NdZ7viClzIiCmsmR1JXgVkV+LAfmcZfkwjU+zBDmbLu7/G1Y8Q8U2tFRdvax1n8f
vN4WY1nm3xQEqbUL5jI6
=Mp0Q
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message