commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan <pantless...@gmail.com>
Subject Pool blocking but not hitting max size?
Date Mon, 13 Apr 2015 14:36:39 GMT
Hello, I'm not a developer on the project I'm supporting, but I have
repeatedly seen this happen and would appreciate some input or advice.
We're using version 1.6 of the commons pool, I don't believe we could
upgrade without good reason.

We're running WebLogic and periodically see thread count shoot up to
the work manager maximum, and throughput grinds to a halt. All threads
are blocked waiting on the commons pool, which I thought was strange
as heap dumps show output like:

Type   Name                           Value
int    evictLastIndex                 -1
ref    _evictor                       null
int    _numActive                     206
ref    _factory
org.springframework.aop.target.CommonsPoolTargetSource @ 0x610...
ref    _pool                          java.util.LinkedList @ 0x610...
long   _softMinEvictableIdleTimeMillis-1
long   _minEvictableIdleTimeMillis    1800000
int    _numTestsPerEvictionRun        3
long   _timeBetweenEvictionRunsMillis -1
boolean_testWhileIdle                 FALSE
boolean_testOnReturn                  FALSE
boolean_testOnBorrow                  FALSE
byte   _whenExhaustedAction           1
long   _maxWait                       -1
int    _maxActive                     4096
int    _minIdle                       10
int    _maxIdle                       8
booleanclosed                         FALSE

So from that, I would have expected numActive to be much closer to
4096 which is the maxActive we set. Am I looking at the wrong place?
Why is this pool blocking? This has happened multiple times, and each
time numActive is between 200 and 300. The pool is being used (I believe)
to limit the number of concurrent security authorizations, so each borrow
then pings an external service, and that external service does not appear
overloaded at all.

I did notice that when I drilled down into the pool's linked list, it
only ever has 8 members in it, I would have expected numActive members
at least. I don't fully understand the pool though.

Here's a partial thread dump showing this behavior (addresses truncated,
but they all blocked on the same pool object):

"[ACTIVE] ExecuteThread: '160' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1f
waiting for monitor entry [0x000...]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
        - waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
        at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '159' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x0000... nid=0x4b1e
waiting for monitor entry [0x000...]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(
GenericObjectPool.java:781)
        - waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
        at org.springframework.aop.target.CommonsPoolTargetSource.getTarget(
CommonsPoolTargetSource.java:244)
--
"[ACTIVE] ExecuteThread: '158' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1d
waiting for monitor entry [0x000...]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
        - waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
        at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '157' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1c
waiting for monitor entry [0x000...]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
        - waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
        at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '156' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1b
waiting for monitor entry [0x000...]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.commons.pool.impl.GenericObjectPool.returnObject(
GenericObjectPool.java:916)
        - waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
        at org.springframework.aop.target.CommonsPoolTargetSource.
releaseTarget(CommonsPoolTargetSource.java:252)
--
"[ACTIVE] ExecuteThread: '155' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=10 tid=0x000... nid=0x4b1a
waiting for monitor entry [0x000...]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(
GenericObjectPool.java:781)
        - waiting to lock <0x000...> (a
org.apache.commons.pool.impl.GenericObjectPool)
        at org.springframework.aop.target.CommonsPoolTargetSource.getTarget(
CommonsPoolTargetSource.java:244)
--
... repeated hundreds of times.

Thanks for any advice you may have.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message