commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy McArthur" <sandy...@apache.org>
Subject Re: pool: stack, cycle and eviction
Date Tue, 30 May 2006 20:57:47 GMT
On 5/30/06, Tom <tbee@tbee.org> wrote:
>
> > It seems to me you are worrying about pool's performance under a light
> > load when what matters is pool's performance under heavy load.
> Agreed and we do use the pools under heavy load. The contrived scenario
> was to show, using graspable numbers, that because of the cyclic nature
> the actual pool size is not determined by the real load on the pool, but
> the duration of the eviction thread. The scenario shows that even though
> the load is only 1, it maintains a size of 6. The example can be
> upscaled by a 1000 but IMHO still holds: if a pool due to a peak has
> sized to, say, 10000, but normally 20 process each obtain one object 50
> times within the eviction time slot (which can be 5 seconds), 1000
> objects will remain in the pool instead of the for the load required 20.
> In this case a overhead of 98%
>
> I do not mean to insult anyone; it was a scenario I stumbled upon when
> discussing pooling mechanism with my system admin, I just am trying to
> confirm if this is real or if I am mistaken. Using a stack (LIFO) with
> an eviction thread would clean up a pool in these scenarios.

I think a LIFO pool plus an evictor will fix the wasted resources
problem under low load following a high load.

> > Personally I don't care if my app isn't perfectly efficient when
> > almost nobody is using it but is still capable of handling a full
> > onslaught of users during peak load times. Pool is pretty good for
> > most people's pooling problems but sure there are gonna be situations
> > that would benefit from less generic pooling implementations.
>
> I think stack is a generic implementation (especially since GOP is
> implemented as a stack in 1.2 ;-) )
>
>
> > Under heavy load a FIFO (queue or cyclic) pool does a better job of
> > distributing load. Also in the case of a validating pool being large
> > because of a previous peak load a FIFO pool will do a better job self
> > managing it's size if idle objects time out before being reused.
> On the first account I agree. However if the pool is used to provide
> multiple connections to the same resource, load balancing is not the
> issue. Or if the resource is uniquely allocated while an object is
> reserved, it is not either (once the object is returned to the pool, the
> resource is fully available again).
>
> On the second account I believe my scenario still needs disproving.
>
>
> >
> > Using an evictor validates idle objects can cause a deadlock.
>
> Since the sequence of objects in the stack is random,

Umm, that statement is incorrect. The idle objects in a stack based
pool would have the stalest objects on the bottom. It may be random
with respect to the order the object were created but the order with
respect to when they were returned is very specific.

> I believe it is
> not actually necessary to remove the actual stale objects, only the
> number of stale objects.

In the case of pooling objects that represent represent remote
resources such as JDBC connections or other types of sockets the
stalest idle objects are most likely to have been disconnected by the
other end and are of least value to the application. These idle
objects should be removed first as they are most likely to fail
validation and cause more work to do later when validating an object
before returning it from the pool.

> If the evictor uses the same access as a regular client,

It doesn't use the same access as a regular client. Otherwise the
evictor would be an external process to the pool and wouldn't have
special access to the stalest objects first.

> I do not see why it would deadlock. (I understand your
> locking explanation.) Again at the risk of insulting people: if the pool
> eviction thread uses a two stage approach:
> 1) decide how many objects need to be evicted
> 2) using the regular client call and borrow the determined number of
> objects from the pool, but then use an inside call to destroy the object

Are you saying the evictor should call borrowObject() and then call
invalidateObject() for however many objects need to be evicted?

> I think the locking will not be an issue or am I mistaken?

The evictor would need special knowledge of the client code to know
the "regular client call" method. I guess pool could come up with
another interface to implement for the evictor so it could acquire
needed locks before acquiring locks internal to the pool. While
creating yet another interface to implement is the most
straightforward solution to solving this deadlock it isn't the only
one.

Here is an example from Dbcp if it had eviction running:

Thread A:
1. Asks Dbcp for a new JDBC connection.
2. Dbcp synchronizes itself as needed and then asks it's internal Pool
for an existing JDBC connection.
3. Pool synchronizes itself and realized it needs to create a new
connection instance.
4. Pool calls out to the PoolableObjectFactory.makeObject method which
goes back into Dbcp code that also synchronizes on the Dbcp locks.

Thread Evictor:
1. Asks pool to synchronize on itself and do some idle object eviction.
2.a (with validation) For each idle object Pool validates, it needs to
PoolableObjectFactory.activeObject, validateObject, and then either
passiveObject or destroyObject. Any of those methods can synchronized
on the internal Dbcp locks mentioned above.
2.b (without validation) For each idle object Pool checks the idle
timeout, it needs to check some internal last used timestamp and then
possible call destroyObject. Any destroyObject can synchronized on the
internal Dbcp locks mentioned above.

Thread A acquires DbcpLock and then PoolLock in that order. (The
second acquisition of DbcpLock isn't relevant as it is already
acquired.)
Thread Evictor acquires PoolLock and then maybe DbcpLock no matter how
you configure it to behave. So yea, the dead lock is still an issue.

Now, this deadlock issue goes away if the Pool never calls out to the
PoolableObjectFactory methods while it is holding it's internal
"PoolLock".

> > Probably because no one has submitted an implementation nor have any
> > commiters felt there was the demand to spend time on it.
> That would be my trigger then, if not for:
>
> >
> > That said, the unreleased pool 2.0, currently in the trunk, has a new
> > pool dubbed the composite pool which can be configured to behave as a
> > LIFO (stack) and has an idle evictor.
>
> Which means that I will not do double work. Any estimate on the release
> date?

Not really, just when it's done. My spare time for pool has decreased
with the approach of my wedding. But you can always get the code from
subversion trunk or a nightly build but do realize they are
unsupported and the API of new code may not be stable yet. Feed back
is welcomes though.

trunk: http://svn.apache.org/repos/asf/jakarta/commons/proper/pool/trunk/
nightly: http://people.apache.org/builds/jakarta-commons/nightly/commons-pool/

Personally, I think the composite pool code in the trunk is quite
robust and solid. The current problem is it is not really any faster
than the existing pool implementations.

I'm slowly working on a refactoring that seems to scale almost
exponentially performance wise with the number of CPUs but it
currently doesn't implement the maxActive or maxIdle limits correctly,
run the unit tests and you'll see. One of nice the side effects of
this refactoring is that the deadlock I described above almost goes
away because the required locking is so minimally scoped. After I fix
the maxActive/Idle problems I'll try to eliminate the risk of deadlock
completely. I may punt this until Pool 3.0 where we require Java 5 and
the java.util.concurrent classes are available.

refactoring: http://svn.apache.org/repos/asf/jakarta/commons/proper/pool/branches/performance-ideas/

> But I might have an addition to the pooling logic: I created a
> ObjectPoolWrapper to get rid of those nasty checked exceptions (yes,
> each his own preference) and used that to add a "shutdown" mode.
> Shutdown is a kind of delayed close: when in shutdown the pool does not
> allow borrowing new objects, but does accept returning of objects (and
> destroys them immediately) until all active objects have been returned
> and then the pool closes.

Pool 2.0 implements that "shutdown mode" behavior. I hope to remove
the "nasty checked exceptions" in Pool 3.0:
http://wiki.apache.org/jakarta-commons/PoolRoadMap Since removing
checked exceptions from an interface breaks API compatibility with
existing implementations I'm not eager to push that change into Pool
2.0.

-- 
Sandy McArthur

"He who dares not offend cannot be honest."
- Thomas Paine

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org


Mime
View raw message