commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phil Steitz" <phil.ste...@gmail.com>
Subject Re: [POOL] Offer of help for a 1.4 release
Date Sat, 05 Jan 2008 01:28:51 GMT
On Jan 4, 2008 4:09 AM,  <kutzi@gmx.de> wrote:
> Mark, Thomas, thanks for your replies,
>
> Phil Steitz wrote:
> > On Jan 3, 2008 12:40 PM, Mark Thomas <markt@apache.org> wrote:
> >> Christoph Kutzinski wrote:
> >>> - creating a new object means the pool is exhausted which in turn usually
means that we have a high-load situation.
> >>> - creation of new objects is expensive (probably even more in high-load
situations). This is why we originally used the pool
> >>> - so in conclusion it is probably a bad idea to create multiple object in
parallel
> >> I don't see how serializing object creation can help performance. If you
> >> have a test case and some numbers that show otherwise, I would be very
> >> interested in taking a look.
>
> I have no test case for this, I just have my reasoning and my
> observation on our live system that connection creation (lets call the
> 'objects' database connections as that is probably 95% of the use cases
> of commons-pool :-) ) takes much longer under high load situations:
> While connection creation in 'idle' mode takes something between 100 and
> 200 ms, it takes several seconds (the longest I've seen was 27 seconds!)
> under peak loads.
>
>
> >>
> >> If you are really worried about the cost of object creation then you can
> >> configure the pool to create all the objects at start-up and block until a
> >> free object is available.
>
>
> That is unfortunately not possible under our current configuration as we
> have set up our application servers to use all connections our database
> server can handle when their pools have reached their maximum size.
> For example: we have 40 application servers with a pool max-size of 40.
> Our database server can just handle (because of its memory
> configuration) 1600 connections.
> If we would configure the pools to fetch all connections at startup, we
> would lose the ability to do updates to our application-software (we
> have a 2-stage approach to doing updates: we startup the 2nd stage with
> the new software then configure the load balancer to use the 2nd stage
> and only afterwards stop the 1st stage) without major hassle.
>
So I guess you use the idle object evictor to trim back the
connections in use by the pools and do the maintenance under low load
conditions where you can count on only about 50% max connection
utilization?  Am I following this correctly?  I am not sure I follow
what you are doing here.
>
>
>
> > Thanks for the feedback, Christoph; but I agree with Mark.  I suspect
> > most pool users keep the default whenExhaustedAction, which is to
> > block.  That means that objects get created a) on demand, when there
> > are no idle instances, but maxActive has not been reached b) when
> > addObject is invoked to prefill or augment the idle object pool
> > explicitly or c) when minIdle is set and the idle object evictor runs.
> >  Even when a) happens during a load spike, it is better to do the
> > makes in parallel, especially if there is some latency involved and
> > there are resources available to process the makes in parallel (which
> > will be the case in, e.g. a database connection pool).
>
> Phil, I cannot follow your reasoning here. What makes you think that
> there are "resources available to process the makes in parallel"? What
> resources do you think of anyway? I'm thinking about resources as
> processor-cycles on the database server and these are usually not
> available during peak load times.
>
I understand now what you meant.  What I meant was really that the
operation could be done in parallel.

> I still think that serial connection creation is a good thing as it will
> help to keep unnecessary load from the database server:
> As connections borrowed from the pool are held only for a comparably
> short time (at least in our case), the probability that a connection was
> returned to pool by a different thread in the near future is quite high.
>
> So, by serializing connection creation and rechecking, if a connection
> is available, before starting to create a new one, you won't burden the
> db-server with unnecessary load.
>

The 1.2 / 1.4-RC1 code does "recheck" before initiating additional
makes - i.e., it will not initiate a makeObject if an idle object has
been returned to the pool or if maxActive has been reached.  I think I
understand your point though, but again it doesn't seem natural to use
client thread synchronization in the connection pool as a load
dampening mechanism for the database.

> Another thing to consider: If the db-server is under high load, creating
> connections in parallel probably won't give you any time benefits. While
> in idle mode it may be true that:
> When I get 1 connection in 100 ms, I also get n (say 4) connections in
> ~100ms
> under high load situations it is much different as all processors on the
> db-server are busy with other jobs. So it will probably look much more
> like this:
> 1 connection in 2 seconds
> 2 connections in 4 seconds
> ...

I would be interested to see real data on this and also impacts of
connection request load spikes on various engines (i.e., how well they
handle bursts of "simultaneous" connection requests).  The engines are
going to end up queueing them anyway and it may be better to leave
that concern to them.
>
> So in this case serial creation might be even better from the
> application side of view, as you already have 1 connection after 2
> seconds (the 2nd after 4, and so on), while using parallel creation you
> would have to wait 4 seconds to get a connection.
>
>
> After all these are all considerations by me without any hard evidence
> supporting it, besides some observations I made in our live system
> during peak loads. So you can reject them, if you don't think that they
> would hold for the majority for the pool clients out there. But I still
> think that for our use case this would be the best way to proceed.
>
Thanks again for the feedback.   Any other opinions / suggestions on
this are appreciated.  I suggest the following compromise, which would
also fix the maxActive exceeded by one issue I discovered with
1.2/1.4-RC1 yesterday:

(*) Move makeObject back inside synchronized scope in borrowObject.
(patch below)

This addresses Christoph's production use case (assuming I have
understood it correctly) and also closes the maxActive exceeded by one
problem that my testing uncovered yesterday.  It does not have a huge
performance impact in the tests that I have run and still leaves
activation and validation (the per-request operations) outside of
synchronized scope.  More elegant solutions to both problems are
certainly possible and we can consider them in subsequent releases -
including configurable serialization of just the makes.

If there are no objections and my full suite of soak tests succeed and
do not show bad performance, I will proceed with this change and roll
RC2 this weekend.

Phil
-----------------------------------------------

Index: src/java/org/apache/commons/pool/impl/GenericObjectPool.java
===================================================================
--- src/java/org/apache/commons/pool/impl/GenericObjectPool.java	(revision
608225)
+++ src/java/org/apache/commons/pool/impl/GenericObjectPool.java	(working copy)
@@ -911,7 +911,7 @@
         long starttime = System.currentTimeMillis();
         for(;;) {
             ObjectTimestampPair pair = null;
-
+            boolean newlyCreated = false;
             synchronized (this) {
                 assertOpen();
                 // if there are any sleeping, just grab one of those
@@ -964,19 +964,16 @@
                     }
                 }
                 _numActive++;
-            }

-            // create new object when needed
-            boolean newlyCreated = false;
-            if(null == pair) {
-                try {
-                    Object obj = _factory.makeObject();
-                    pair = new ObjectTimestampPair(obj);
-                    newlyCreated = true;
-                } finally {
-                    if (!newlyCreated) {
-                        // object cannot be created
-                        synchronized (this) {
+                // create new object when needed
+                if(null == pair) {
+                    try {
+                        Object obj = _factory.makeObject();
+                        pair = new ObjectTimestampPair(obj);
+                        newlyCreated = true;
+                    } finally {
+                        if (!newlyCreated) {
+                            // object cannot be created
                             _numActive--;
                             notifyAll();
                         }

Phil.
>
> Christoph
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message