tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phil Steitz <phil.ste...@gmail.com>
Subject Re: "Too Many Connections" exceptions after moving to Tomcat 8
Date Fri, 31 Jul 2015 00:05:15 GMT
On 7/30/15 9:03 AM, Christopher Schultz wrote:
> Jerry,
>
> On 7/29/15 3:25 PM, Jerry Malcolm wrote:
> > Well, it appears that we are slowly getting to the bottom of this.
> > But with every answer, I get a few more questions....
>
> > First, I installed the latest TC8 on my laptop, copied my
> > server.conf and conf/catalina folder to it and started it up just
> > to see what errors I got.  After changing out an obsolete listener,
> > it came up.  I found all of the <resource> parm exceptions in
> > stderr.  So that question is cleared up.  Thanks for clarifying
> > where to find that.
>
> If you have an obsolete Listener, you probably copied your server.xml
> from Tomcat 7 to Tomcat 8 which, while being less of a disaster than
> with previous version-pairs, is not good practice.
>
> Instead, start with the stock server.xml that comes with your Tomcat
> version and modify it to suit. These days, you should pretty much only
> have to configure the <Executor> and <Connector> elements, unless you
> have a particularly exotic <Host> configuration.
>
> > The site is a wedding vendor advertiser site that spans two major
> > cities.  There is no user login.  Simply a very huge online
> > catalog. I'm certain it's deployed only once.  Whether I need that
> > many connections is a valid question.  As far as I know, I haven't
> > hit the limit in normal operation until now.  Could possibly reduce
> > the count if I collect statistics.
>
> Our user load is roughly 250 concurrently logged-in users per Tomcat
> node, and we have maxTotal="20". I never get alarms about hitting that
> maximum. Your requirements may be different.
>
> > I've been monitoring the production server logs all day watching to
> > be sure connection pool doesn't dry up again.  About an hour ago,
> > there was a single huge dump in stdout of approx 2000
> > 'logAbandoned' exceptions. They showed connections from 1am right
> > after my last bounce of the server thru 1:35pm.
>
> It looks like your startup process (likely loading and caching stuff
> from the db on launch) is leaky. That can run-up your connection could
> quite quickly.
>
> > The good news is with the stack trace on one of them I was able to
> > see the bug causing the leak.
>
> Good.
> > But why did it decide to wait over 12 hours accumulating abandoned
> > connections before dumping them back in the pool?
>
> I was about to say the following, but markt says it might be a bug in
> DBCP.

The bug is in Commons Pool (POOL-300).  It is not flushing its
abandoned object log.  That means abandoned traces won't appear (in
the default System.out configuration) until some have been
accumulated. 
> I'll say it anyway:
>
> DBCP 2 looks like it only checks for abandoned connections "on borrow"
> so it might only log their abandonment when you see a flurry of
> connection-checkouts occurring, not when the connections are returned
> to the pool. DBCP 1 would complain pretty much immediately when the
> timeout was reached and the connection hadn't been returned.

When DBCP checks for abandoned connections depends in its
configuration properties.  There are two relevant properties:
removeAbandonedOnBorrow and removeAbandonedOnMaintenance.
>
> > I realize from now knowing the code bug that the leak is a slow
> > drip that is continually leaking on a regular basis. But since that
> > last 12-hour accumulated dump, the abandoning has returned to
> > silence. Since leaks are occurring regularly and would be timing
> > out regularly, shouldn't I see a similar 'slow drip' of
> > logAbandoned entries in stdout instead of a big dump every 12
> > hours?
>
> > It's going to take a day or two to fix the leak, test, and deploy.
>
> For testing, set maxTotal="1". You'll find your leaks *very quickly*
> that way, because everything will come grinding to a halt when you try
> to fetch that second connection from the pool.
>
> > If indeed abandoned connections are now correctly being returned to
> > the pool, then I presume we are back to working the way it did on
> > TC7. Still not sure why it started working now.  But I guess once I
> > get the leak fixed and if TC8 is now configured to handle abandoned
> > connections, I'm good.  Still would like to know about the
> > mega-dump vs. trickle of abandoned connections being logged.
>
> You should be able to run in testing with an upgraded DBCP 2. You
> might have to build it from trunk, though. I'm not sure if you are
> okay with that, but it might help you with your testing.

The thing to swap out is Commons Pool.  There is a release VOTE in
progress now for an RC including a fix for POOL-300.

A workaround that should work is to get a reference to the
BasicDataSource instance, say, bds and do

bds.setAbandonedLogWriter(new PrintWriter(System.out, true));

before using the pool.

Not sure if this will work correctly to get the output properly
directed under tomcat; but it is worth a try.

Phil
>
> Glad you are rooting-out some problems with your code.
>
> I highly recommend the use of findbugs:
> http://findbugs.sourceforge.net/
>
> It automatically detects stuff like this in your code, as well as a
> whole bunch of other things. It sometimes finds false positives, but
> you can knock-up a configuration file to exclude those things in very
> specific cases if need be.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message