tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Carville <step...@totalflood.com>
Subject Re: Virtual Hosts with Apache and Tomcat
Date Mon, 01 Mar 2004 04:14:18 GMT
On Sunday February 29 2004 11:58 am, Christopher Schultz wrote:
> Stephen,
>
> > I am having a problem with tomcat opening up up a number of connections
> > to an oracle server that never get closed.  This causes the number of
> > open connections to build up over time and, eventually, causes the oracle
> > server to use all of its swap.
>
> That's not good :(

Tell me about it :-)

> > Restarting tomcat clers this up.
>
> That's good! :)

And bad.  Every time I restart, Tomcat loses the state information for 
established login sessions.  Customer don't like that.

> > I think there is a problem with some jsp's opening connections and then
> > not closig them but the developers claim (surprise) their code is clean.
>
> It's tough to make sure that database connections (and statements, and
> result sets) get cleaned up in JSPs, unless you have a talented JSP
> author. (Most JSP authors aren't that talented, unless they are also
> good Java developers, in which case they would have implemented the DB
> access in a servlet and just used the JSP for display. Anywho...)

I know they use ODBC for the database connections and there is a pool manager 
in there somewhere.  there is a jar file shared by all the jsp's that handles 
the connection pooling and a bunch of other stuff.  The pool manager is 
PoolmanBean.class.  I don't know enough about Java to say if that is a 
standard library or not.

I guess I don't know a lot about this case but I'm learning more :-)

Anyway, right after startup there are 10 connections.  If I open the main page 
and login, opens another connection closes.  Logging out adds another 
connection.  Both of these close but, apparently, none  of the tne original 
connectiions are not being used and, as time goes on, more connections get 
added to this anomolous pool.

I see someone just uploaded a new version of the jar file with the connection 
code in it so teh above may not be accurate....

> If the number of connections keeps going up and never tapers off or
> stops altogether, then something is misconfigured with your connection
> pools. Even if the engineers say that the pages are clean, you should
> protect the app server (and the DB server) from being swamped by capping
> the number of DB connections allowed. Ever. Any decent DB connection
> pool lets you specify this kind of thing. You should set that value to
> something reasonable. You can get away with a suprisingly low number of
> these.

Tried that.  Capped it at 35 and the webserver stopped servicing any DB 
request as soon as the pool reached 35.  This is why I believe the pool 
management is faulty and/or something is hogging all the connections.

> (I was consulting on a big project that was a somewhat DB intensive,
> web-based app. They had the app server configured to accept 75
> simultaneous connections. They also set the db connection pool size to
> 75. I asked why and they basically said "so that every HTTP connection
> can get a db connection". Duh. I talked to management and make them put
> in debugging information to find out how many connections were ever in
> use simultaneously. Seven. (Suckers). They also didn't realize that
> Oracle takes like 10MB per connection on the backend, and they had six
> physical app servers running two separate copies of the application.
> That's 75 * 6 * 2 * 10MB = 900MB. Good thing the DB server had 3.5GB of
> RAM, but still...)

Oracle 9i takes 16M per connections.  So Oracle claims.  I've tested it as 
high as 20M.  I generally use 18M as a guideline

> > The explanation they give is:
> >
> > "The increase in number of connections beyond the “CACHE_MAX_SIZE”
> > setting in the app1.properties file is due to the private labeled sites.
> > For each virtual host (private labeled site), there will be a separate
> > JVM running the Tomcat web server space. For each of these JVMs, there
> > will be a separate database connection cache pool to serve the user
> > requests. This is the designed functionality of a web server that will
> > support virtual hosts."
> >
> > I don't know tomcat near as well as I do Apache but this sounds like
> > someone is blowing smoke.
>
> This isn't too outrageous, actually. If each webapp has its own
> connection pool, and they are configured to have at maximum, say, 10
> connections, then you'll probably end up with 10 * webapp_count
> connections to the database server, regardless of the number of
> Tomcats/JVMs are running.
>
> If Tomcat is configured to handle the connection to the database (say,
> through a Realm and a JNDI-configured connection pool), you might be
> able to share connections between all of the webapps. If you solve the
> private-labelling problem by using multiple webapps, but through the
> same database, this approach seems like an excellent idea; configure
> Tomcat to provide a JNDI-based connection pool, and then configure the
> separate applications to use that pool. That way, you can control the
> total number of connections across all private labels, instead of having
> them be independent.
>
> > If I run ps on the server it looks to me like there is
> > only one instance and if I restart tomcat, _all_ virtual hosts are
> > restarted.
>
> Yeah, then it's definately separate webapps running on a single instance
> of Tomcat. Try to pitch the above idea to your engineers and see what
> they say (probably something like "it's fine the way it is!").

Basically yes.  That is what they are saying.  The proposal now is to write 
custom code to handle virtual domains.  AFAIK, there is nothing wrong with 
the way Apache and Tomcat do that already.  It is a waste of time and money 
to reinvent a wheel the Apache group has already done better.

> > I don't care who is right or wrong but I do want to clear up this
> > problem. Any ideas?  If you need any more information, just ask.
>
> I think I'd need to know if the connections were really never going
> away. Use netstat to find out what state they're in. If they all say
> ESTABLISHED, then you've got a connection leak.  If many of them say
> TIME_WAIT or something like that, then you might have a problem with
> either the client or the server not properly hanging up the phone. If
> it's the former, then yell at your engineers. 

It is definitely ESTABLISHED  I have a little script that calls netstat on the 
Oracle box (Solaris 8) greps for the port (1521) and pipes that thru some 
perl that prints a count of the connections from each client node that are 
ESTABLISHED versus TIME WAIT.  The particular web server that is causing the 
problem is almost always in the ESTABLISHED column and the connection that do 
get closed generally do not belong the the tomcat process.

I used another program to track the persistence of a socket by host:port.  I 
found that many of the opened a connection from Tomcat on the offending web 
server are not closed until Tomcat is restarted.

> Cap that connection pool
> size at something reasonable, like ten connections. After that, the
> application starves. That's good for the app server and the database,
> while bad for your application. You can use Jakarta Commons' DBCP as
> your connections pool. It has some wonderful debug options, like giving
> you a stack trace for the code that obtained the connection if that
> connection isn't returned within a certain amount of time. That can save
> days or weeks of code reviews. If your connections are in TIME_WAIT, see
> how long they stay that way. Waiting 5-10 minutes for a connection like
> that to get cleaned up is not unheard of. If they're piling up on top of
> one anothor and /never/ going away, it's time to talk to a system
> administrator. If the syadmin is you, it's time to talk to the guy you
> go to when you don't know things. Everyone needs a guy (or girl!) like
> that. :)

I'll mention DBCP and see what happens

> Good luck!
>
> Let me know if I can offer any more help.
>
> -chris

-- 
Stephen Carville
UNIX and Network Administrator
DPSI
310-342-3602
stephen@totalflood.com
--
Most people prefer believing their leaders are just and fair even in the face 
of contrary evidence.  Perhaps this is because, once a man acknowledges that  
the government he lives under is corrupt and cares nothing for justice or  
fairness, that man also has to choose what he will do about it.


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


Mime
View raw message