tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gnath <gautam_exquis...@yahoo.com>
Subject Re: Tomcat 6.0.35-SocketException: Too many open files issue with
Date Fri, 27 Jan 2012 05:31:21 GMT
Hello Chris, 


After seeing the initial connection pool issue, i started searching online for help and i
found this article :
http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html
so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar.
Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar.

Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded
the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it. 


I agree what you are saying about leaking the connection and will plan to set the logAbandoned
flag as you suggested. 


However, i was about to file a new issue but would like to describe here as well. So we have
2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar
and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree
that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't
been releasing the connections and was consistently growing slowly. So i collected thread
dump and i saw a deadlock :

Found one Java-level deadlock:
=============================
"catalina-exec-1":
  waiting to lock monitor 0x000000005d7944b8 (object 0x00000005bd522568, a com.mysql.jdbc.Connection),
  which is held by "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]"
"[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
  waiting to lock monitor 0x000000005dcdea28 (object 0x00000005bd659ce8, a com.mysql.jdbc.ResultSet),
  which is held by "catalina-exec-1"

Java stack information for the threads listed above:
===================================================
"catalina-exec-1":
        at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177)
        - waiting to lock <0x00000005bd522568> (a com.mysql.jdbc.Connection)
        at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583)
        at com.mysql.jdbc.Field.getName(Field.java:487)
        at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593)
        at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926)
        - locked <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
        at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401)

"[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
        at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
        - waiting to lock <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
        at com.mysql.jdbc.Statement.realClose(Statement.java:1606)
        - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
        - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
        at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703)
        at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901)
        - locked <0x00000005bd525ba0> (a java.lang.Object)
        - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
        - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
        at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126)
        at com.mysql.jdbc.Connection.realClose(Connection.java:4422)
        at com.mysql.jdbc.Connection.close(Connection.java:2098)
        - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
        at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320)



Please help us on this. Could it be a problem with tomcat-jdbc.jar?

Thanks
-G


________________________________
 From: Christopher Schultz <chris@christopherschultz.net>
To: Tomcat Users List <users@tomcat.apache.org> 
Sent: Thursday, January 26, 2012 9:41 AM
Subject: Re: Tomcat 6.0.35-SocketException: Too many open files  issue with
 
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

G,

On 1/25/12 11:53 PM, gnath wrote:
> As you have suggested, i started collecting the thread dumps

Thread dumps will set you free. Well, not really. Instead, they will
tell you where your webapp is breaking, which usually means more work
for you. But at least the hard part is done: finding out what's breaking.

> when it happened again and we saw some kind of DBCP Connection pool
> issues leading to 'Too Many open files' issue.

That will definitely do it.

> So we decided to replace the commons DBCP with tomcat-jdbc.jar
> (with same configuration properties).

Why?

> After this change, it seemed for few hours but started seeing in
> the logs where the Connection Pool jar could not give any 
> connections and seems to be all the connections are busy. So we
> went ahead and added a configuration property
> 'removeAbandoned=true' in our Datasource configuration.

I would go back to DBCP unless you think you need to switch for some
reason.

I suspect you are leaking database connections and don't have a
suitable timeout for removal of "lost" database connections (or maybe
didn't have that set up in the first place).

You really need to enable "logAbandoned" so you can find out where
your connection leaks are, and fix them. In development, set
maxActive="1" and leave it there, forever. Also, set
logAbandoned="true" and always run like that in development. Running
like that in production isn't a bad idea, either.

> We are still watching the performance and the server behavior
> after these changes. Will keep you posted on how things will turn
> out or if i see any further issues.

I suspect you are still leaking connections, but your pool is now
silently cleaning-up after the mess your webapp is making. Instrument
your pool. Fix your leaks.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8hkDkACgkQ9CaO5/Lv0PCxFgCgs+EiV/CNjmCNekeDwKHgnNtZ
5LYAoKZUkIAJOK0eItkoHBF3wScK9lQf
=AyL4
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message