commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <>
Subject Re: [DBCP] Connection just obtained from datasource is invalid
Date Fri, 17 Nov 2017 19:22:16 GMT
On 11/16/2017 5:21 PM, Phil Steitz wrote:
>> How can a connection that I *just* retrieved from the pool be closed?
> Should not happen.  In most cases like this, it turns out not to be
> the pool that closed the connection.  The most common causes of this
> are network connectivity problems or the server closing the
> connection due to idle timeout.  The latter seems unlikely given
> that you have idle eviction configured.  Is it possible that some
> clients are holding connections for a very long time without using
> them?  The evictor only acts on connections that are idle in the
> pool (i.e. it does not do anything to connections that are checked
> out to clients).

Thank you for your reply.

My program does its work once a minute, and I had also configured the
idle eviction to run once a minute.

Just to see what it would do, I slightly changed the run interval for
the idle eviction -- for one of the object pools, adding 943
milliseconds, and 944 milliseconds for the other.  I've only seen the
problem happen once since making that change, where before it would
happen several times per day.  This isn't actual evidence, but it is

Although a network problem is always a possibility, it's not likely. 
It's a gigabit LAN with Cisco switches.  I will trace the network path
and check all the switch logs to see whether that might indeed be the

> That should not be possible, as once a connection has been checked
> out, it is not eligible for eviction.  And even if it were evicted
> by the pool, the pool would set the DelegatingConnection's closed
> property to true so isValid would return false rather than
> throwing.  It looks like the situation here is that the pool thinks
> the connection is open, but the underlying physical connection has
> been closed.

The code is within a try/catch block that traps all exceptions, so I am
100 percent certain that isValid is not throwing an exception, it's just
returning false.  When I first started investigating my strange alarms,
I couldn't tell what was wrong at all until I decided to add some code
to try a query on the connection that said it wasn't valid.   I fully
expected that query to fail, I was just doing it to try and discover why
isValid returned false.

This is a very recent problem, but the code hasn't had any changes since
well before the problem began.  That does lend credibility to the
possibility of a network problem, which as I said, I will investigate.

> It would be good to see your full pool config and a stack trace of
> the exception above.  You might also look at the server logs to see
> what is happening on the Mysql side.

Tell me exactly what I need to do in order to gather the pool config
you're after.  Are you looking for Java code, or some kind of
information from the objects that I need to gather?  Here's the code
that defines a datasource.  This is the section for the master server, I
have a virtually identical code section for the "main" server, which is
a replication slave on the MySQL side.

   * Create a datasource (connection pool) for the master database server.
  ConnectionFactory cfMaster = new
DriverManagerConnectionFactory(masterUrl, dbUser, dbPass);
  PoolableConnectionFactory pcfMaster = new
PoolableConnectionFactory(cfMaster, null);
  pcfMaster.setValidationQueryTimeout(Const.FIVE_SECONDS / 1000);
  opMaster = new GenericObjectPool<>(pcfMaster);
  opMaster.setMaxTotal(numShards * 5);
  opMaster.setNumTestsPerEvictionRun(numShards * 5);
  opMaster.setTimeBetweenEvictionRunsMillis(Const.ONE_MINUTE + 943);
  opMaster.setMinEvictableIdleTimeMillis(Const.ONE_MINUTE * 5);
  dsMaster = new PoolingDataSource<>(opMaster);

The numShards value is 6 on the server where I am doing the investigation.

The error log on all my MySQL servers isn't saying anything, but we
haven't changed the logging config, so I don't know if that's expected.

This is the stacktrace when I try a query on a connection that has just
failed the isValid check:

WARN  - 2017-11-16 13:10:20.229;   418; a; Problem with master DB server
noted.  Additional log may follow.
com.REDACTED.idxbuild.util.BuildException: Arbitrary DB query failed!
        at com.REDACTED.idxbuild.config.ConfigItem.load(
        at com.REDACTED.idxbuild.config.ConfigDb.load(
        at com.REDACTED.idxbuild.index.Chain$Flags.load(
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
operations allowed after connection closed.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
        at java.lang.reflect.Constructor.newInstance(
        at com.mysql.jdbc.Util.handleNewInstance(
        at com.mysql.jdbc.Util.getInstance(
        at com.mysql.jdbc.SQLError.createSQLException(
        at com.mysql.jdbc.SQLError.createSQLException(
        at com.mysql.jdbc.SQLError.createSQLException(
        at com.mysql.jdbc.SQLError.createSQLException(
        ... 5 more

While reviewing the specifics of that stacktrace, I discovered a tiny
bug in my code, where problems talking to the main DB server were being
incorrectly classified as problems with the master server.  I think I've
fixed that problem, but I will have to wait for it to happen again
before I can be sure.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message