hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Quinn <squ...@moxiegroup.com>
Subject HBaseAdmin Creating New Connections for Same HBase Configuration
Date Wed, 11 Jan 2012 15:39:16 GMT
Hello,

I'm a relatively new HBase user, and have recently upgraded to the CDH3u2
(HBase 0.90.4-cdh3u2) version and after the upgrade I'm running into a new
problem that appears to be a code problem in HBase - but wanted to post it
here for validation.

I'm running a client web application (within an OSGi container just for
extra fun) that does a lot of queries against HBase using the Java client
via HBaseAdmin to connect. My symptom is that after the upgrade I'm running
into these errors consistently:

   org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.

And I confirmed in my zookeeper log that I am indeed running out of
connections.

Digging into my code I validated that I am reusing the same HBase
Configuration instance (via a static singleton) everytime I construct a new
HBaseAdmin instance, so that led me to dig into the HBase code a bit and I
discovered the new HBaseAdmin constructor looks like so:

  public HBaseAdmin(Configuration c)
  throws MasterNotRunningException, ZooKeeperConnectionException {
    this.conf = HBaseConfiguration.create(c);
    this.connection = HConnectionManager.getConnection(this.conf);
    this.pause = this.conf.getLong("hbase.client.pause", 1000);
    this.numRetries = this.conf.getInt("hbase.client.retries.number", 10);
    this.retryLongerMultiplier =
this.conf.getInt("hbase.client.retries.longer.multiplier", 10);
    this.connection.getMaster();
  }

Where as in prior versions it looked like so:

  public HBaseAdmin(Configuration conf)
  throws MasterNotRunningException, ZooKeeperConnectionException {
    this.connection = HConnectionManager.getConnection(conf);
    this.conf = conf;
    this.pause = conf.getLong("hbase.client.pause", 1000);
    this.numRetries = conf.getInt("hbase.client.retries.number", 10);
    this.retryLongerMultiplier =
conf.getInt("hbase.client.retries.longer.multiplier", 10);
    this.connection.getMaster();
  }

So, I put a break point in the new version and confirmed that the call to
"HBaseConfiguration.create(c)" is creating a new Configuration instance
each time the constructor is invoked, which then leads the
"HConnectionManager.getConnection(this.conf)" to create a new connection
instance each time.  I'm guessing that's the reason why I'm running out of
ZooKeeper connections, but it seems like others would have run into a
similar problem so I'm doubting my conclusion a bit!  I did confirm that if
I change the HBaseAdmin constructor code back to how it looks in previous
versions that I no longer have the issue.

Can anyone confirm or deny if this updated constructor is problematic?

Thanks,

      -Shawn

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message