hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From T Vinod Gupta <tvi...@readypulse.com>
Subject Re: HBaseAdmin Creating New Connections for Same HBase Configuration
Date Wed, 11 Jan 2012 17:04:09 GMT
yeah your usage looks inefficient. what you need to do is keep a copy of
HTable object around and reuse it. but that is not thread safe. so as an
alternative, you need to create a HTablePool and get a HTable from it each
time you need to connect to hbase. after you are done, release the HTable
back to the pool. This is much more efficient.

thanks

On Wed, Jan 11, 2012 at 7:39 AM, Shawn Quinn <squinn@moxiegroup.com> wrote:

> Hello,
>
> I'm a relatively new HBase user, and have recently upgraded to the CDH3u2
> (HBase 0.90.4-cdh3u2) version and after the upgrade I'm running into a new
> problem that appears to be a code problem in HBase - but wanted to post it
> here for validation.
>
> I'm running a client web application (within an OSGi container just for
> extra fun) that does a lot of queries against HBase using the Java client
> via HBaseAdmin to connect. My symptom is that after the upgrade I'm running
> into these errors consistently:
>
>   org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
>
> And I confirmed in my zookeeper log that I am indeed running out of
> connections.
>
> Digging into my code I validated that I am reusing the same HBase
> Configuration instance (via a static singleton) everytime I construct a new
> HBaseAdmin instance, so that led me to dig into the HBase code a bit and I
> discovered the new HBaseAdmin constructor looks like so:
>
>  public HBaseAdmin(Configuration c)
>  throws MasterNotRunningException, ZooKeeperConnectionException {
>    this.conf = HBaseConfiguration.create(c);
>    this.connection = HConnectionManager.getConnection(this.conf);
>    this.pause = this.conf.getLong("hbase.client.pause", 1000);
>    this.numRetries = this.conf.getInt("hbase.client.retries.number", 10);
>    this.retryLongerMultiplier =
> this.conf.getInt("hbase.client.retries.longer.multiplier", 10);
>    this.connection.getMaster();
>  }
>
> Where as in prior versions it looked like so:
>
>  public HBaseAdmin(Configuration conf)
>  throws MasterNotRunningException, ZooKeeperConnectionException {
>    this.connection = HConnectionManager.getConnection(conf);
>    this.conf = conf;
>    this.pause = conf.getLong("hbase.client.pause", 1000);
>    this.numRetries = conf.getInt("hbase.client.retries.number", 10);
>    this.retryLongerMultiplier =
> conf.getInt("hbase.client.retries.longer.multiplier", 10);
>    this.connection.getMaster();
>  }
>
> So, I put a break point in the new version and confirmed that the call to
> "HBaseConfiguration.create(c)" is creating a new Configuration instance
> each time the constructor is invoked, which then leads the
> "HConnectionManager.getConnection(this.conf)" to create a new connection
> instance each time.  I'm guessing that's the reason why I'm running out of
> ZooKeeper connections, but it seems like others would have run into a
> similar problem so I'm doubting my conclusion a bit!  I did confirm that if
> I change the HBaseAdmin constructor code back to how it looks in previous
> versions that I no longer have the issue.
>
> Can anyone confirm or deny if this updated constructor is problematic?
>
> Thanks,
>
>      -Shawn
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message