hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Quinn <squ...@moxiegroup.com>
Subject Re: HBaseAdmin Creating New Connections for Same HBase Configuration
Date Wed, 11 Jan 2012 20:29:56 GMT
Thanks for the quick responses!  HTablePool's look like what we'll probably
move to, although that approach seems inherently less efficient than the
old constructor which would only require one HConnection for the entire
application, instead of one connection for every table (potentially per
thread) in the client application.  The HBASE-4805 stuff looks promising
and potentially worth the extra effort for some of the higher volume
aspects of the application once we move to 0.92 or later.

Thanks again,

      -Shawn

On Wed, Jan 11, 2012 at 1:01 PM, Lars H <lhofhansl@yahoo.com> wrote:

> If you can use 0.92 or later  or backport a patch check out HBASE-4805.
> It gives you an option to manage your HConnection directly and create very
> lightweight HTable objects when you need them.
>
> -- Lars
>
> T Vinod Gupta <tvinod@readypulse.com> schrieb:
>
> >yeah your usage looks inefficient. what you need to do is keep a copy of
> >HTable object around and reuse it. but that is not thread safe. so as an
> >alternative, you need to create a HTablePool and get a HTable from it each
> >time you need to connect to hbase. after you are done, release the HTable
> >back to the pool. This is much more efficient.
> >
> >thanks
> >
> >On Wed, Jan 11, 2012 at 7:39 AM, Shawn Quinn <squinn@moxiegroup.com>
> wrote:
> >
> >> Hello,
> >>
> >> I'm a relatively new HBase user, and have recently upgraded to the
> CDH3u2
> >> (HBase 0.90.4-cdh3u2) version and after the upgrade I'm running into a
> new
> >> problem that appears to be a code problem in HBase - but wanted to post
> it
> >> here for validation.
> >>
> >> I'm running a client web application (within an OSGi container just for
> >> extra fun) that does a lot of queries against HBase using the Java
> client
> >> via HBaseAdmin to connect. My symptom is that after the upgrade I'm
> running
> >> into these errors consistently:
> >>
> >>   org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> >> connect to ZooKeeper but the connection closes immediately. This could
> be a
> >> sign that the server has too many connections (30 is the default).
> Consider
> >> inspecting your ZK server logs for that error and then make sure you are
> >> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> >> more information.
> >>
> >> And I confirmed in my zookeeper log that I am indeed running out of
> >> connections.
> >>
> >> Digging into my code I validated that I am reusing the same HBase
> >> Configuration instance (via a static singleton) everytime I construct a
> new
> >> HBaseAdmin instance, so that led me to dig into the HBase code a bit
> and I
> >> discovered the new HBaseAdmin constructor looks like so:
> >>
> >>  public HBaseAdmin(Configuration c)
> >>  throws MasterNotRunningException, ZooKeeperConnectionException {
> >>    this.conf = HBaseConfiguration.create(c);
> >>    this.connection = HConnectionManager.getConnection(this.conf);
> >>    this.pause = this.conf.getLong("hbase.client.pause", 1000);
> >>    this.numRetries = this.conf.getInt("hbase.client.retries.number",
> 10);
> >>    this.retryLongerMultiplier =
> >> this.conf.getInt("hbase.client.retries.longer.multiplier", 10);
> >>    this.connection.getMaster();
> >>  }
> >>
> >> Where as in prior versions it looked like so:
> >>
> >>  public HBaseAdmin(Configuration conf)
> >>  throws MasterNotRunningException, ZooKeeperConnectionException {
> >>    this.connection = HConnectionManager.getConnection(conf);
> >>    this.conf = conf;
> >>    this.pause = conf.getLong("hbase.client.pause", 1000);
> >>    this.numRetries = conf.getInt("hbase.client.retries.number", 10);
> >>    this.retryLongerMultiplier =
> >> conf.getInt("hbase.client.retries.longer.multiplier", 10);
> >>    this.connection.getMaster();
> >>  }
> >>
> >> So, I put a break point in the new version and confirmed that the call
> to
> >> "HBaseConfiguration.create(c)" is creating a new Configuration instance
> >> each time the constructor is invoked, which then leads the
> >> "HConnectionManager.getConnection(this.conf)" to create a new connection
> >> instance each time.  I'm guessing that's the reason why I'm running out
> of
> >> ZooKeeper connections, but it seems like others would have run into a
> >> similar problem so I'm doubting my conclusion a bit!  I did confirm
> that if
> >> I change the HBaseAdmin constructor code back to how it looks in
> previous
> >> versions that I no longer have the issue.
> >>
> >> Can anyone confirm or deny if this updated constructor is problematic?
> >>
> >> Thanks,
> >>
> >>      -Shawn
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message