hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronen Itkin <ro...@taykey.com>
Subject Re: HBase Rest & Zookeeper issue
Date Mon, 17 Oct 2011 09:57:31 GMT
Hey Andrew,

Thanks for the response,
Please note that the majority of zookeeper client connections are being made
from the Rest service itself and not from any other Java HBase Client,
Moreover, Rest Client address the Rest Service itself and not zookeepeer
(while The rest Service works with zookeeper-->hbase at the background).
So, I guess its not something in our code, but something regarding Rest
and/or ZooKeeper operations.

To be certain that the connections grow mainly because the Rest Service and
not because of other HBase Java Clients, I will shut it down for 5-6 Hours.

Can you please aleborate more about using a "shared single instance" ? is
that something configurable for the Rest implementation?
Is there any possibility that a Java HBase client is not using cache for
zookeeper queries?

Thanks,

On Mon, Oct 17, 2011 at 4:39 AM, Andrew Purtell <apurtell@apache.org> wrote:

> It may be sufficient for you to simply use the latest ASF release of HBase
> 0.90 instead of the CDH version. (Or use a snapshot from
> nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests a
> particular problem. I went through the REST code on the head of the 0.90
> branch and did not find it; the code is (re)using a shared
> Configuration instance.
>
> If you have written client HBase java code, you should also audit your code
> and insure you are NOT creating a new Configuration object instance whenever
> you create a new HTable object, instead share a single instance. Also,
> insure you are not creating HTable objects without passing in a shared
> Configuration.
>
>
> Another option is to patch the HBase client, to address the likely
> underlying problem.See HBASE-3777 / HBASE-4508.
>
>     https://issues.apache.org/jira/browse/HBASE-3777
>     https://issues.apache.org/jira/browse/HBASE-4508
>
> Best regards,
>
>
>        - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> ----- Original Message -----
> > From: Ronen Itkin <ronen@taykey.com>
> > To: user@hbase.apache.org
> > Cc:
> > Sent: Sunday, October 16, 2011 8:32 PM
> > Subject: HBase Rest & Zookeeper issue
> >
> > Hey all!
> >
> > At first it my seem like a Zookeeper issue, but the thing is that it is
> very
> > HBase related so I figured to send it to this mailing list as well.
> > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed
> > cluster with 3 nodes.
> > Though I have installed Zookeeper separately, its only purpose right now
> is
> > to serve HBase clients (hbase-0.90.3-cdh3u1).
> > Seems like everything is working great for a day or two and then out the
> > blue, Zookeeper refuses to serve clients requests,
> > alerting for the wide known issue of exceeding the defined maximum
> > connections allowed for a client.
> > I changed this value to 500 and it seems that it just held on for more
> time
> > and finally reached the threshold of 500 and then stopped working.
> > So, I set it to 1000 and right now I am waiting to see what will happen.
> >
> > Meanwhile, by auditing the Zookeeper connections manually it seems like
> they
> > keep growing in a slow - promising pace towards the threshold of 1000
> > connections.
> > I have tried to analyze the connections and it looks like almost all of
> the
> > connections were established from the HBase Rest service towards
> zookeeper.
> >
> > Can someone think of a reason why it keep growing? Does someone has any
> > other suggestions regarding this issue?
> > Are there any known issues regarding this situation, combining Rest and
> > Zookeeper?
> >
> > If it helps, this is my environment design:
> >
> >    - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including
> Rest)*,
> >    *Zookeeper01*
> >    - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including
> >    Rest)*, *Zookeeper02*
> >    - Server 3 - *Zookeeper03*
> >    - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.*
> >
> > The type of operations I am committing on HBase using Zookeeper are:
> >
> >    - Simple read/writes/updates using SCAN / MapReduce / Cascading ..
> >    - Queries of the *Rest* service.
> >
> >
> > Thanks a lot !!!
> > *   *Ronen Itkin
> >
>



-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message