hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: HBase Rest & Zookeeper issue
Date Mon, 17 Oct 2011 02:39:44 GMT
It may be sufficient for you to simply use the latest ASF release of HBase 0.90 instead of
the CDH version. (Or use a snapshot from nightly.cloudera.com ?) The leaking of ZooKeeper
connections suggests a particular problem. I went through the REST code on the head of the
0.90 branch and did not find it; the code is (re)using a shared
Configuration instance.

If you have written client HBase java code, you should also audit your code and insure you
are NOT creating a new Configuration object instance whenever you create a new HTable object,
instead share a single instance. Also, insure you are not creating HTable objects without
passing in a shared Configuration.

Another option is to patch the HBase client, to address the likely underlying problem.See
HBASE-3777 / HBASE-4508. 


Best regards,

       - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

----- Original Message -----
> From: Ronen Itkin <ronen@taykey.com>
> To: user@hbase.apache.org
> Cc: 
> Sent: Sunday, October 16, 2011 8:32 PM
> Subject: HBase Rest & Zookeeper issue
> Hey all!
> At first it my seem like a Zookeeper issue, but the thing is that it is very
> HBase related so I figured to send it to this mailing list as well.
> I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed
> cluster with 3 nodes.
> Though I have installed Zookeeper separately, its only purpose right now is
> to serve HBase clients (hbase-0.90.3-cdh3u1).
> Seems like everything is working great for a day or two and then out the
> blue, Zookeeper refuses to serve clients requests,
> alerting for the wide known issue of exceeding the defined maximum
> connections allowed for a client.
> I changed this value to 500 and it seems that it just held on for more time
> and finally reached the threshold of 500 and then stopped working.
> So, I set it to 1000 and right now I am waiting to see what will happen.
> Meanwhile, by auditing the Zookeeper connections manually it seems like they
> keep growing in a slow - promising pace towards the threshold of 1000
> connections.
> I have tried to analyze the connections and it looks like almost all of the
> connections were established from the HBase Rest service towards zookeeper.
> Can someone think of a reason why it keep growing? Does someone has any
> other suggestions regarding this issue?
> Are there any known issues regarding this situation, combining Rest and
> Zookeeper?
> If it helps, this is my environment design:
>    - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including Rest)*,
>    *Zookeeper01*
>    - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including
>    Rest)*, *Zookeeper02*
>    - Server 3 - *Zookeeper03*
>    - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.*
> The type of operations I am committing on HBase using Zookeeper are:
>    - Simple read/writes/updates using SCAN / MapReduce / Cascading ..
>    - Queries of the *Rest* service.
> Thanks a lot !!!
> *   *Ronen Itkin

View raw message