hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: HBase Rest & Zookeeper issue
Date Mon, 17 Oct 2011 12:44:10 GMT
Pardon for any confusion (too late? :-) ) but this looks like a known HBase client bug.

HBASE-3777 (https://issues.apache.org/jira/browse/HBASE-3777) has background. 

Here is what I recommend:

  1) Use a build from the head of 0.90 branch, not a CDH build (I have no idea what revision
that corresponds to)*

and if that is not sufficient

  2) We will need to apply HBASE-3777 to 0.90 branch.

* - I reviewed the REST code on the 0.90 branch. It takes care to reuse a Configuration object
and a common HTable pool for all table access.

  - Andy


----- Original Message -----
> From: Ronen Itkin <ronen@taykey.com>
> To: user@hbase.apache.org; Andrew Purtell <apurtell@apache.org>
> Cc: 
> Sent: Monday, October 17, 2011 6:12 PM
> Subject: Re: HBase Rest & Zookeeper issue
> 
> Hey,
> 
> Another update, I had just stopped the Rest Service and the number of
> zookeeper client connections from that server dropped from 513 to 3.
> 
> 
> 
> On Mon, Oct 17, 2011 at 11:57 AM, Ronen Itkin <ronen@taykey.com> wrote:
> 
>>  Hey Andrew,
>> 
>>  Thanks for the response,
>>  Please note that the majority of zookeeper client connections are being
>>  made from the Rest service itself and not from any other Java HBase Client,
>>  Moreover, Rest Client address the Rest Service itself and not zookeepeer
>>  (while The rest Service works with zookeeper-->hbase at the background).
>>  So, I guess its not something in our code, but something regarding Rest
>>  and/or ZooKeeper operations.
>> 
>>  To be certain that the connections grow mainly because the Rest Service and
>>  not because of other HBase Java Clients, I will shut it down for 5-6 Hours.
>> 
>>  Can you please aleborate more about using a "shared single 
> instance" ? is
>>  that something configurable for the Rest implementation?
>>  Is there any possibility that a Java HBase client is not using cache for
>>  zookeeper queries?
>> 
>>  Thanks,
>> 
>> 
>>  On Mon, Oct 17, 2011 at 4:39 AM, Andrew Purtell 
> <apurtell@apache.org>wrote:
>> 
>>>  It may be sufficient for you to simply use the latest ASF release of 
> HBase
>>>  0.90 instead of the CDH version. (Or use a snapshot from
>>>  nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests a
>>>  particular problem. I went through the REST code on the head of the 
> 0.90
>>>  branch and did not find it; the code is (re)using a shared
>>>  Configuration instance.
>>> 
>>>  If you have written client HBase java code, you should also audit your
>>>  code and insure you are NOT creating a new Configuration object 
> instance
>>>  whenever you create a new HTable object, instead share a single 
> instance.
>>>  Also, insure you are not creating HTable objects without passing in a 
> shared
>>>  Configuration.
>>> 
>>> 
>>>  Another option is to patch the HBase client, to address the likely
>>>  underlying problem.See HBASE-3777 / HBASE-4508.
>>> 
>>>     https://issues.apache.org/jira/browse/HBASE-3777
>>>     https://issues.apache.org/jira/browse/HBASE-4508
>>> 
>>>  Best regards,
>>> 
>>> 
>>>         - Andy
>>> 
>>>  Problems worthy of attack prove their worth by hitting back. - Piet 
> Hein
>>>  (via Tom White)
>>> 
>>> 
>>>  ----- Original Message -----
>>>  > From: Ronen Itkin <ronen@taykey.com>
>>>  > To: user@hbase.apache.org
>>>  > Cc:
>>>  > Sent: Sunday, October 16, 2011 8:32 PM
>>>  > Subject: HBase Rest & Zookeeper issue
>>>  >
>>>  > Hey all!
>>>  >
>>>  > At first it my seem like a Zookeeper issue, but the thing is that 
> it is
>>>  very
>>>  > HBase related so I figured to send it to this mailing list as 
> well.
>>>  > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a 
> distributed
>>>  > cluster with 3 nodes.
>>>  > Though I have installed Zookeeper separately, its only purpose 
> right now
>>>  is
>>>  > to serve HBase clients (hbase-0.90.3-cdh3u1).
>>>  > Seems like everything is working great for a day or two and then 
> out the
>>>  > blue, Zookeeper refuses to serve clients requests,
>>>  > alerting for the wide known issue of exceeding the defined maximum
>>>  > connections allowed for a client.
>>>  > I changed this value to 500 and it seems that it just held on for 
> more
>>>  time
>>>  > and finally reached the threshold of 500 and then stopped working.
>>>  > So, I set it to 1000 and right now I am waiting to see what will 
> happen.
>>>  >
>>>  > Meanwhile, by auditing the Zookeeper connections manually it seems 
> like
>>>  they
>>>  > keep growing in a slow - promising pace towards the threshold of 
> 1000
>>>  > connections.
>>>  > I have tried to analyze the connections and it looks like almost 
> all of
>>>  the
>>>  > connections were established from the HBase Rest service towards
>>>  zookeeper.
>>>  >
>>>  > Can someone think of a reason why it keep growing? Does someone 
> has any
>>>  > other suggestions regarding this issue?
>>>  > Are there any known issues regarding this situation, combining 
> Rest and
>>>  > Zookeeper?
>>>  >
>>>  > If it helps, this is my environment design:
>>>  >
>>>  >    - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master 
> (Including
>>>  Rest)*,
>>>  >    *Zookeeper01*
>>>  >    - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master 
> (Including
>>>  >    Rest)*, *Zookeeper02*
>>>  >    - Server 3 - *Zookeeper03*
>>>  >    - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region 
> Server.*
>>>  >
>>>  > The type of operations I am committing on HBase using Zookeeper 
> are:
>>>  >
>>>  >    - Simple read/writes/updates using SCAN / MapReduce / Cascading 
> ..
>>>  >    - Queries of the *Rest* service.
>>>  >
>>>  >
>>>  > Thanks a lot !!!
>>>  > *   *Ronen Itkin
>>>  >
>>> 
>> 
>> 
>> 
>>  --
>>  *
>>  Ronen Itkin*
>>  Taykey | www.taykey.com
>> 
>> 
> 
> 
> -- 
> *
> Ronen Itkin*
> Taykey | www.taykey.com
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message