hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronen Itkin <ro...@taykey.com>
Subject HBase Rest & Zookeeper issue
Date Sun, 16 Oct 2011 12:32:00 GMT
Hey all!

At first it my seem like a Zookeeper issue, but the thing is that it is very
HBase related so I figured to send it to this mailing list as well.
I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed
cluster with 3 nodes.
Though I have installed Zookeeper separately, its only purpose right now is
to serve HBase clients (hbase-0.90.3-cdh3u1).
Seems like everything is working great for a day or two and then out the
blue, Zookeeper refuses to serve clients requests,
alerting for the wide known issue of exceeding the defined maximum
connections allowed for a client.
I changed this value to 500 and it seems that it just held on for more time
and finally reached the threshold of 500 and then stopped working.
So, I set it to 1000 and right now I am waiting to see what will happen.

Meanwhile, by auditing the Zookeeper connections manually it seems like they
keep growing in a slow - promising pace towards the threshold of 1000
connections.
I have tried to analyze the connections and it looks like almost all of the
connections were established from the HBase Rest service towards zookeeper.

Can someone think of a reason why it keep growing? Does someone has any
other suggestions regarding this issue?
Are there any known issues regarding this situation, combining Rest and
Zookeeper?

If it helps, this is my environment design:

   - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including Rest)*,
   *Zookeeper01*
   - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including
   Rest)*, *Zookeeper02*
   - Server 3 - *Zookeeper03*
   - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.*

The type of operations I am committing on HBase using Zookeeper are:

   - Simple read/writes/updates using SCAN / MapReduce / Cascading ..
   - Queries of the *Rest* service.


Thanks a lot !!!
*   *Ronen Itkin

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message