zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronen Itkin <ro...@taykey.com>
Subject Re: Client connections keep growing and growing
Date Sun, 16 Oct 2011 17:20:59 GMT
Actually I did noticed something interesting.
I have noticed that the majority of the client connection towards zookeeper
are being made from a Rest Service (rest api for hbase).
So I guess that i need to understand why Rest keep opening new sessions.
Any ideas??

And yes, I sent the same question to an HBase mailing list - no comments
yet.

thanks,


On Sun, Oct 16, 2011 at 5:58 PM, Camille Fournier <skamille@gmail.com>wrote:

> Seems like a hbase problem to me... Zk doesn't randomly create or hang on
> to
> client connections. Did you ask on the hbase lists?
>
> C
>
> On Sunday, October 16, 2011, Ronen Itkin <ronen@taykey.com> wrote:
> > Hey all!
> > How are you?
> >
> > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed
> > cluster with 3 nodes.
> > Though I have installed Zookeeper separately, its only purpose right now
> is
> > to serve HBase clients (hbase-0.90.3-cdh3u1).
> > Seems like everything is working great for a day or two and that out the
> > blue, Zookeeper refuses to serve clients requests,
> > alerting for the wide known issue of exceeding the defined maximum
> > connections allowed for a client.
> > I changed this value to 500 and it seems that it just held on for more
> time
> > and finally reached the threshold of 500 and then stopped working.
> > So, I set it to 1000 and right now I am waiting to see what will happen.
> >
> > Meanwhile, by auditing the Zookeeper connections manually it seems like
> they
> > keep growing in a slow - promising pace towards the threshold of 1000
> > connections.
> >
> > Can someone think of a reason why it happens? Why it keep growing? Does
> > someone has any other suggestions regarding this issue?
> > What is the recommended value for Zookeeper maximum connections per
> client?
> > can it overload the server if set to a high value? what is  a high value
> in
> > this case?
> >
> > If it helps, this is my environment design:
> >
> >   - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including
> Rest)*,
> >   *Zookeeper01*
> >   - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including
> >   Rest)*, *Zookeeper02*
> >   - Server 3 - *Zookeeper03*
> >   - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.*
> >
> > The type of operations I am committing on HBase using Zookeeper are:
> >
> >   - Simple read/writes/updates using SCAN / MapReduce / Cascading ..
> >   - Queries of the Rest service.
> >
> >
> > Thanks a lot !!!
> > *   *Ronen Itkin
> >
> > <http://www.taykey.com/>
> >
>



-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message