Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 74.125.83.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=oeiU1U7sxLXffa78EI52uIJHQsmseX1dAgLINDBci05cPKt/QuhzP8qkj4JoLVq+6e
         lOJ7hJ9mX1YiMGTPofPf2RRjn3aUT6x65GaxXWZ2SCxp6NcFDElL9qWF9i1FLlYsxgqM
         x7LrjNH0393ETQHYel02Ss7aGTlsFl1bNYXb4=
MIME-Version: 1.0
In-Reply-To: <BANLkTik248VJvtCp69kqxwf3AkWv8c--uQ@mail.gmail.com>
References: <BANLkTi=xmhtseakmJyBuR0e-ycLaMhqaPA@mail.gmail.com>
	<352C8051-7AD3-400E-B9D8-F3CAD7FF6F95@gmail.com>
	<BANLkTik248VJvtCp69kqxwf3AkWv8c--uQ@mail.gmail.com>
Date: Sun, 15 May 2011 20:00:05 -0700
Message-ID: <BANLkTinU_YF=6=n8LvqQJXBYBUTUjTuOnw@mail.gmail.com>
Subject: Re: Zookeeper Configuration Challenges (I think)
From: Ted Yu <yuzhihong@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=bcaec544ed7237374004a35bda99

--bcaec544ed7237374004a35bda99
Content-Type: text/plain; charset=ISO-8859-1

If you don't let hbase manage zookeeper, yes.
Otherwise you need to set hbase.zookeeper.property.maxClientCnxns in
hbase-site.xml

Next hbase major release (with HBASE-3777) would behave much better.

On Sun, May 15, 2011 at 7:33 PM, Barney Frank <barneyfranks1@gmail.com>wrote:

> Will, do!
>
> Just in the zoo.cfg and not set it in the hbase-site.xml, correct?
>
> On Sun, May 15, 2011 at 9:20 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Please increase the max client connections, dramatically.
> >
> >
> >
> > On May 15, 2011, at 6:15 PM, Barney Frank <barneyfranks1@gmail.com>
> wrote:
> >
> > > I am looking for some advice on any changes to minimize these errors.
> > >
> > > Running Hbase standalone on version cdh3u0 and set Zoo.cfg to be 300
> max
> > > client connections. I use only the java api and use new Htable() for
> each
> > > request (no pooling).
> > >
> > > Running at lower volumes (50 requests/sec), I do not have any
> performance
> > > issues.  At high volumes of read/write requests (~200 requests/sec) via
> > the
> > > java API, I see the following exceptions in my client (JBOSS) logs:
> > >
> > > FYI, once the volumes decrease, everything seems to recover nicely.
> > >
> > > 2011-05-16 00:40:07,344 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (http-0.0.0.0-8080-43-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > Client
> > > session timed out, have not heard from server in 32852ms for sessionid
> > > 0x12fd6beb2180378, closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,344 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (http-0.0.0.0-8443-5-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > Client
> > > session timed out, have not heard from server in 32599ms for sessionid
> > > 0x12fd6beb2180379, closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,345 INFO  [org.apache.zookeeper.ClientCnxn]
> > (Interaction
> > > Logger Wrapup-SendThread(ip-10-46-181-169.ec2.internal:2181)) Client
> > session
> > > timed out, have not heard from server in 32849ms for sessionid
> > > 0x12fd6beb2180377, closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,345 INFO  [org.apache.zookeeper.ClientCnxn]
> (Contact
> > > History-SendThread(ip-10-46-181-169.ec2.internal:2181)) Client session
> > timed
> > > out, have not heard from server in 32850ms for sessionid
> > 0x12fd6beb2180376,
> > > closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,345 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181)) Client session
> > > timed out, have not heard from server in 32850ms for sessionid
> > > 0x12fd6beb2180371, closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,369 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181)) Client session
> > > timed out, have not heard from server in 42353ms for sessionid
> > > 0x12fd6beb2180372, closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,369 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (http-0.0.0.0-8080-1-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > Client
> > > session timed out, have not heard from server in 42353ms for sessionid
> > > 0x12fd6beb2180375, closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,370 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181)) Client session
> > > timed out, have not heard from server in 42386ms for sessionid
> > > 0x12fd6beb2180373, closing socket connection and attempting reconnect
> > > 2011-05-16 00:40:07,369 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181)) Client session
> > > timed out, have not heard from server in 42368ms for sessionid
> > > 0x12fd6beb2180374, closing socket connection and attempting reconnect
> > > 011-05-16 00:40:07,445 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > (http-0.0.0.0-8080-43-EventThread) hconnection-0x12fd6beb2180378
> Received
> > > ZooKeeper Event, type=None, state=Disconnected, path=null
> > > 2011-05-16 00:40:07,445 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > (http-0.0.0.0-8080-43-EventThread) hconnection-0x12fd6beb2180378
> Received
> > > Disconnected from ZooKeeper, ignoring
> > > 2011-05-16 00:40:07,445 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (Interaction
> Logger
> > > Wrapup-EventThread) hconnection-0x12fd6beb2180377 Received ZooKeeper
> > Event,
> > > type=None, state=Disconnected, path=null
> > > 2011-05-16 00:40:07,445 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (Interaction
> Logger
> > > Wrapup-EventThread) hconnection-0x12fd6beb2180377 Received Disconnected
> > from
> > > ZooKeeper, ignoring
> > > 2011-05-16 00:40:07,445 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (Contact
> > > History-EventThread) hconnection-0x12fd6beb2180376 Received ZooKeeper
> > Event,
> > > type=None, state=Disconnected, path=null
> > > 2011-05-16 00:40:07,445 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > (Timer-0-EventThread)
> > > hconnection-0x12fd6beb2180371 Received ZooKeeper Event, type=None,
> > > state=Disconnected, path=null
> > > 2011-05-16 00:40:07,446 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (Contact
> > > History-EventThread) hconnection-0x12fd6beb2180376 Received
> Disconnected
> > > from ZooKeeper, ignoring
> > > 2011-05-16 00:40:07,446 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > (Timer-0-EventThread)
> > > hconnection-0x12fd6beb2180371 Received Disconnected from ZooKeeper,
> > ignoring
> > > 2011-05-16 00:40:07,454 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > (http-0.0.0.0-8443-5-EventThread) hconnection-0x12fd6beb2180379
> Received
> > > ZooKeeper Event, type=None, state=Disconnected, path=null
> > > 2011-05-16 00:40:07,454 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > (http-0.0.0.0-8443-5-EventThread) hconnection-0x12fd6beb2180379
> Received
> > > Disconnected from ZooKeeper, ignoring
> > > 2011-05-16 00:40:07,447 ERROR
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > (http-0.0.0.0-8443-5)
> > > hconnection-0x12fd6beb2180379 Unexpected KeeperException creating base
> > node:
> > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > KeeperErrorCode = ConnectionLoss for /hbase/unassigned
> > >        at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > > [:3.3.3-cdh3u0--1]
> > >        at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > [:3.3.3-cdh3u0--1]
> > >        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> > > [:3.3.3-cdh3u0--1]
> > >        at
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
> > > [:]
> > >        at
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:148)
> > > [:]
> > > at
> > >
> >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
> > > [:3.3.3-cdh3u0--1]
> > >        at
> > > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
> > > [:3.3.3-cdh3u0--1]
> > > 2011-05-16 00:40:19,728 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (http-0.0.0.0-8443-5-EventThread) EventThread shut down
> > > 2011-05-16 00:40:19,729 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (http-0.0.0.0-8443-5-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > Unable
> > > to reconnect to ZooKeeper service, session 0x12fd6beb2180379 has
> expired,
> > > closing socket connection
> > > 2011-05-16 00:40:19,730 DEBUG
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > (Timer-0-EventThread)
> > > hconnection-0x12fd6beb2180374 Received ZooKeeper Event, type=None,
> > > state=Expired, path=null
> > > 2011-05-16 00:40:19,730 INFO
> > >
> >
> [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation]
> > > (Timer-0-EventThread) This client just lost it's session with
> ZooKeeper,
> > > trying to reconnect.
> > > 2011-05-16 00:40:19,730 INFO
> > >
> >
> [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation]
> > > (Timer-0-EventThread) Trying to reconnect to zookeeper
> > > 2011-05-16 00:40:19,731 DEBUG
> [org.apache.hadoop.hbase.zookeeper.ZKUtil]
> > > (Timer-0-EventThread) hconnection opening connection to ZooKeeper with
> > > ensemble (10.46.181.169:2181)
> > > 2011-05-16 00:40:19,731 INFO  [org.apache.zookeeper.ZooKeeper]
> > > (Timer-0-EventThread) Initiating client connection, connectString=
> > > 10.46.181.169:2181 sessionTimeout=180000 watcher=hconnection
> > > 2011-05-16 00:40:19,732 INFO  [org.apache.zookeeper.ClientCnxn]
> > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181)) Unable to
> > reconnect
> > > to ZooKeeper service, session 0x12fd6beb2180374 has expired, closing
> > socket
> > > connection
> > >
> > > *** I get a bunch of these ***
> > > 2011-05-16 00:40:19,847 WARN
>  [org.apache.hadoop.hbase.zookeeper.ZKUtil]
> > > (Interaction Logger Wrapup) hconnection-0x12fd6beb2180377 Unable to get
> > > children of node /hbase/rs
> > >
> > >
> > > *** Then a bunch of these ***
> > > 2011-05-16 00:40:19,881 ERROR
> > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (Interaction
> Logger
> > > Wrapup) hconnection-0x12fd6beb2180377 Received unexpected
> > KeeperException,
> > > re-throwing exception:
> > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/rs
> > >        at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
> > > [:3.3.3-cdh3u0--1]
> > >        at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > [:3.3.3-cdh3u0--1]
> > >        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> > > [:3.3.3-cdh3u0--1]
> > >        at
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getNumberOfChildren(ZKUtil.java:495)
> > > [:]
> > >        at
> > > org.apache.hadoop.hbase.client.HTable.getCurrentNrHRS(HTable.java:207)
> > [:]
> > >        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:182)
> > [:]
> > >        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> > [:]
> > >        at InteractionLogger.run(InteractionLogger.java:139) [:]
> > >        at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]
> > >
> > > ***Then a lot of these ***
> > > 2011-05-16 00:42:13,789 WARN  [InteractionLogger] (Interaction Logger
> > > Wrapup)  java.io.IOException: Unexpected ZooKeeper exception
> > >        at
> > > org.apache.hadoop.hbase.client.HTable.getCurrentNrHRS(HTable.java:210)
> > [:]
> > >        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:182)
> > [:]
> > >        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> > [:]
> > >        at
> > > stateful.session.InteractionLogger.run(InteractionLogger.java:139) [:]
> > >        at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]
> > > Caused by:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /hbase/rs
> > >        at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
> > > [:3.3.3-cdh3u0--1]
> > >        at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > [:3.3.3-cdh3u0--1]
> > >        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> > > [:3.3.3-cdh3u0--1]
> > >        at
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getNumberOfChildren(ZKUtil.java:495)
> > > [:]
> > >        at
> > > org.apache.hadoop.hbase.client.HTable.getCurrentNrHRS(HTable.java:207)
> > [:]
> > >        ... 4 more
> > >
> > > Any help would be greatly appreciated.
> >
>

--bcaec544ed7237374004a35bda99--