hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Zookeeper Configuration Challenges (I think)
Date Mon, 16 May 2011 16:09:56 GMT
Glad I can help.
Saving an project which uses HBase adds credit to the whole community.

On Mon, May 16, 2011 at 9:06 AM, Barney Frank <barneyfranks1@gmail.com>wrote:

> It was not set in hbase-env.sh.
>
> The errors now seem to be gone.
>
> Thanks for your prompt attention after my cry for help.
>
> On Mon, May 16, 2011 at 9:01 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > From hbase-default.xml:
> >
> > If HBASE_MANAGES_ZK is set in hbase-env.sh
> >    this is the list of servers which we will start/stop ZooKeeper on.
> >
> > Normally I would let client use the same hbase-site.xml as what server
> > uses.
> >
> > After increasing maxClientCnxns, do you observe the same problem ?
> >
> > Cheers
> >
> > On Mon, May 16, 2011 at 6:25 AM, Barney Frank <barneyfranks1@gmail.com
> > >wrote:
> >
> > > OK, I must be doing something wrong.  This will be the death of me if I
> > > don't pass my scalability testing on Wednesday for my project to get
> > > approved.
> > >
> > > Running on version 0.90.1-cdh3u0 using the pseudo-distributed mode
> > > for Hadoop and Hbase. ZK mode is standalone.
> > >
> > > How can I tell if Hbase is managing ZK?  I looked in the hbase-site.xml
> > for
> > > hbase server, distributed was set to true, xceivers set, and rootdir. I
> > > could add the hbase.zookeeper.property.maxClientCnxns here, correct?
> >  Would
> > > I need to set it on the client hbase-site.xml too?
> > >
> > > Otherwise I did set the maxClientCnxns within zoo.cfg to be very large.
> > >
> > > Do I need to restart any of the servers?  I have been restarting the
> > client
> > > and the hbase master and rs when changing their hbase-site.xml.
> > >
> > > FYI I am also doing a lot of (like 10/request)
> > mytable.incrementColumnValue
> > > and probably the same amount of puts.  Is there a way to do an
> > > incrementColmnValue using puts?  Maybe that would help my performance?
> > >
> > >
> > >
> > >
> > >
> > > On Sun, May 15, 2011 at 10:00 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > If you don't let hbase manage zookeeper, yes.
> > > > Otherwise you need to set hbase.zookeeper.property.maxClientCnxns in
> > > > hbase-site.xml
> > > >
> > > > Next hbase major release (with HBASE-3777) would behave much better.
> > > >
> > > > On Sun, May 15, 2011 at 7:33 PM, Barney Frank <
> barneyfranks1@gmail.com
> > > > >wrote:
> > > >
> > > > > Will, do!
> > > > >
> > > > > Just in the zoo.cfg and not set it in the hbase-site.xml, correct?
> > > > >
> > > > > On Sun, May 15, 2011 at 9:20 PM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > > >
> > > > > > Please increase the max client connections, dramatically.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On May 15, 2011, at 6:15 PM, Barney Frank <
> barneyfranks1@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > I am looking for some advice on any changes to minimize
these
> > > errors.
> > > > > > >
> > > > > > > Running Hbase standalone on version cdh3u0 and set Zoo.cfg
to
> be
> > > 300
> > > > > max
> > > > > > > client connections. I use only the java api and use new
> Htable()
> > > for
> > > > > each
> > > > > > > request (no pooling).
> > > > > > >
> > > > > > > Running at lower volumes (50 requests/sec), I do not have
any
> > > > > performance
> > > > > > > issues.  At high volumes of read/write requests (~200
> > requests/sec)
> > > > via
> > > > > > the
> > > > > > > java API, I see the following exceptions in my client (JBOSS)
> > logs:
> > > > > > >
> > > > > > > FYI, once the volumes decrease, everything seems to recover
> > nicely.
> > > > > > >
> > > > > > > 2011-05-16 00:40:07,344 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > >
> > > (http-0.0.0.0-8080-43-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > > > > > Client
> > > > > > > session timed out, have not heard from server in 32852ms
for
> > > > sessionid
> > > > > > > 0x12fd6beb2180378, closing socket connection and attempting
> > > reconnect
> > > > > > > 2011-05-16 00:40:07,344 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > >
> > > (http-0.0.0.0-8443-5-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > > > > > Client
> > > > > > > session timed out, have not heard from server in 32599ms
for
> > > > sessionid
> > > > > > > 0x12fd6beb2180379, closing socket connection and attempting
> > > reconnect
> > > > > > > 2011-05-16 00:40:07,345 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > (Interaction
> > > > > > > Logger Wrapup-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > > Client
> > > > > > session
> > > > > > > timed out, have not heard from server in 32849ms for sessionid
> > > > > > > 0x12fd6beb2180377, closing socket connection and attempting
> > > reconnect
> > > > > > > 2011-05-16 00:40:07,345 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > (Contact
> > > > > > > History-SendThread(ip-10-46-181-169.ec2.internal:2181))
Client
> > > > session
> > > > > > timed
> > > > > > > out, have not heard from server in 32850ms for sessionid
> > > > > > 0x12fd6beb2180376,
> > > > > > > closing socket connection and attempting reconnect
> > > > > > > 2011-05-16 00:40:07,345 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181))
Client
> > > > session
> > > > > > > timed out, have not heard from server in 32850ms for sessionid
> > > > > > > 0x12fd6beb2180371, closing socket connection and attempting
> > > reconnect
> > > > > > > 2011-05-16 00:40:07,369 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181))
Client
> > > > session
> > > > > > > timed out, have not heard from server in 42353ms for sessionid
> > > > > > > 0x12fd6beb2180372, closing socket connection and attempting
> > > reconnect
> > > > > > > 2011-05-16 00:40:07,369 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > >
> > > (http-0.0.0.0-8080-1-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > > > > > Client
> > > > > > > session timed out, have not heard from server in 42353ms
for
> > > > sessionid
> > > > > > > 0x12fd6beb2180375, closing socket connection and attempting
> > > reconnect
> > > > > > > 2011-05-16 00:40:07,370 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181))
Client
> > > > session
> > > > > > > timed out, have not heard from server in 42386ms for sessionid
> > > > > > > 0x12fd6beb2180373, closing socket connection and attempting
> > > reconnect
> > > > > > > 2011-05-16 00:40:07,369 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181))
Client
> > > > session
> > > > > > > timed out, have not heard from server in 42368ms for sessionid
> > > > > > > 0x12fd6beb2180374, closing socket connection and attempting
> > > reconnect
> > > > > > > 011-05-16 00:40:07,445 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > > (http-0.0.0.0-8080-43-EventThread)
> hconnection-0x12fd6beb2180378
> > > > > Received
> > > > > > > ZooKeeper Event, type=None, state=Disconnected, path=null
> > > > > > > 2011-05-16 00:40:07,445 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > > (http-0.0.0.0-8080-43-EventThread)
> hconnection-0x12fd6beb2180378
> > > > > Received
> > > > > > > Disconnected from ZooKeeper, ignoring
> > > > > > > 2011-05-16 00:40:07,445 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> (Interaction
> > > > > Logger
> > > > > > > Wrapup-EventThread) hconnection-0x12fd6beb2180377 Received
> > > ZooKeeper
> > > > > > Event,
> > > > > > > type=None, state=Disconnected, path=null
> > > > > > > 2011-05-16 00:40:07,445 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> (Interaction
> > > > > Logger
> > > > > > > Wrapup-EventThread) hconnection-0x12fd6beb2180377 Received
> > > > Disconnected
> > > > > > from
> > > > > > > ZooKeeper, ignoring
> > > > > > > 2011-05-16 00:40:07,445 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (Contact
> > > > > > > History-EventThread) hconnection-0x12fd6beb2180376 Received
> > > ZooKeeper
> > > > > > Event,
> > > > > > > type=None, state=Disconnected, path=null
> > > > > > > 2011-05-16 00:40:07,445 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > (Timer-0-EventThread)
> > > > > > > hconnection-0x12fd6beb2180371 Received ZooKeeper Event,
> > type=None,
> > > > > > > state=Disconnected, path=null
> > > > > > > 2011-05-16 00:40:07,446 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (Contact
> > > > > > > History-EventThread) hconnection-0x12fd6beb2180376 Received
> > > > > Disconnected
> > > > > > > from ZooKeeper, ignoring
> > > > > > > 2011-05-16 00:40:07,446 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > (Timer-0-EventThread)
> > > > > > > hconnection-0x12fd6beb2180371 Received Disconnected from
> > ZooKeeper,
> > > > > > ignoring
> > > > > > > 2011-05-16 00:40:07,454 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > > (http-0.0.0.0-8443-5-EventThread) hconnection-0x12fd6beb2180379
> > > > > Received
> > > > > > > ZooKeeper Event, type=None, state=Disconnected, path=null
> > > > > > > 2011-05-16 00:40:07,454 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > > (http-0.0.0.0-8443-5-EventThread) hconnection-0x12fd6beb2180379
> > > > > Received
> > > > > > > Disconnected from ZooKeeper, ignoring
> > > > > > > 2011-05-16 00:40:07,447 ERROR
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > (http-0.0.0.0-8443-5)
> > > > > > > hconnection-0x12fd6beb2180379 Unexpected KeeperException
> creating
> > > > base
> > > > > > node:
> > > > > > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > > > > > KeeperErrorCode = ConnectionLoss for /hbase/unassigned
> > > > > > >        at
> > > > > > >
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > > > > > >
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
> > > > > > > [:]
> > > > > > >        at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:148)
> > > > > > > [:]
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > > > > > >
> > > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > > 2011-05-16 00:40:19,728 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > > (http-0.0.0.0-8443-5-EventThread) EventThread shut down
> > > > > > > 2011-05-16 00:40:19,729 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > >
> > > (http-0.0.0.0-8443-5-SendThread(ip-10-46-181-169.ec2.internal:2181))
> > > > > > Unable
> > > > > > > to reconnect to ZooKeeper service, session 0x12fd6beb2180379
> has
> > > > > expired,
> > > > > > > closing socket connection
> > > > > > > 2011-05-16 00:40:19,730 DEBUG
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> > > > > > (Timer-0-EventThread)
> > > > > > > hconnection-0x12fd6beb2180374 Received ZooKeeper Event,
> > type=None,
> > > > > > > state=Expired, path=null
> > > > > > > 2011-05-16 00:40:19,730 INFO
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation]
> > > > > > > (Timer-0-EventThread) This client just lost it's session
with
> > > > > ZooKeeper,
> > > > > > > trying to reconnect.
> > > > > > > 2011-05-16 00:40:19,730 INFO
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation]
> > > > > > > (Timer-0-EventThread) Trying to reconnect to zookeeper
> > > > > > > 2011-05-16 00:40:19,731 DEBUG
> > > > > [org.apache.hadoop.hbase.zookeeper.ZKUtil]
> > > > > > > (Timer-0-EventThread) hconnection opening connection to
> ZooKeeper
> > > > with
> > > > > > > ensemble (10.46.181.169:2181)
> > > > > > > 2011-05-16 00:40:19,731 INFO  [org.apache.zookeeper.ZooKeeper]
> > > > > > > (Timer-0-EventThread) Initiating client connection,
> > connectString=
> > > > > > > 10.46.181.169:2181 sessionTimeout=180000 watcher=hconnection
> > > > > > > 2011-05-16 00:40:19,732 INFO  [org.apache.zookeeper.ClientCnxn]
> > > > > > > (Timer-0-SendThread(ip-10-46-181-169.ec2.internal:2181))
Unable
> > to
> > > > > > reconnect
> > > > > > > to ZooKeeper service, session 0x12fd6beb2180374 has expired,
> > > closing
> > > > > > socket
> > > > > > > connection
> > > > > > >
> > > > > > > *** I get a bunch of these ***
> > > > > > > 2011-05-16 00:40:19,847 WARN
> > > > >  [org.apache.hadoop.hbase.zookeeper.ZKUtil]
> > > > > > > (Interaction Logger Wrapup) hconnection-0x12fd6beb2180377
> Unable
> > to
> > > > get
> > > > > > > children of node /hbase/rs
> > > > > > >
> > > > > > >
> > > > > > > *** Then a bunch of these ***
> > > > > > > 2011-05-16 00:40:19,881 ERROR
> > > > > > > [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher]
> (Interaction
> > > > > Logger
> > > > > > > Wrapup) hconnection-0x12fd6beb2180377 Received unexpected
> > > > > > KeeperException,
> > > > > > > re-throwing exception:
> > > > > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > > > > KeeperErrorCode = Session expired for /hbase/rs
> > > > > > >        at
> > > > > > >
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > > > > > >
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getNumberOfChildren(ZKUtil.java:495)
> > > > > > > [:]
> > > > > > >        at
> > > > > > >
> > > >
> org.apache.hadoop.hbase.client.HTable.getCurrentNrHRS(HTable.java:207)
> > > > > > [:]
> > > > > > >        at
> > > > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:182)
> > > > > > [:]
> > > > > > >        at
> > > > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> > > > > > [:]
> > > > > > >        at InteractionLogger.run(InteractionLogger.java:139)
[:]
> > > > > > >        at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]
> > > > > > >
> > > > > > > ***Then a lot of these ***
> > > > > > > 2011-05-16 00:42:13,789 WARN  [InteractionLogger] (Interaction
> > > Logger
> > > > > > > Wrapup)  java.io.IOException: Unexpected ZooKeeper exception
> > > > > > >        at
> > > > > > >
> > > >
> org.apache.hadoop.hbase.client.HTable.getCurrentNrHRS(HTable.java:210)
> > > > > > [:]
> > > > > > >        at
> > > > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:182)
> > > > > > [:]
> > > > > > >        at
> > > > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> > > > > > [:]
> > > > > > >        at
> > > > > > >
> > stateful.session.InteractionLogger.run(InteractionLogger.java:139)
> > > > [:]
> > > > > > >        at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]
> > > > > > > Caused by:
> > > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > > > > KeeperErrorCode = Session expired for /hbase/rs
> > > > > > >        at
> > > > > > >
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > > > > > >
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> > > > > > > [:3.3.3-cdh3u0--1]
> > > > > > >        at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getNumberOfChildren(ZKUtil.java:495)
> > > > > > > [:]
> > > > > > >        at
> > > > > > >
> > > >
> org.apache.hadoop.hbase.client.HTable.getCurrentNrHRS(HTable.java:207)
> > > > > > [:]
> > > > > > >        ... 4 more
> > > > > > >
> > > > > > > Any help would be greatly appreciated.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message