Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Message-ID: <25845200.492201285909774130.JavaMail.jira@thor>
Date: Fri, 1 Oct 2010 01:09:34 -0400 (EDT)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Subject: [jira] Updated: (HBASE-3062) ZooKeeper
 KeeperException$ConnectionLossException is a "recoverable" exception; we
 should retry a while on server startup at least.
In-Reply-To: <30490777.491801285909285396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-3062:
-------------------------

    Attachment: 3063-v2.txt

This one will let let cluster start (No NPE when go to get port in zk).

> ZooKeeper KeeperException$ConnectionLossException is a "recoverable" exception; we should retry a while on server startup at least.
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-3062
>                 URL: https://issues.apache.org/jira/browse/HBASE-3062
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.90.0
>
>         Attachments: 3063-v2.txt, 3063.txt
>
>
> On startup of daemons we'll sometimes fail (new master) with something like the following:
> {code}
> 2010-09-30 23:38:36,979 INFO org.apache.hadoop.hbase.zookeeper.ZKUtil: regionserver60020 opening connection to ZooKeeper with quorum (sv2borg182:20001,sv2borg181:20001,sv2borg180:20001)
> 2010-09-30 23:38:36,979 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=sv2borg182:20001,sv2borg181:20001,sv2borg180:20001 sessionTimeout=60000 watcher=regionserver60020
> 2010-09-30 23:38:36,980 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server sv2borg182/10.20.20.182:20001
> 2010-09-30 23:38:36,980 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to sv2borg182/10.20.20.182:20001, initiating session
> 2010-09-30 23:38:36,981 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
> 2010-09-30 23:38:37,083 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: regionserver60020 Unexpected KeeperException creating base node
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>     at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
>     at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:807)
>     at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:107)
>     at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:438)
>     at org.apache.hadoop.hbase.regionserver.HRegionServer.initialize(HRegionServer.java:420)
>     at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:305)
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>     at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2436)
>     at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:60)
>     at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:75)
>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>     at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>     at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2460)
> {code}
> I'll see it over on master too the odd time.
> Currently we fail out (though in above case, because of order in which we do startup the regionserver actually hung up because RPC was started before zk)
> We should retry this error some at least on startup because its 'recoverable' (See http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling).  In fact, we should probably retry always until we get a session expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.