hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Zookeeper issue? (ConnectionLoss for /hbase/hbaseid)
Date Thu, 15 Nov 2012 23:41:36 GMT
HBase stop/start did not helped...
ZooKeeper stop/start did not helped..

I reduced the code to the minimum:
		config = HBaseConfiguration.create();
		config.set("hbase.zookeeper.quorum", "cube");
		new HBaseAdmin(config);

But still getting the exception.

On the same machine, bin/hbase zkcli is working fine giving me a
result to get /hbase.
bin/hbase shell is working fine too.

I'm a bit lost here. I tryied to google the issue with no much
results. I will continue to dig...

JM

2012/11/15, Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
> Hi,
>
> I ran a 30h MapReduce job, and now I'm not able to connect anymore to
> my HBase cluster.
>
> The MapReduce was configured on ReadOnly mode. So only the log table
> received data. Everything else was just ready.
>
> Today I killed the job to replace one of the servers which is too slow
> and now I'm not able to connect to ZooKeeper anymore.
>
> Below is the stack trace, and at the bottom is the ZKDump.
>
> I think if I restart everything it should be working, but I'm
> wondering if there is any information on this situation which might
> help to prevent this to happend in the futur? I might be able to
> reproduce that since I can re-run the job almost anytime.
>
> I don't seems to have to many connections. The HBase shell is replying
> correctly. It's really only the Java application using ZooKeeper which
> is not working.
>
> I'm running HBase 0.94.2, ZK 3.4.3, Hadoop 1.0.3 all installed separatly.
>
> JM
>
> 2012-11-15 18:09:33,684 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:33,810 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> 2012-11-15 18:09:34,800 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:35,902 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:36,003 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> 2012-11-15 18:09:37,004 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:38,106 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:39,209 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:40,311 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:40,412 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> 2012-11-15 18:09:41,414 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:42,516 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:43,618 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:44,720 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:45,822 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:46,924 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:48,027 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:49,129 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:49,230 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> 2012-11-15 18:09:49,231 [main] ERROR
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - ZooKeeper
> exists failed after 3 retries
> 2012-11-15 18:09:49,231 [main] WARN
> org.apache.hadoop.hbase.zookeeper.ZKUtil - hconnection 0x1781735b
> Unable to set watcher on znode (/hbase/hbaseid)
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> 	at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:262)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:63)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:645)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:620)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:215)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:143)
> 2012-11-15 18:09:49,232 [main] ERROR
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher - hconnection
> 0x1781735b Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> 	at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:262)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:63)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:645)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:620)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:215)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:143)
> 2012-11-15 18:09:49,233 [main] WARN
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> - Can't retrieve clusterId from Zookeeper
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> 	at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:262)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:63)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:645)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:620)
> 	at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:215)
> 	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:143)
> Creating table_page_crc. 2012-11-15 18:09:49,255
> [main-SendThread(cube:21818)] WARN  org.apache.zookeeper.ClientCnxn -
> Session 0x0 for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:49,357 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 2012-11-15 18:09:50,231 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:50,358 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:51,334 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:51,460 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:51,561 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 2012-11-15 18:09:52,436 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:52,562 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:53,538 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:53,664 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:54,640 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:54,766 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:55,742 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:55,868 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:55,969 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 2012-11-15 18:09:56,844 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:56,970 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:57,946 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:58,072 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:59,048 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:09:59,174 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:00,150 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:00,276 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:01,253 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:01,378 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:02,355 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:02,480 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:03,457 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:03,582 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:04,559 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:04,684 [main-SendThread(cube:21818)] WARN
> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connexion refuse
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
> 	at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-11-15 18:10:04,785 [main] WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 2012-11-15 18:10:04,785 [main] ERROR
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - ZooKeeper
> exists failed after 3 retries
> 2012-11-15 18:10:04,785 [main] WARN
> org.apache.hadoop.hbase.zookeeper.ZKUtil -
> catalogtracker-on-hconnection 0x1781735b Unable to set watcher on
> znode /hbase/root-region-server
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> 	at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85)
> 	at
> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241)
> 2012-11-15 18:10:04,786 [main] ERROR
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher -
> catalogtracker-on-hconnection 0x1781735b Received unexpected
> KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> 	at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85)
> 	at
> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241)
> 2012-11-15 18:10:04,787 [main] FATAL
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> - Unexpected exception during initialization, aborting
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> 	at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85)
> 	at
> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241)
> java.io.IOException: Attempt to start root/meta tracker failed.
> 	at
> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:247)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225)
> 	at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> 	at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
> 	at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85)
> 	at
> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242)
> 	... 5 more
>
>
>
>
>
> HBase is rooted at /hbase
> Active master address: node3,60000,1352914424646
> Backup master addresses:
> Region server holding ROOT: node5,60020,1352914415945
> Region servers:
>  latitude,60020,1352914441019
>  node6,60020,1352914439333
>  node5,60020,1352914415945
>  node2,60020,1353003249113
>  phenom,60020,1352914434321
>  node4,60020,1353009796200
>  node3,60020,1352914437336
>  node1,60020,1352914438812
> Quorum Server Statistics:
>  cube:2181
>   Zookeeper version: 3.4.3-1240972, built on 02/06/2012 10:48 GMT
>   Clients:
>    /192.168.23.7:32878[1](queued=0,recved=36,sent=36)
>    /192.168.23.5:49773[1](queued=0,recved=25,sent=25)
>    /192.168.23.7:32875[1](queued=0,recved=25,sent=25)
>    /192.168.23.5:49774[1](queued=0,recved=25,sent=25)
>    /192.168.23.8:49753[1](queued=0,recved=25,sent=25)
>    /192.168.23.10:55277[1](queued=0,recved=25,sent=25)
>    /192.168.23.7:32877[1](queued=0,recved=25,sent=25)
>    /192.168.23.6:44692[1](queued=0,recved=25,sent=25)
>    /192.168.23.4:52766[1](queued=0,recved=25,sent=25)
>    /192.168.23.7:32876[1](queued=0,recved=25,sent=25)
>    /192.168.23.2:35018[1](queued=0,recved=25,sent=25)
>    /192.168.23.8:49752[1](queued=0,recved=25,sent=25)
>    /192.168.23.7:32899[0](queued=0,recved=1,sent=0)
>    /192.168.23.7:32874[1](queued=0,recved=25,sent=25)
>    /192.168.23.10:55275[1](queued=0,recved=25,sent=25)
>    /192.168.23.9:37189[1](queued=0,recved=25,sent=25)
>    /192.168.23.2:35019[1](queued=0,recved=25,sent=25)
>    /192.168.23.9:37190[1](queued=0,recved=25,sent=25)
>    /192.168.23.4:52765[1](queued=0,recved=25,sent=25)
>    /192.168.23.6:44693[1](queued=0,recved=25,sent=25)
>
>   Latency min/avg/max: 0/0/18
>   Received: 487
>   Sent: 486
>   Outstanding: 0
>   Zxid: 0xc0002bbdf
>   Mode: standalone
>   Node count: 31
>

Mime
View raw message