hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jayavelu jaisenthilkumar <joysent...@gmail.com>
Subject Re: Failed to create /hbase.... KeeperErrorCode = ConnectionLoss for /hbase
Date Sat, 03 Apr 2010 13:58:40 GMT
Hi Guys,
               Thankyou so much for everybody.  I actually removed all
the dfs files in the temp directory,  formatted everything using
namenode -format. and started dfs and hbase now both are perfectly
running. i can able to create table now, and put some file into it.

@Patrick:  I dont know the Zookeeper version, i  keep HBase to run the
zookeeper by setting the value

export HBASE_MANAGES_ZK=true

i gave 3 servern domain names as values to hbase.zookeeper.quorum

Regs,
senthil


On 2 April 2010 21:41, Patrick Hunt <phunt@apache.org> wrote:
> Thanks Mahadev.
>
> Jaya, One other thing, you might check that the disk is not reaching
> capacity, ZK needs to create snapshot and txn log files, one thing that
> could have caused this is if we were unable to pre allocate the txn log
> fully.
>
> Regards,
>
> Patrick
>
> On 04/02/2010 11:40 AM, Mahadev Konar wrote:
>>
>> HI Jaya,
>>   This is a bug in zookeeper and was fixed in 3.3 release. Here is the
>> jira
>> for that:
>>
>> http://issues.apache.org/jira/browse/ZOOKEEPER-596
>>
>>
>> The workarnd for this would be to just shutdown the zookeeper node that is
>> reporting this error. The other 2 zookeeper nodes will still be working
>> and
>> then you can bring up the shot down zookeeper server node after a while
>> and
>> it should heal itself.
>>
>>
>> Let me know if you see any problems.
>>
>> Thanks
>> mahadev
>>
>>
>> On 4/2/10 10:52 AM, "jayavelu jaisenthilkumar"<joysenthil@gmail.com>
>>  wrote:
>>
>>> Hi Patrick,
>>>                     I grep for Too * in the logs directory.  It didnt
>>> yield any results.
>>>
>>> hadoop@Hadoopserver:~/hbase-0.20.3/logs>grep Too *
>>> hadoop@Hadoopserver:~/hbase-0.20.3/logs>
>>>
>>> also i saw HQorum running in the all nodes. which can be checked with the
>>> ./jps
>>>
>>> eg.
>>> hadoop@Hadoopclient bin]$ ./jps
>>> 3081 HRegionServer
>>> 2990 HQuorumPeer
>>> 3532 Jps
>>>
>>> Log file in Zookeeper says some thing like
>>> 2010-04-02 17:51:21,071 INFO
>>> org.apache.zookeeper.server.ZooKeeperServer: Created server
>>> 2010-04-02 17:51:21,071 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Sending new
>>> notification.
>>> 2010-04-02 17:51:21,071 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Sending new
>>> notification.
>>> 2010-04-02 17:51:21,075 INFO
>>> org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
>>> /tmp/hbase-hadoop/zookeeper/version-2/snapshot.2000000000
>>> 2010-04-02 17:51:21,077 WARN
>>> org.apache.zookeeper.server.quorum.QuorumPeer: Unexpected exception
>>> java.io.IOException: Invalid magic number 0 != 1514884167
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCre
>>> ated(FileTxnLog.java:450)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInput
>>> Archive(FileTxnLog.java:466)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog
>>> (FileTxnLog.java:433)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTx
>>> nLog.java:415)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(File
>>> TxnLog.java:394)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:265)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.
>>> java:126)
>>>         at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:203)
>>>         at
>>> org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:260)
>>>         at
>>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:538)
>>> 2010-04-02 17:51:21,077 INFO
>>> org.apache.zookeeper.server.quorum.Leader: Shutdown called
>>> java.lang.Exception: shutdown Leader! reason: Forcing shutdown
>>>         at
>>> org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:371)
>>>         at
>>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:544)
>>> 2010-04-02 17:51:21,080 INFO
>>> org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
>>> 2010-04-02 17:51:21,082 WARN
>>> org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of
>>> session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>> 2010-04-02 17:51:21,082 INFO
>>> org.apache.zookeeper.server.NIOServerCnxn: closing session:0x0
>>> NIOServerCnxn: java.nio.channels.SocketChannel[connected
>>> local=/192.168.1.1:2222 remote=/192.168.1.2:37151]
>>> 2010-04-02 17:51:21,085 WARN
>>> org.apache.zookeeper.server.persistence.FileTxnLog: Unexpected
>>> exception
>>> java.io.IOException: Invalid magic number 0 != 1514884167
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCre
>>> ated(FileTxnLog.java:450)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInput
>>> Archive(FileTxnLog.java:466)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog
>>> (FileTxnLog.java:433)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTx
>>> nLog.java:415)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(File
>>> TxnLog.java:394)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:265)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog.getLastLoggedZxid(FileTxnLo
>>> g.java:226)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.getLastLoggedZxid(FileT
>>> xnSnapLog.java:194)
>>>         at
>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeer.getLastLoggedZxid(QuorumPeer.jav
>>> a:394)
>>>         at
>>>
>>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeader
>>> Election.java:576)
>>>         at
>>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:515)
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: New election:
>>> 137438953472
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Notification:
>>> 1, 133143986180, 1905, 0, LOOKING, LOOKING, 1
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Adding vote
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Notification:
>>> 2, 133143986180, 1905, 0, LOOKING, LOOKING, 2
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Adding vote
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Notification:
>>> 2, 133143986180, 1905, 0, LOOKING, LOOKING, 1
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Adding vote
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Notification:
>>> 0, 137438953472, 1905, 0, LOOKING, LOOKING, 0
>>> 2010-04-02 17:51:21,088 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Adding vote
>>> 2010-04-02 17:51:21,089 INFO
>>> org.apache.zookeeper.server.quorum.QuorumPeer: LEADING
>>> 2010-04-02 17:51:21,089 INFO
>>> org.apache.zookeeper.server.ZooKeeperServer: Created server
>>> 2010-04-02 17:51:21,089 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Sending new
>>> notification.
>>> 2010-04-02 17:51:21,089 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Sending new
>>> notification.
>>> 2010-04-02 17:51:21,091 INFO
>>> org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
>>> /tmp/hbase-hadoop/zookeeper/version-2/snapshot.2000000000
>>> 2010-04-02 17:51:21,092 WARN
>>> org.apache.zookeeper.server.quorum.QuorumPeer: Unexpected exception
>>> java.io.IOException: Invalid magic number 0 != 1514884167
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCre
>>> ated(FileTxnLog.java:450)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInput
>>> Archive(FileTxnLog.java:466)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog
>>> (FileTxnLog.java:433)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTx
>>> nLog.java:415)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(File
>>> TxnLog.java:394)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:265)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.
>>> java:126)
>>>         at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:203)
>>>         at
>>> org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:260)
>>>         at
>>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:538)
>>> 2010-04-02 17:51:21,092 INFO
>>> org.apache.zookeeper.server.quorum.Leader: Shutdown called
>>> java.lang.Exception: shutdown Leader! reason: Forcing shutdown
>>>         at
>>> org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:371)
>>>         at
>>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:544)
>>> 2010-04-02 17:51:21,095 INFO
>>> org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
>>> 2010-04-02 17:51:21,097 WARN
>>> org.apache.zookeeper.server.persistence.FileTxnLog: Unexpected
>>> exception
>>> java.io.IOException: Invalid magic number 0 != 1514884167
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCre
>>> ated(FileTxnLog.java:450)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInput
>>> Archive(FileTxnLog.java:466)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog
>>> (FileTxnLog.java:433)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTx
>>> nLog.java:415)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(File
>>> TxnLog.java:394)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:265)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnLog.getLastLoggedZxid(FileTxnLo
>>> g.java:226)
>>>         at
>>>
>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.getLastLoggedZxid(FileT
>>> xnSnapLog.java:194)
>>>         at
>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeer.getLastLoggedZxid(QuorumPeer.jav
>>> a:394)
>>>         at
>>>
>>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeader
>>> Election.java:576)
>>>         at
>>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:515)
>>> 2010-04-02 17:51:21,100 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: New election:
>>> 137438953472
>>> 2010-04-02 17:51:21,101 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Notification:
>>> 1, 133143986180, 1906, 0, LOOKING, LOOKING, 1
>>> 2010-04-02 17:51:21,101 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Adding vote
>>> 2010-04-02 17:51:21,101 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Notification:
>>> 0, 137438953472, 1906, 0, LOOKING, LOOKING, 0
>>> 2010-04-02 17:51:21,101 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Adding vote
>>> 2010-04-02 17:51:21,101 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Notification:
>>> 0, 137438953472, 1906, 0, LOOKING, LOOKING, 1
>>> 2010-04-02 17:51:21,101 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: Adding vote
>>> 2010-04-02 17:51:21,303 INFO
>>> org.apache.zookeeper.server.quorum.FastLeaderElection: About to leave
>>> instance:0, 137438953472, 0, LEADING
>>> 2010-04-02 17:51:21,303 INFO
>>> org.apache.zookeeper.server.quorum.QuorumPeer: LEADING
>>> 2010-04-02 17:51:21,303 INFO
>>> org.apache.zookeeper.server.ZooKeeperServer: Created server
>>> 2010-04-02 17:51:21,307 INFO
>>> org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
>>> /tmp/hbase-hadoop/zookeeper/version-2/snapshot.2000000000
>>>
>>>
>>>
>>> Regs,
>>> Senthil
>>>
>>>
>>>
>>> On 2 April 2010 17:28, Patrick Hunt<phunt@apache.org>  wrote:
>>>>
>>>> I would suggest taking a look at my prior email again:
>>>>
>>>>> The client makes a connection to the server (tcp) but cannot
>>>>> establish a session. This could be because you don't have a
>>>>> functioning quorum, or perhaps maxClientCnxns is exceeded (does the
>>>>> log mention "Too many connections from" anywhere?), etc... the server
>>>>> logs will shed more light.
>>>>
>>>> So far you've only included the client logs, as I mentioned it's clear
>>>> from
>>>> the client log that the server is rejecting the session from the client.
>>>> Looking at the ZK servers logs for the time period in question will shed
>>>> more light.
>>>>
>>>> Patrick
>>>>
>>>> On 04/02/2010 05:06 AM, jayavelu jaisenthilkumar wrote:
>>>>>
>>>>> HI Ted Yu, I cleaned the stuff you mentioned.  Still the problem
>>>>> exists. Cant able to stop the Master of hbase. Manually need to kill
>>>>> the Master. :-( while stopping using the ./stophbase
>>>>>
>>>>> I see the Master log files like 2010-04-02 13:45:04,949 INFO
>>>>> org.apache.zookeeper.ClientCnxn: Attempting connection to server
>>>>> Hadoopserver/192.168.1.65:2222 2010-04-02 13:45:04,949 INFO
>>>>> org.apache.zookeeper.ClientCnxn: Priming connection to
>>>>> java.nio.channels.SocketChannel[connected local=/ 192.168.1.65:37478
>>>>> remote=Hadoopserver/192.168.1.65:2222] 2010-04-02 13:45:04,949 INFO
>>>>> org.apache.zookeeper.ClientCnxn: Server connection successful
>>>>> 2010-04-02 13:45:04,950 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@17cff66
>>>>> java.io.IOException: Read error rc = -1
>>>>> java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:04,951 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:04,951 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:05,051 WARN
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>>>>> /hbase -- check quorum servers,
>>>>> currently=Hadoopclient1:2222,Hadoopclient:2222,Hadoopserver:2222
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608) at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWra
>>>>> pper.java:405)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKee
>>>>> perWrapper.java:428)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.writeMasterAddress(ZooKee
>>>>> perWrapper.java:516)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:
>>>>> 263)
>>>>>
>>>>>
>>>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:245)
>>>>>
>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method) at
>>>>>
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
>>>>> sorImpl.java:39)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
>>>>> torAccessorImpl.java:27)
>>>>>
>>>>>
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>
>>>>> at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1241)
>>>>> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
>>>>> 2010-04-02 13:45:05,560 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Attempting connection to server Hadoopclient1/192.168.1.2:2222
>>>>> 2010-04-02 13:45:05,560 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Priming connection to java.nio.channels.SocketChannel[connected
>>>>> local=/ 192.168.1.65:49673 remote=Hadoopclient1/192.168.1.2:2222]
>>>>> 2010-04-02 13:45:05,560 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>> connection successful 2010-04-02 13:45:05,561 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@163a320 java.io.IOException: Read error
>>>>> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:05,561 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:05,561 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:05,662 DEBUG
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master 2010-04-02
>>>>> 13:45:06,467 INFO org.apache.zookeeper.ClientCnxn: Attempting
>>>>> connection to server Hadoopclient/192.168.1.3:2222 2010-04-02
>>>>> 13:45:06,468 INFO org.apache.zookeeper.ClientCnxn: Priming connection
>>>>> to java.nio.channels.SocketChannel[connected local=/
>>>>> 192.168.1.65:52198 remote=Hadoopclient/192.168.1.3:2222] 2010-04-02
>>>>> 13:45:06,468 INFO org.apache.zookeeper.ClientCnxn: Server connection
>>>>> successful 2010-04-02 13:45:06,469 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@6ac615 java.io.IOException: Read error rc
>>>>> = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:06,469 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:06,469 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:06,570 WARN
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>>>>> /hbase -- check quorum servers,
>>>>> currently=Hadoopclient1:2222,Hadoopclient:2222,Hadoopserver:2222
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608) at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWra
>>>>> pper.java:405)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKee
>>>>> perWrapper.java:428)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.writeMasterAddress(ZooKee
>>>>> perWrapper.java:516)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:
>>>>> 263)
>>>>>
>>>>>
>>>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:245)
>>>>>
>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method) at
>>>>>
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
>>>>> sorImpl.java:39)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
>>>>> torAccessorImpl.java:27)
>>>>>
>>>>>
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>
>>>>> at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1241)
>>>>> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
>>>>> 2010-04-02 13:45:07,154 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Attempting connection to server Hadoopserver/192.168.1.65:2222
>>>>> 2010-04-02 13:45:07,155 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Priming connection to java.nio.channels.SocketChannel[connected
>>>>> local=/ 192.168.1.65:37484 remote=Hadoopserver/192.168.1.65:2222]
>>>>> 2010-04-02 13:45:07,155 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>> connection successful 2010-04-02 13:45:07,156 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@1f2be27 java.io.IOException: Read error
>>>>> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:07,156 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:07,156 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:07,257 DEBUG
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master 2010-04-02
>>>>> 13:45:07,573 INFO org.apache.zookeeper.ClientCnxn: Attempting
>>>>> connection to server Hadoopclient1/192.168.1.2:2222 2010-04-02
>>>>> 13:45:07,573 INFO org.apache.zookeeper.ClientCnxn: Priming connection
>>>>> to java.nio.channels.SocketChannel[connected local=/
>>>>> 192.168.1.65:49680 remote=Hadoopclient1/192.168.1.2:2222] 2010-04-02
>>>>> 13:45:07,573 INFO org.apache.zookeeper.ClientCnxn: Server connection
>>>>> successful 2010-04-02 13:45:07,574 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@1a1f370 java.io.IOException: Read error
>>>>> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:07,574 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:07,574 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:07,675 WARN
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>>>>> /hbase -- check quorum servers,
>>>>> currently=Hadoopclient1:2222,Hadoopclient:2222,Hadoopserver:2222
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608) at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWra
>>>>> pper.java:405)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKee
>>>>> perWrapper.java:428)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.writeMasterAddress(ZooKee
>>>>> perWrapper.java:516)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:
>>>>> 263)
>>>>>
>>>>>
>>>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:245)
>>>>>
>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method) at
>>>>>
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
>>>>> sorImpl.java:39)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
>>>>> torAccessorImpl.java:27)
>>>>>
>>>>>
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>
>>>>> at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1241)
>>>>> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
>>>>> 2010-04-02 13:45:07,822 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Attempting connection to server Hadoopclient/192.168.1.3:2222
>>>>> 2010-04-02 13:45:07,823 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Priming connection to java.nio.channels.SocketChannel[connected
>>>>> local=/ 192.168.1.65:52203 remote=Hadoopclient/192.168.1.3:2222]
>>>>> 2010-04-02 13:45:07,823 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>> connection successful 2010-04-02 13:45:07,824 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@1757337 java.io.IOException: Read error
>>>>> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:07,824 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:07,825 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:07,925 DEBUG
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master 2010-04-02
>>>>> 13:45:08,622 INFO org.apache.zookeeper.ClientCnxn: Attempting
>>>>> connection to server Hadoopserver/192.168.1.65:2222 2010-04-02
>>>>> 13:45:08,623 INFO org.apache.zookeeper.ClientCnxn: Priming connection
>>>>> to java.nio.channels.SocketChannel[connected local=/
>>>>> 192.168.1.65:37489 remote=Hadoopserver/192.168.1.65:2222] 2010-04-02
>>>>> 13:45:08,623 INFO org.apache.zookeeper.ClientCnxn: Server connection
>>>>> successful 2010-04-02 13:45:08,624 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@1df3d59 java.io.IOException: Read error
>>>>> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:08,624 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:08,624 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:08,725 WARN
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>>>>> /hbase -- check quorum servers,
>>>>> currently=Hadoopclient1:2222,Hadoopclient:2222,Hadoopserver:2222
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608) at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWra
>>>>> pper.java:405)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKee
>>>>> perWrapper.java:428)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.writeMasterAddress(ZooKee
>>>>> perWrapper.java:516)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:
>>>>> 263)
>>>>>
>>>>>
>>>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:245)
>>>>>
>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method) at
>>>>>
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
>>>>> sorImpl.java:39)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
>>>>> torAccessorImpl.java:27)
>>>>>
>>>>>
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>
>>>>> at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1241)
>>>>> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
>>>>> 2010-04-02 13:45:09,402 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Attempting connection to server Hadoopclient1/192.168.1.2:2222
>>>>> 2010-04-02 13:45:09,403 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Priming connection to java.nio.channels.SocketChannel[connected
>>>>> local=/ 192.168.1.65:49685 remote=Hadoopclient1/192.168.1.2:2222]
>>>>> 2010-04-02 13:45:09,403 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>> connection successful 2010-04-02 13:45:09,403 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@134c5ff java.io.IOException: Read error
>>>>> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:09,404 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:09,404 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:09,504 DEBUG
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master 2010-04-02
>>>>> 13:45:09,573 INFO org.apache.zookeeper.ClientCnxn: Attempting
>>>>> connection to server Hadoopclient/192.168.1.3:2222 2010-04-02
>>>>> 13:45:09,576 INFO org.apache.zookeeper.ClientCnxn: Priming connection
>>>>> to java.nio.channels.SocketChannel[connected local=/
>>>>> 192.168.1.65:52209 remote=Hadoopclient/192.168.1.3:2222] 2010-04-02
>>>>> 13:45:09,576 INFO org.apache.zookeeper.ClientCnxn: Server connection
>>>>> successful 2010-04-02 13:45:09,577 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@1b3278a java.io.IOException: Read error
>>>>> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:09,577 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:09,577 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:09,678 WARN
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>>>>> /hbase -- check quorum servers,
>>>>> currently=Hadoopclient1:2222,Hadoopclient:2222,Hadoopserver:2222
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608) at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWra
>>>>> pper.java:405)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKee
>>>>> perWrapper.java:428)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.writeMasterAddress(ZooKee
>>>>> perWrapper.java:516)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:
>>>>> 263)
>>>>>
>>>>>
>>>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:245)
>>>>>
>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method) at
>>>>>
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
>>>>> sorImpl.java:39)
>>>>>
>>>>>
>>>> at
>>>>>
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
>>>>> torAccessorImpl.java:27)
>>>>>
>>>>>
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>
>>>>> at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1241)
>>>>> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
>>>>> 2010-04-02 13:45:09,809 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Attempting connection to server Hadoopserver/192.168.1.65:2222
>>>>> 2010-04-02 13:45:09,809 INFO org.apache.zookeeper.ClientCnxn:
>>>>> Priming connection to java.nio.channels.SocketChannel[connected
>>>>> local=/ 192.168.1.65:37494 remote=Hadoopserver/192.168.1.65:2222]
>>>>> 2010-04-02 13:45:09,809 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>> connection successful 2010-04-02 13:45:09,810 WARN
>>>>> org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to
>>>>> sun.nio.ch.SelectionKeyImpl@e8f8c5 java.io.IOException: Read error rc
>>>>> = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>> 2010-04-02 13:45:09,810 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown input java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:09,811 WARN org.apache.zookeeper.ClientCnxn:
>>>>> Ignoring exception during shutdown output java.net.SocketException:
>>>>> Transport endpoint is not connected at
>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>
>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>
>>>>>
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>
>>>>> 2010-04-02 13:45:09,911 DEBUG
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>>>
>>>>>
>>>>> Any one have any idea of this strange behaviour
>>>>>
>>>>>
>>>>> Regs. Senthil On 1 April 2010 21:55, Ted Yu<yuzhihong@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Please check the following entry in hbase-env.sh:
>>>>>>
>>>>>> hbase-env.sh:# The directory where pid files are stored. /tmp by
>>>>>> default. hbase-env.sh:# export HBASE_PID_DIR=/var/hadoop/pids
>>>>>>
>>>>>> If pid file is stored under /tmp, it might have been cleaned up.
>>>>>>
>>>>>> On Thu, Apr 1, 2010 at 11:44 AM, Jean-Daniel
>>>>>> Cryans<jdcryans@apache.org
>>>>>>>
>>>>>>> wrote:
>>>>>>
>>>>>>> If the master doesn't shut down, it means it's waiting on
>>>>>>> something... you looked at the logs?
>>>>>>>
>>>>>>> You say you ran ./jps ... did you install that in the local
>>>>>>> directory? Also what do you mean "it didn't work as well"? What
>>>>>>> didn't work? The command didn't return anything or the HMaster
>>>>>>> process wasn't listed?
>>>>>>>
>>>>>>> Also did you check the zookeeper logs like Patrick said? You
>>>>>>> should see in there when the master tries to connect, and you
>>>>>>> should see why it wasn't able to do so.
>>>>>>>
>>>>>>> To help you I need more data about your problem.
>>>>>>>
>>>>>>> J-D
>>>>>>>
>>>>>>> On Thu, Apr 1, 2010 at 11:39 AM, jayavelu jaisenthilkumar
>>>>>>> <joysenthil@gmail.com>    wrote:
>>>>>>>>
>>>>>>>> Hi Daniel, I removed the property tags from the
>>>>>>>> hbase-site.xml.
>>>>>>>>
>>>>>>>> Same error occurs.
>>>>>>>>
>>>>>>>> Also one strange behaviour,  If i give ./stop-hbase.sh , the
>>>>>>>> terminal
>>>>>>>
>>>>>>> says
>>>>>>>>
>>>>>>>> stopping master .................................... and never
>>>>>>>> stopped.
>>>>>>>>
>>>>>>>> I couldnt able to ./jps to check the java in this scenario, it
>>>>>>>> didnt
>>>>>>
>>>>>> work
>>>>>>>>
>>>>>>>> aswell.  So I killed the Hmaster start ( ps -ef | grep java)
>>>>>>>>
>>>>>>>> Also manually need to kill Hregionserver both on master, slave1
>>>>>>>> and
>>>>>>>
>>>>>>> slave2.
>>>>>>>>
>>>>>>>> Any suggestions please.......
>>>>>>>>
>>>>>>>> Regs, senthil On 31 March 2010 19:15, Jean-Daniel
>>>>>>>> Cryans<jdcryans@apache.org>
>>>>>>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> You set the tick time like this:
>>>>>>>>>
>>>>>>>>> <property>  <name>hbase.zookeeper.property.tickTime</name>
>>>>>>>>> <value>1</value>  <description>Property from ZooKeeper's
>>>>>>>>> config zoo.cfg. The number of milliseconds of each tick.
>>>>>>>>> See zookeeper.session.timeout description.</description>
>>>>>>>>>
>>>>>>>>> 1 means HBase has to report to zookeeper every 1 millisecond
>>>>>>>>> and if for any reason it doesn't after 20ms, the session is
>>>>>>>>> expired (!!). I recommend using the default value.
>>>>>>>>>
>>>>>>>>> Also you should keep the same config on every node, rsync can
>>>>>>>>> do
>>>>>>>
>>>>>>> wonders.
>>>>>>>>>
>>>>>>>>> J-D
>>>>>>>>>
>>>>>>>>> On Wed, Mar 31, 2010 at 9:24 AM, jayavelu jaisenthilkumar
>>>>>>>>> <joysenthil@gmail.com>    wrote:
>>>>>>>>>>
>>>>>>>>>> Hi, I am using 1 master and 2 slaves one has password for
>>>>>>
>>>>>> ssh.
>>>>>>>>>>
>>>>>>>>>> I am using hadoop0.20.1 and hbase0.20.3(direct one not
>>>>>>>>>> upgraded one)
>>>>>>>>>>
>>>>>>>>>> 1)The slave one with password is could not be disabled, i
>>>>>>>>>> removed
>>>>>>
>>>>>> the
>>>>>>>>>
>>>>>>>>> whole
>>>>>>>>>>
>>>>>>>>>> .ssh directory try to ssh-keygen with passwordless phrase,
>>>>>>>>>> still i
>>>>>>
>>>>>> am
>>>>>>>>>
>>>>>>>>> asked
>>>>>>>>>>
>>>>>>>>>> for the password  when i "ssh localhost"
>>>>>>>>>>
>>>>>>>>>> 2) I am able to run hadoop and successfuly run the
>>>>>>>>>> Mapreduce in the
>>>>>>>>>
>>>>>>>>> hadoop
>>>>>>>>>>
>>>>>>>>>> environment as per the Running Hadoop On Ubuntu Linux
>>>>>>>>>> (Multi-Node
>>>>>>>>>
>>>>>>>>> Cluster)
>>>>>>>>>>
>>>>>>>>>> by noel
>>>>>>>>>>
>>>>>>>>>> 3) I am now following the tutorial hbase: overview HBase
>>>>>>>>>> 0.20.3 API
>>>>>>>>>>
>>>>>>>>>> Its not clearly stated as the mulitnode cluster hadoop for
>>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> distributed
>>>>>>>>>>
>>>>>>>>>> mode hbase.
>>>>>>>>>>
>>>>>>>>>> I ran the hdfs and the hbase using start-dfs.sh and
>>>>>>>>>> start-hbase.sh respectively.
>>>>>>>>>>
>>>>>>>>>> The master log indicates connection loss on the /hbase :  (
>>>>>>>>>> is this
>>>>>>>
>>>>>>> hbase
>>>>>>>>>
>>>>>>>>> is
>>>>>>>>>>
>>>>>>>>>> created by Hbase or should we do to create it again
>>>>>>>>>>
>>>>>>>>>> 2010-03-31 16:45:57,850 INFO org.apache.zookeeper.
>>>>>>>>>> ClientCnxn: Attempting connection to server Hadoopserver/
>>>>>>>>>
>>>>>>>>> 192.168.1.65:2222
>>>>>>>>>>
>>>>>>>>>> 2010-03-31 16:45:57,858 INFO
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Priming
>>>>>>>>>>
>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>>>>>>>>> local=/ 192.168.1.65:43017
>>>>>>>>>> remote=Hadoopserver/192.168.1.65:2222] 2010-03-31
>>>>>>>>>> 16:45:57,881 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>>>>>>> connection successful 2010-03-31 16:45:57,883 WARN
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Exception
>>>>>>>>>>
>>>>>>>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@11c2b67
>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>
>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>
>>>>>>>>>> lim=4 cap=4] at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>
>>>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:57,885 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Ignoring
>>>>>>>>>>
>>>>>>>>>> exception during shutdown input java.net.SocketException:
>>>>>>>>>> Transport endpoint is not connected at
>>>>>>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>>>>>>>
>>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>>>>>>
>>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:57,885 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Ignoring
>>>>>>>>>>
>>>>>>>>>> exception during shutdown output java.net.SocketException:
>>>>>>>>>> Transport endpoint is not connected at
>>>>>>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>>>>>>>
>>>>>>>
>>>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>
>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:57,933 INFO
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>>>>>>>
>>>>>>>>>> -ROOT- region unset (but not set to be reassigned)
>>>>>>>>>> 2010-03-31 16:45:57,934 INFO
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>>>>>>>
>>>>>>>>>> ROOT inserted into regionsInTransition 2010-03-31
>>>>>>>>>> 16:45:58,024 DEBUG
>>>>>>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed
>>>>>>>>>> to read:
>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>>>>>>
>>>>>>>>>>
>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>>>>>>>>
>>>>>>>>>> 2010-03-31 16:45:58,422 INFO
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Attempting
>>>>>>>>>>
>>>>>>>>>> connection to server Hadoopclient1/192.168.1.2:2222
>>>>>>>>>> 2010-03-31 16:45:58,423 INFO
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Priming
>>>>>>>>>>
>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>>>>>>>>> local=/ 192.168.1.65:51219
>>>>>>>>>> remote=Hadoopclient1/192.168.1.2:2222] 2010-03-31
>>>>>>>>>> 16:45:58,423 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>>>>>>> connection successful 2010-03-31 16:45:58,436 WARN
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Exception
>>>>>>>>>>
>>>>>>>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@17b6643
>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>
>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>
>>>>>>>>>> lim=4 cap=4] at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>
>>>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:58,437 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Ignoring
>>>>>>>>>>
>>>>>>>>>> exception during shutdown input java.net.SocketException:
>>>>>>>>>> Transport endpoint is not connected at
>>>>>>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>>>>>>>
>>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>>>>>>
>>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:58,437 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Ignoring
>>>>>>>>>>
>>>>>>>>>> exception during shutdown output java.net.SocketException:
>>>>>>>>>> Transport endpoint is not connected at
>>>>>>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>>>>>>>
>>>>>>>
>>>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>
>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:58,537 WARN
>>>>>>>>>>
>>>>>>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed
>>>>>>>>>> to create
>>>>>>>>>
>>>>>>>>> /hbase
>>>>>>>>>>
>>>>>>>>>> -- check quorum servers,
>>>>>>>>>> currently=Hadoopclient1:2222,Hadoopclient:2222,Hadoopserver:2222
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>>>>>>
>>>>>>>>>> KeeperErrorCode = ConnectionLoss for /hbase at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>>>>>>>
>>>>>>>>>
>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>>>>>>>
>>>>>>>>>
>>>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWr
>>>>>> apper.java:405)
>>>>>>>>>>
>>>>>>
>>>> at
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKe
>>>>>> eperWrapper.java:428)
>>>>>>>>>>
>>>>>>
>>>> at
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.writeMasterAddress(ZooKe
>>>>>> eperWrapper.java:516)
>>>>>>>>>>
>>>>>>
>>>> at
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java
>>>>>> :263)
>>>>>>>>>>
>>>>>>
>>>> at
>>>>>>
>>>>>> org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:245)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>>>>>>
>>>>>>>>>>
>>>> Method)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce
>>>>>> ssorImpl.java:39)
>>>>>>>>>>
>>>>>>
>>>> at
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru
>>>>>> ctorAccessorImpl.java:27)
>>>>>>>>>>
>>>>>>
>>>> at
>>>>>>
>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>
>>>>>> org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1241)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
>>>>>>>>>>
>>>>>>>>>>
>>>> 2010-03-31 16:45:58,549 INFO org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Attempting
>>>>>>>>>>
>>>>>>>>>> connection to server Hadoopclient/192.168.1.3:2222
>>>>>>>>>> 2010-03-31 16:45:58,550 INFO
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Priming
>>>>>>>>>>
>>>>>>>>>> connection to java.nio.channels.SocketChannel[connected
>>>>>>>>>> local=/ 192.168.1.65:56142
>>>>>>>>>> remote=Hadoopclient/192.168.1.3:2222] 2010-03-31
>>>>>>>>>> 16:45:58,550 INFO org.apache.zookeeper.ClientCnxn: Server
>>>>>>>>>> connection successful 2010-03-31 16:45:58,577 WARN
>>>>>>>>>> org.apache.zookeeper.ClientCnxn:
>>>>>>>
>>>>>>> Exception
>>>>>>>>>>
>>>>>>>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@76e8a7
>>>>>>>>>> java.io.IOException: Read error rc = -1
>>>>>>>
>>>>>>> java.nio.DirectByteBuffer[pos=0
>>>>>>>>>>
>>>>>>>>>> lim=4 cap=4] at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
>>>>>>>>>>
>>>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:58,577 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Ignoring
>>>>>>>>>>
>>>>>>>>>> exception during shutdown input java.net.SocketException:
>>>>>>>>>> Transport endpoint is not connected at
>>>>>>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>>>>>>>
>>>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>>>>>>>>>
>>>>>>
>>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:58,577 WARN org.apache.zookeeper.ClientCnxn:
>>>>>>
>>>>>> Ignoring
>>>>>>>>>>
>>>>>>>>>> exception during shutdown output java.net.SocketException:
>>>>>>>>>> Transport endpoint is not connected at
>>>>>>>>>> sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>>>>>>>>>
>>>>>>>
>>>>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>
>>>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>>>>>>>>>>
>>>>>>>
>>>> at
>>>>>>>
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>>>>>>>>>>
>>>>>>>
>>>> 2010-03-31 16:45:58,678 DEBUG
>>>>>>>>>>
>>>>>>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed
>>>>>>>>>> to read:
>>>>>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>>>>>>
>>>>>>>>>>
>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>>>>>>>>
>>>>>>>>>> my hbase-site.xml ( Master)<configuration>  <property>
>>>>>>>>>> <name>hbase.rootdir</name>
>>>>>>>>>> <value>hdfs://Hadoopserver:54310/hbase</value>
>>>>>>>>>> <description>The directory shared by region servers. Should
>>>>>>>>>> be fully-qualified to include the filesystem to use. E.g:
>>>>>>>>>> hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description>
>>>>>>>>>> </property>  <property>
>>>>>>>>>> <name>hbase.cluster.distributed</name>  <value>true</value>
>>>>>>>>>> <description>The mode the cluster will be in. Possible
>>>>>>>>>> values are false: standalone and pseudo-distributed setups
>>>>>>>>>> with managed
>>>>>>>>>
>>>>>>>>> Zookeeper
>>>>>>>>>>
>>>>>>>>>> true: fully-distributed with unmanaged Zookeeper Quorum
>>>>>>>>>> (see hbase-env.sh)</description>  </property>  <property>
>>>>>>>>>> <name>hbase.zookeeper.quorum</name>
>>>>>>>>>> <value>Hadoopserver,Hadoopclient1,Hadoopclient</value>
>>>>>>>>>> <description>Comma separated list of servers in the
>>>>>>>>>> ZooKeeper
>>>>>>>
>>>>>>> Quorum.
>>>>>>>>>>
>>>>>>>>>> For example, "host1.mydomain.com,host2.mydomain.com,
>>>>>>>>>
>>>>>>>>> host3.mydomain.com".
>>>>>>>>>>
>>>>>>>>>> By default this is set to localhost for local and
>>>>>>>
>>>>>>> pseudo-distributed
>>>>>>>>>>
>>>>>>>>>> modes of operation. For a fully-distributed setup, this
>>>>>>>>>> should be set
>>>>>>
>>>>>> to
>>>>>>>
>>>>>>> a
>>>>>>>>>>
>>>>>>>>>> full list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK
>>>>>>>>>> is set in hbase-env.sh this is the list of servers which we
>>>>>>>>>> will start/stop ZooKeeper
>>>>>>
>>>>>> on.
>>>>>>>>>>
>>>>>>>>>> </description>  </property>  <property>
>>>>>>>>>> <name>hbase.zookeeper.property.tickTime</name>
>>>>>>>>>> <value>1</value>  <description>Property from ZooKeeper's
>>>>>>>>>> config zoo.cfg. The number of milliseconds of each tick.
>>>>>>>>>> See zookeeper.session.timeout description.</description>
>>>>>>>>>> </property>  <property>  <name>zookeeper.retries</name>
>>>>>>>>>> <value>5</value>  <description>How many times to retry
>>>>>>>>>> connections to ZooKeeper.
>>>>>>
>>>>>> Used
>>>>>>>>>
>>>>>>>>> for
>>>>>>>>>>
>>>>>>>>>> reading/writing root region location, checking/writing out
>>>>>>>>>> of
>>>>>>>
>>>>>>> safe
>>>>>>>>>>
>>>>>>>>>> mode. Used together with ${zookeeper.pause} in an
>>>>>>>>>> exponential backoff fashion when making queries to
>>>>>>>>>> ZooKeeper.</description>  </property>  <property>
>>>>>>>>>> <name>hbase.zookeeper.property.clientPort</name>
>>>>>>>>>> <value>2222</value>  <description>Property from ZooKeeper's
>>>>>>>>>> config zoo.cfg. the port at which the clients will
>>>>>>>>>> connect.</description>  </property>
>>>>>>>>>>
>>>>>>>>>> <property>  <name>dfs.replication</name>  <value>3</value>
>>>>>>>>>> <description>Default block replication. The actual number
>>>>>>>>>> of replications can be specified when the file is
>>>>>>>>>
>>>>>>>>> created.
>>>>>>>>>>
>>>>>>>>>> The default is used if replication is not specified in
>>>>>>>>>> create time.</description>  </property>  </configuration>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The hbase-site.xml in slave1, slave2<configuration>
>>>>>>>>>> <property>  <name>hbase.rootdir</name>
>>>>>>>>>> <value>hdfs://Hadoopserver:54310/hbase</value>
>>>>>>>>>> <description>The directory shared by region servers. Should
>>>>>>>>>> be fully-qualified to include the filesystem to use. E.g:
>>>>>>>>>> hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description>
>>>>>>>>>> </property>  <property>
>>>>>>>>>> <name>hbase.cluster.distributed</name>  <value>true</value>
>>>>>>>>>> <description>The mode the cluster will be in. Possible
>>>>>>>>>> values are false: standalone and pseudo-distributed setups
>>>>>>>>>> with managed
>>>>>>>>>
>>>>>>>>> Zookeeper
>>>>>>>>>>
>>>>>>>>>> true: fully-distributed with unmanaged Zookeeper Quorum
>>>>>>>>>> (see hbase-env.sh)</description>  </property>  <property>
>>>>>>>>>> <name>hbase.zookeeper.quorum</name>
>>>>>>>>>> <value>Hadoopserver,Hadoopclient1,Hadoopclient</value>
>>>>>>>>>> <description>Comma separated list of servers in the
>>>>>>>>>> ZooKeeper
>>>>>>>
>>>>>>> Quorum.
>>>>>>>>>>
>>>>>>>>>> For example, "host1.mydomain.com,host2.mydomain.com,
>>>>>>>>>
>>>>>>>>> host3.mydomain.com".
>>>>>>>>>>
>>>>>>>>>> By default this is set to localhost for local and
>>>>>>>
>>>>>>> pseudo-distributed
>>>>>>>>>>
>>>>>>>>>> modes of operation. For a fully-distributed setup, this
>>>>>>>>>> should be set
>>>>>>
>>>>>> to
>>>>>>>
>>>>>>> a
>>>>>>>>>>
>>>>>>>>>> full list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK
>>>>>>>>>> is set in hbase-env.sh this is the list of servers which we
>>>>>>>>>> will start/stop ZooKeeper
>>>>>>
>>>>>> on.
>>>>>>>>>>
>>>>>>>>>> </description>  </property>
>>>>>>>>>>
>>>>>>>>>> <property>
>>>>>>>>>> <name>hbase.zookeeper.property.clientPort</name>
>>>>>>>>>> <value>2222</value>  <description>Property from ZooKeeper's
>>>>>>>>>> config zoo.cfg. the port at which the clients will
>>>>>>>>>> connect.</description>  </property>
>>>>>>>>>>
>>>>>>>>>> <property>  <name>dfs.replication</name>  <value>3</value>
>>>>>>>>>> <description>Default block replication. The actual number
>>>>>>>>>> of replications can be specified when the file is
>>>>>>>>>
>>>>>>>>> created.
>>>>>>>>>>
>>>>>>>>>> The default is used if replication is not specified in
>>>>>>>>>> create time.</description>  </property>  </configuration>
>>>>>>>>>>
>>>>>>>>>> Regionservers ( only on Master in my case Hadoopserver)
>>>>>>>>>>
>>>>>>>>>> Hadoopserver Hadoopclient1 Hadoopclient
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regionservers ( on slaves) localhost
>>>>>>>>>>
>>>>>>>>>> Blocked with this error for the past one week. Googled so
>>>>>>>>>> much didnt
>>>>>>>
>>>>>>> get
>>>>>>>>>
>>>>>>>>> any
>>>>>>>>>>
>>>>>>>>>> solution.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> REgs. senthil
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message