hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuti Awasthi <stutiawas...@hcl.com>
Subject Queries on Zookeeper failure and RegionServer restartup
Date Tue, 20 Sep 2011 07:58:16 GMT
Hi all,

I have 2 node cluster. I run Regionserver, Zookeeper on both nodes and Master on 1 and Backup
Master on other.

Here what I did : I stopped Zookeeper on 1 node and after that I was unable to access Hbase.

ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper
but the connection closes immediately. This could be a sign that the server has too many connections
(30 is the default). Consider inspecting your ZK server logs for that error and then make
sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more
information.

Queries :

1.        If one of the zookeeper is going down , cluster is inaccessible then why we are
running multiple zookeeper nodes?

2.       Is there some way that if one of zookeeper nodes are working, cluster can be accessible?

Some other test :
If I stop RegionServer and Master on 1 node, then bakupMaster becomes Master and I can access
the Hbase cluster but when I try to restart Region server on the same node on which I have
shut down it gives me following error . How to fix this ?

2011-09-20 12:06:03,647 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING
region server serverName=master,60020,1316500563205, load=(requests=0, regions=0, usedHeap=22,
maxHeap=993): Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server
master,60020,1316500563205 has been rejected; Reported time is too far out of sync with master.
 Time difference of 352381ms > max allowed of 30000ms
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException:
Server master,60020,1316500563205 has been rejected; Reported time is too far out of sync
with master.  Time difference of 352381ms > max allowed of 30000ms
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
                at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
                at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1479)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:571)
                at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException:
Server master,60020,1316500563205 has been rejected; Reported time is too far out of sync
with master.  Time difference of 352381ms > max allowed of 30000ms
                at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:181)
                at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:129)
                at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:615)


Your inputs are required

Thanks
Stuti

________________________________
::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named
recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or
opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of
HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and
/ or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited.
If you have
received this email in error please delete it and notify the sender immediately. Before opening
any mail and
attachments please check them for viruses and defect.

-----------------------------------------------------------------------------------------------------------------------

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message