hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leonid Fedotov <lfedo...@hortonworks.com>
Subject Re: 答复: how config multi regionserver, or what is wrong?
Date Tue, 11 Dec 2012 07:28:27 GMT
Your nodes are too much out of sync with time.
> Reported time is too far out of sync with master.  Time difference of 496371ms > max
allowed of 30000ms
You need to set up time synchronization service for your cluster.

Thank you!

Sincerely,
Leonid Fedotov
Hortonworks support team


On Dec 10, 2012, at 6:17 PM, tgh wrote:

> Meanwhile , log is master , that is, blade1, is like this,  there are some ERRor like
this, for  
> 
> 2012-09-01 06:31:05,558 INFO org.apache.hadoop.hbase.master.ServerManager: Registering
server=blade2,60020,1346452716636, regionCount=0, userLoad=false
> 2012-09-01 06:31:05,569 WARN org.apache.hadoop.hbase.master.ServerManager: Server blade4,60020,1346451768443
has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms
> max allowed of 30000ms
> 2012-09-01 06:31:05,581 WARN org.apache.hadoop.hbase.master.ServerManager: Server blade5,60020,1346452001672
has been rejected; Reported time is too far out of sync with master.  Time difference of 263137ms
> max allowed of 30000ms
> 2012-09-01 06:31:05,583 ERROR org.apache.hadoop.hbase.master.HMaster: Region server serverName=blade4,60020,1346451768443,
load=(requests=0, regions=0, usedHeap=142, maxHeap=966) reported a fatal error:
> ABORTING region server serverName=blade4,60020,1346451768443, load=(requests=0, regions=0,
usedHeap=142, maxHeap=966): Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException:
Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync
with master.  Time difference of 496371ms > max allowed of 30000ms
> Cause:
> org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException:
Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync
with master.  Time difference of 496371ms > max allowed of 30000ms
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
>        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1574)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1531)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>        at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException:
Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync
with master.  Time difference of 496371ms > max allowed of 30000ms
>        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:193)
>        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:141)
>        at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:675)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:601)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> 
>        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>        at $Proxy5.regionServerStartup(Unknown Source)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1570)
>        ... 3 more
> 
> 
> -----邮件原件-----
> 发件人: user-return-32385-guanhua.tian=ia.ac.cn@hbase.apache.org [mailto:user-return-32385-guanhua.tian=ia.ac.cn@hbase.apache.org]
代表 tgh
> 发送时间: 2012年12月11日 9:59
> 收件人: user@hbase.apache.org
> 主题: 答复: how config multi regionserver, or what is wrong?
> 
> And our hosts is follows
> 
> [root@blade1 ~]# cat /etc/hosts
> 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
> ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
> 
> 192.168.76.233 blade1
> 192.168.76.234 blade2
> 192.168.76.235 blade3
> 192.168.76.236 blade4
> 192.168.76.237 blade5
> 192.168.76.238 blade6
> 192.168.76.239 blade7
> 192.168.76.240 blade8
> 
> 192.168.76.245 fnode1
> 192.168.76.246 fnode2
> [root@blade1 ~]#
> 
> -----邮件原件-----
> 发件人: user-return-32384-guanhua.tian=ia.ac.cn@hbase.apache.org [mailto:user-return-32384-guanhua.tian=ia.ac.cn@hbase.apache.org]
代表 tgh
> 发送时间: 2012年12月11日 9:00
> 收件人: user@hbase.apache.org
> 主题: 答复: how config multi regionserver, or what is wrong?
> 
> Thank you for your reply,
> And the configuration file is here, 
> Could you help me,
> 
> 
> Thank you
> ---------------------------
> Tian Guanhua
> 
> 
> 
> [root@blade1 conf]# cat regionservers 
> blade1
> blade2
> blade3
> blade4
> blade5
> blade6
> blade7
> blade8
> [root@blade1 conf]#
> [root@blade1 conf]# vim hbase-site.xml
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <!-- 
> -->
> <configuration>
> <property>
>    <name>hbase.rootdir</name>
>    <value>hdfs://blade1:9000/hbase</value>
>    <description>The directory shared by RegionServers.</description>
> </property>
> <property>
>    <name>hbase.cluster.distributed</name>
>    <value>true</value>
> </property>
> <property>
>    <name>hbase.zookeeper.quorum</name>
>    <value>blade1,blade2,blade3</value>
> </property>
> <property>
>    <name>hbase.zookeeper.property.dataDir</name>
>   <value>/home/liuxin/zookeeper/data</value>
> </property>
> <property>
>  <name>dfs.support.append</name>
>   <value>true</value>
> </property>
> <property> 
>  <name>dfs.datanode.max.xcievers</name> 
>   <value>4096</value>
>   </property>
> <property> 
> <name>hbase.master</name> 
> <value>blade1:60000</value> 
> </property> 
> </configuration>
> 
> 
> 
> 
> 
> 
> -----邮件原件-----
> 发件人: user-return-32370-guanhua.tian=ia.ac.cn@hbase.apache.org [mailto:user-return-32370-guanhua.tian=ia.ac.cn@hbase.apache.org]
代表 Jean-Marc Spaggiari
> 发送时间: 2012年12月10日 20:54
> 收件人: user@hbase.apache.org
> 主题: Re: how config multi regionserver, or what is wrong?
> 
> Hi Tian,
> 
> Can you share you configuration files?
> 
> Do you have something like that on your hbase-site.xml file?
> 
>  <property>
>    <name>hbase.cluster.distributed</name>
>    <value>true</value>
>    <description>The mode the cluster will be in. Possible values are
>      false: standalone and pseudo-distributed setups with managed Zookeeper
>      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
>    </description>
>  </property>
> 
> 
> JM
> 
> 2012/12/10, tgh <guanhua.tian@ia.ac.cn>:
>> Hi
>> 	I try to use hbase, and now ,I have a problem with hbase 
>> configuration, I use 8node for try, and it seems to work, hadoop, 
>> zookeeper, hbase all boot up, and it could insert into with putAPI , 
>> But when I try to use masterIP:60010 to manage it, I find there are 
>> only one regionserver there, and it is localhost:60030, why?
>> 	I have set regionserver file , there are 8 nodes there, and all could 
>> ssh to each other without passwd,
>> 
>> 	And I also use hdfsNamenodeIP:50070 to see HDFS, and it seems OK, the 
>> data have been balanced across 8node, but I wander hbase only have one 
>> regionserver to work, although when I star/stop regionserver , there 
>> are 8region server to start and stop,
>> 
>> 	And I try to put data into hbase, it seems ok at first, but after 
>> 200million in hbase, it seems really hard to insert more into it, it 
>> is very slow, and I use masterIP:60010 to manage it, I find there are 
>> only one regionserver there, and it is localhost:60030, why?
>> 
>> 	
>> 	Could you help me,
>> 
>> 	
>> 
>> Thank you
>> -------------------
>> Tian Guanhua
>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message