hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arthur.hk.chan@gmail.com" <arthur.hk.c...@gmail.com>
Subject Re: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a roll of the active NN
Date Mon, 04 Aug 2014 12:37:25 GMT
Hi,

Thanks for your reply.
It was about StandBy Namenode not promoted to Active.
Can you please advise what the path of ZKFC logs?  

"Similar to Namenode status web page, a Cluster Web Console is added in federation to monitor
the federated cluster at http://<any_nn_host:port>/dfsclusterhealth.jsp. Any Namenode
in the cluster can be used to access this web page” 
What is the default port for the cluster console? I tried 8088 but no luck.

Please advise.

Regards
Arthur




On 4 Aug, 2014, at 7:22 pm, Brahma Reddy Battula <brahmareddy.battula@huawei.com> wrote:

> HI,
> 
> 
> DO you mean Active Namenode which is killed is not transition to STANDBY..?
> 
> >>> Here Namenode will not start as standby if you kill..Again you need to start
manually.
>         
>       Automatic failover means when over Active goes down Standy Node will transition
to Active automatically..it's not like starting killed process and making the Active(which
is standby.)
> 
> Please refer the following doc for same ..( Section : Verifying automatic failover)
> 
> http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html
> 
> OR
> 
>  DO you mean Standby Namenode is not transition to ACTIVE..?
> 
> >>>> Please check ZKFC logs,, Mostly this might not happen from the logs
you pasted
> 
> 
> Thanks & Regards
>  
> Brahma Reddy Battula
>  
> 
> From: Arthur.hk.chan@gmail.com [arthur.hk.chan@gmail.com]
> Sent: Monday, August 04, 2014 4:38 PM
> To: user@hadoop.apache.org
> Cc: Arthur.hk.chan@gmail.com
> Subject: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a roll of
the active NN
> 
> Hi,
> 
> I have setup Hadoop 2.4.1 HA Cluster using Quorum Journal, I am verifying automatic failover,
after killing the process of namenode from Active one, the name node was not failover to standby
node, 
> 
> Please advise
> Regards
> Arthur
> 
> 
> 2014-08-04 18:54:40,453 WARN org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer:
Unable to trigger a roll of the active NN
> java.net.ConnectException: Call From standbynode  to  activenode:8020 failed on connection
exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
> at org.apache.hadoop.ipc.Client.call(Client.java:1414)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:271)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:313)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
> at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462)
> at org.apache.hadoop.ipc.Client.call(Client.java:1381)
> ... 11 more
> 2014-08-04 18:55:03,458 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020,
call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from activenode:54571 Call#17
Retry#1: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported
in state standby
> 2014-08-04 18:55:06,683 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 8020,
call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from activenode:54571 Call#17
Retry#3: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported
in state standby
> 2014-08-04 18:55:16,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 8020,
call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from activenode:54602 Call#0
Retry#1: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported
in state standby
> 2014-08-04 18:55:19,530 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020,
call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from activenode:54610 Call#17
Retry#5: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported
in state standby
> 2014-08-04 18:55:20,756 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020,
call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from activenode:54602 Call#0
Retry#3: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported
in state standby


Mime
View raw message