Hi Yu
=A0=A0

=A0=A0 I think when un= plug the nic ,the ssh could not make through because it can not connect to= =A0 failed=A0 active NN.
Suppose that ,the sshfence will failed.

=A0=A0 Am I right?

2= 013/12/3 YouPeng Yang <yypvsxf19870706@gmail.com>

Hi Yu

=A0 Thanks for your resp= onse.
=A0 I'm sure my ssh setup is good. Ssh from=A0 act NN to= stanby nn need no password.
=A0

<= div>=A0=A0

I attached my config
------core-site.xml-----= ------------

<configuration>
=A0<property>=
=A0=A0=A0=A0 <name>fs.defaultFS</name>
=A0=A0=A0=A0 <value>hdfs://lklcluster</value>
=A0=A0=A0=A0 &= lt;final>true</final>
=A0</property>
=A0
=A0<pro= perty>
=A0=A0=A0=A0 <name>hadoop.tmp.dir</name>
=A0=A0= =A0=A0 <value>/home/hadoop/tmp2</value>
=A0</property>

</configuration>

---= ----hdfs-site.xml----------
---

<configurati= on>
=A0<property>
=A0=A0=A0=A0 <name>dfs.namenode.name= .dir</name>
=A0=A0=A0 <value>/home/hadoop/namedir2</value>=A0
=A0</property>

=A0<property>
=A0=A0=A0=A0 <name>= ;dfs.datanode.data.dir</name>
=A0=A0=A0=A0 <value>/home/hado= op/datadir2</value>
=A0</property>

=A0<property>= ;
=A0=A0 <name>dfs.nameservices</name>
=A0=A0 <value>lklcluster</value>
</property>

&l= t;property>
=A0=A0=A0 <name>dfs.ha.namenodes.lklcluster</nam= e>
=A0=A0=A0 <value>nn1,nn2</value>
</property><= br><property>
=A0 <name>dfs.namenode.rpc-address.lklcluster.nn1</name>
=A0= <value>hadoop2:8020</value>
</property>
<proper= ty>
=A0=A0=A0 <name>dfs.namenode.rpc-address.lklcluster.nn2<= /name>
=A0=A0=A0 <value>hadoop3:8020</value>
</property>
<= br><property>
=A0 <name>dfs.namenode.http-address.lklcluster= .nn1</name>
=A0=A0=A0 <value>hadoop2:50070</value>
= </property>

<property>
=A0=A0=A0 <name>dfs.namenode.http-address.lkl= cluster.nn2</name>
=A0=A0=A0 <value>hadoop3:50070</value&= gt;
</property>

<property>
=A0 <name>dfs.nam= enode.shared.edits.dir</name>
=A0 <value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/lklclust= er</value>
</property>
<property>
=A0 <name&g= t;dfs.client.failover.proxy.provider.lklcluster</name>
=A0 <val= ue>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProv= ider</value>
</property>
<property>
=A0 <name>dfs.ha.fencing.met= hods</name>
=A0 <value>sshfence</value>
</proper= ty>

<property>
=A0 <name>dfs.ha.fencing.ssh.privat= e-key-files</name>
=A0=A0 <value>/home/hadoop/.ssh/id_rsa</value>
</property= >
=A0=A0=A0
<property>=A0
=A0=A0=A0 <name>dfs.ha.= fencing.ssh.connect-timeout</name>=A0
=A0=A0=A0=A0 <value>5= 000</value>=A0
</property>=A0

<property>
=A0 <name>dfs.journalnode.edits.dir</name&= gt;
=A0=A0 <value>/home/hadoop/journal/data</value>
</= property>

<property>
=A0=A0 <name>dfs.ha.automatic= -failover.enabled</name>
=A0=A0=A0=A0=A0 <value>true</value>
</property>
=A0= =A0=A0=A0=A0=A0=A0
<property>
=A0=A0=A0=A0 <name>ha.zook= eeper.quorum</name>
=A0=A0=A0=A0 <value>hadoop1:2181,hadoop2= :2181,hadoop3:2181</value>
</property>

</configuration>

=

2013/12/3 Azuryy Yu <= azuryyyu@gmail.com>

This is still because your = fence method configuraed improperly.=A0
plseae paste your fence configu= ration. and double check you can ssh on active NN to standby NN without pas= sword.

On Tue, Dec 3, 2013 at 10:23 AM, YouPeng= Yang <yypvsxf19870706@gmail.com> wrote:

Hi
=A0=A0 Another auto-failover te= sting problem:
=A0
=A0=A0 My HA can auto-failover after I kill= the active NN.When it comes to the unplug=A0 network interface to simulate= the hardware fail,the auto-failover seems=A0 not to work after=A0=A0 wait = for times -the zkfc logs as [1].

=A0=A0 I'm using the default sshfence.
=A0=A0
=A0

[1] zkfc logs-------------------------------= ---------------------------------------------------------
2013-12-03 10:= 05:56,650 INFO org.apache.hadoop.ha.NodeFencer: =3D=3D=3D=3D=3D=3D Beginnin= g Service Fencing Process... =3D=3D=3D=3D=3D=3D
2013-12-03 10:05:56,650 INFO org.apache.hadoop.ha.NodeFencer: Trying method= 1/1: org.apache.hadoop.ha.SshFenceByTcpPort(null)
2013-12-03 10:05:56,6= 51 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connecting to hadoop3... 2013-12-03 10:05:56,651 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: C= onnecting to hadoop3 port 22
2013-12-03 10:05:59,648 WARN org.apache.had= oop.ha.SshFenceByTcpPort: Unable to connect to hadoop3 as user hadoop
com.jcraft.jsch.JSchException: java.net.NoRouteToHostException: No route to= host
=A0=A0=A0 at com.jcraft.jsch.Util.createSocket(Util.java:386)
= =A0=A0=A0 at com.jcraft.jsch.Session.connect(Session.java:182)
=A0=A0=A0= at org.apache.hadoop.ha.SshFenceByTcpPort.tryFence(SshFenceByTcpPort.java:= 100)
=A0=A0=A0 at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:97)
= =A0=A0=A0 at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverCo= ntroller.java:521)
=A0=A0=A0 at org.apache.hadoop.ha.ZKFailoverControlle= r.fenceOldActive(ZKFailoverController.java:494)
=A0=A0=A0 at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailov= erController.java:59)
=A0=A0=A0 at org.apache.hadoop.ha.ZKFailoverContro= ller$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:837)
=A0= =A0=A0 at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveSt= andbyElector.java:900)
=A0=A0=A0 at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveS= tandbyElector.java:799)
=A0=A0=A0 at org.apache.hadoop.ha.ActiveStandbyE= lector.processResult(ActiveStandbyElector.java:415)
=A0=A0=A0 at org.apa= che.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:596)
=A0=A0=A0 at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.jav= a:495)
2013-12-03 10:05:59,649 WARN org.apache.hadoop.ha.NodeFencer: Fen= cing method org.apache.hadoop.ha.SshFenceByTcpPort(null) was unsuccessful.<= br> 2013-12-03 10:05:59,649 ERROR org.apache.hadoop.ha.NodeFencer: Unable to fe= nce service by any configured method.
2013-12-03 10:05:59,650 WARN org.a= pache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of ele= ction
java.lang.RuntimeException: Unable to fence NameNode at hadoop3/10.7.23.124:8020
=A0=A0= =A0 at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverControll= er.java:522)
=A0=A0=A0 at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFai= loverController.java:494)
=A0=A0=A0 at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailov= erController.java:59)
=A0=A0=A0 at org.apache.hadoop.ha.ZKFailoverContro= ller$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:837)
=A0= =A0=A0 at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveSt= andbyElector.java:900)
=A0=A0=A0 at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveS= tandbyElector.java:799)
=A0=A0=A0 at org.apache.hadoop.ha.ActiveStandbyE= lector.processResult(ActiveStandbyElector.java:415)
=A0=A0=A0 at org.apa= che.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:596)
=A0=A0=A0 at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.jav= a:495)
2013-12-03 10:05:59,650 INFO org.apache.hadoop.ha.ActiveStandbyEl= ector: Trying to re-establish ZK session
2013-12-03 10:05:59,676 INFO or= g.apache.zookeeper.ZooKeeper: Session: 0x142931031810260 closed
2013-12-03 10:06:00,678 INFO org.apache.zookeeper.ZooKeeper: Initiating cli= ent connection, connectString=3Dhadoop1:2181,hadoop2:2181,hadoop3:2181 sess= ionTimeout=3D5000 watcher=3Dorg.apache.hadoop.ha.ActiveStandbyElector$Watch= erWithClientRef@5ce2acea
2013-12-03 10:06:00,681 INFO org.apache.zookeeper.ClientCnxn: Opening socke= t connection to server hadoop1/10.7.23.122:2181. Will not attempt to authenticate using SA= SL (Unable to locate a login configuration)
2013-12-03 10:06:00,681 INFO org.apache.zookeeper.ClientCnxn: Socket connec= tion established to hadoop1/10.7.23.122:2181, initiating session
2013-12-03 10:06:00,709 = INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on ser= ver hadoop1/10.7.23.1= 22:2181, sessionid =3D 0x142931031810261, negotiated timeout =3D 5000 2013-12-03 10:06:00,711 INFO org.apache.zookeeper.ClientCnxn: EventThread s= hut down