Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 10 Dec 2015 18:37:11 +0000 (UTC)
From: "Xiao Chen (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12762818.1418987003000.328147.1449772631149@Atlassian.JIRA>
In-Reply-To: <JIRA.12762818.1418987003000@Atlassian.JIRA>
References: <JIRA.12762818.1418987003000@Atlassian.JIRA>
 <JIRA.12762818.1418987003969@arcas>
Subject: [jira] [Commented] (HDFS-7553) fix the TestDFSUpgradeWithHA due to
 BindException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051427#comment-15051427 ] 

Xiao Chen commented on HDFS-7553:
---------------------------------

Not sure if [~xieliang007] is still working on this, but I want to self-promote patch 2 with a way to reproduce. Attached is the reproduce patch which includes patch 2, that fails with the bind exception when restarting NN. Uncommenting the {{httpServer.join();}} at {{NameNode.java#932}} makes the test pass. The test case I ran was {{testNfsUpgrade}}.

In the repro patch, the exception is thrown from {{HttpServer2.openListeners}}, instead of the inner jetty + net classes. But IMHO this already proves that we need a {{join}} to have a successful immediate NN restart. (Arguably though, with this the NN shutdown will be slower, and potentially hang if somehow the webserver hangs in its shutdown. I think this would be the webserver's bug, and could be enhanced in a separate JIRA. E.g. add a timeout to the NN shutdown)

Thanks [~wheat9] for helping understand the problem, and [~cnauroth] for the initial thought - turns out to be exactly the cause! Please feel free try out the repro patch and review patch 2.

> fix the TestDFSUpgradeWithHA due to BindException
> -------------------------------------------------
>
>                 Key: HDFS-7553
>                 URL: https://issues.apache.org/jira/browse/HDFS-7553
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: test
>    Affects Versions: 2.7.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>              Labels: BB2015-05-TBR
>         Attachments: HDFS-7553-001.txt, HDFS-7553.002.patch, HDFS-7553.repro.patch
>
>
> see https://builds.apache.org/job/PreCommit-HDFS-Build/9092//testReport/org.apache.hadoop.hdfs.server.namenode.ha/TestDFSUpgradeWithHA/testNfsUpgrade/ :
> Error Message
> Port in use: localhost:57896
> Stacktrace
> java.net.BindException: Port in use: localhost:57896
> 	at sun.nio.ch.Net.bind0(Native Method)
> 	at sun.nio.ch.Net.bind(Net.java:444)
> 	at sun.nio.ch.Net.bind(Net.java:436)
> 	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> 	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> 	at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> 	at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:868)
> 	at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:809)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:704)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:591)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:763)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:747)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1443)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1815)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1796)
> 	at org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testNfsUpgrade(TestDFSUpgradeWithHA.java:285)


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)