hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10723) TestRenameWhileOpen tries to restart NN with the same rpc port
Date Fri, 05 Aug 2016 18:56:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15409906#comment-15409906
] 

Vrushali C commented on HDFS-10723:
-----------------------------------

I updated the test to use a random port number. The test output that I see is as follows:

{code}
[machine-channapattan hadoop-hdfs (trunk)]$ grep -rni -A2 "nnport=" target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:2:Will now start
MiniDFSCluster with nnport=58999
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-3-2016-08-05
11:51:56,081 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(465)) - starting
cluster: numNameNodes=1, numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-4-2016-08-05
11:51:56,319 [main] DEBUG namenode.NameNode (NameNode.java:initializeGenericKeys(1670)) -
Setting fs.defaultFS to hdfs://127.0.0.1:58999
--
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:1592:Will now
start MiniDFSCluster with nnport=7009
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-1593-2016-08-05
11:52:09,583 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(465)) - starting
cluster: numNameNodes=1, numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-1594-2016-08-05
11:52:09,587 [main] DEBUG namenode.NameNode (NameNode.java:initializeGenericKeys(1670)) -
Setting fs.defaultFS to hdfs://127.0.0.1:7009
--
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:2523:Will now
start MiniDFSCluster with nnport=17516
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-2524-2016-08-05
11:52:20,090 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(465)) - starting
cluster: numNameNodes=1, numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-2525-2016-08-05
11:52:20,093 [main] DEBUG namenode.NameNode (NameNode.java:initializeGenericKeys(1670)) -
Setting fs.defaultFS to hdfs://127.0.0.1:17516
--
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:3450:Will now
start MiniDFSCluster with nnport=28234
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-3451-2016-08-05
11:52:30,808 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(465)) - starting
cluster: numNameNodes=1, numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-3452-2016-08-05
11:52:30,811 [main] DEBUG namenode.NameNode (NameNode.java:initializeGenericKeys(1670)) -
Setting fs.defaultFS to hdfs://127.0.0.1:28234
[machine-channapattan hadoop-hdfs (trunk)]$
{code}

So I still can't upload a patch to HDFS jiras. Hence pasting my patch here:

{code}
[machine-channapattan hadoop (trunk)]$ cat ../HDFS-10723.001.patch
diff --git hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
index 949fc74..9d6b127 100644
--- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
+++ hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
@@ -55,10 +55,16 @@ public void testWhileOpenRenameParent() throws IOException {
     conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1);
     conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
     conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, TestFileCreation.blockSize);
+    System.out.println("Test 1*****************************");

+    // try to get a random port number < 65535
+    // hence take mod 65534
+    final int nnport = (int) (System.currentTimeMillis() % 65534);
+    System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
     // create cluster
-    System.out.println("Test 1*****************************");
-    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+        .nameNodePort(nnport).format(true).build();
+
     FileSystem fs = null;
     try {
       cluster.waitActive();
@@ -72,8 +78,6 @@ public void testWhileOpenRenameParent() throws IOException {
       doNothing().when(spyLog).endCurrentLogSegment(Mockito.anyBoolean());
       DFSTestUtil.setEditLogForTesting(cluster.getNamesystem(), spyLog);

-      final int nnport = cluster.getNameNodePort();
-
       // create file1.
       Path dir1 = new Path("/user/a+b/dir1");
       Path file1 = new Path(dir1, "file1");
@@ -155,13 +159,18 @@ public void testWhileOpenRenameParentToNonexistentDir() throws IOException
{
     conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
     System.out.println("Test 2************************************");

+    // try to get a random port number < 65535
+    // hence take mod 65534
+    final int nnport = (int) (System.currentTimeMillis() % 65534);
+    System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
     // create cluster
-    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+        .nameNodePort(nnport).format(true).build();
+
     FileSystem fs = null;
     try {
       cluster.waitActive();
       fs = cluster.getFileSystem();
-      final int nnport = cluster.getNameNodePort();

       // create file1.
       Path dir1 = new Path("/user/dir1");
@@ -230,13 +239,18 @@ public void testWhileOpenRenameToExistentDirectory() throws IOException
{
     conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
     System.out.println("Test 3************************************");

+    // try to get a random port number < 65535
+    // hence take mod 65534
+    final int nnport = (int) (System.currentTimeMillis() % 65534);
+    System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
     // create cluster
-    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+        .nameNodePort(nnport).format(true).build();
+
     FileSystem fs = null;
     try {
       cluster.waitActive();
       fs = cluster.getFileSystem();
-      final int nnport = cluster.getNameNodePort();

       // create file1.
       Path dir1 = new Path("/user/dir1");
@@ -295,13 +309,18 @@ public void testWhileOpenRenameToNonExistentDirectory() throws IOException
{
     conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
     System.out.println("Test 4************************************");

+    // try to get a random port number < 65535
+    // hence take mod 65534
+    final int nnport = (int) (System.currentTimeMillis() % 65534);
+    System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
     // create cluster
-    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+        .nameNodePort(nnport).format(true).build();
+
     FileSystem fs = null;
     try {
       cluster.waitActive();
       fs = cluster.getFileSystem();
-      final int nnport = cluster.getNameNodePort();

       // create file1.
       Path dir1 = new Path("/user/dir1");
[machine-channapattan hadoop (trunk)]$
{code}

Would appreciate your feedback.

> TestRenameWhileOpen tries to restart NN with the same rpc port
> --------------------------------------------------------------
>
>                 Key: HDFS-10723
>                 URL: https://issues.apache.org/jira/browse/HDFS-10723
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>            Reporter: Kihwal Lee
>         Attachments: testWhileOpenRenameParentToNonexistentDir.txt
>
>
> In {{testWhileOpenRenameParentToNonexistentDir}},
> {code}
>       // restart cluster with the same namenode port as before.
>       // This ensures that leases are persisted in fsimage.
>       cluster.shutdown();
> {code}
> And we are getting test failures in precommit builds.
> {noformat}
> java.net.BindException: Problem binding to [localhost:47395]
>  java.net.BindException: Address already in use;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message