hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4491) Parallel testing HDFS
Date Wed, 04 Sep 2013 19:16:54 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth updated HDFS-4491:
--------------------------------

    Attachment: HDFS-4491--n12.patch

bq. Does MiniCluster work with ports in a different way when run on Windows?

I can repro the problem consistently on Mac, so the problem is not specific to Windows.

bq. Definitely NetUtils.getFreeSocketPort which is used by MiniDFSCluster to find ports for
data nodes is a source of problems as it finds an available socket, the closes it, then returns
the port number to the caller.

Yes, this could be a race, but I'm seeing a different problem.  I'm seeing that the test consistently
uses the same static ports: 50020 for RPC and 50075 for HTTP.

I traced the problem to {{MiniDFSCluster#setupDatanodeAddress}}.  This method doesn't set
ephemeral ports for HTTP and RPC in certain conditions.  I'm attaching version 12 of the patch
with a small change in that method to correct it.  I've had multiple successful parallel test
runs with this patch, even while using netcat to hold ports 50020 and 50075.  Does this change
look good?

I think this is the last issue we would need to resolve before committing.

                
> Parallel testing HDFS
> ---------------------
>
>                 Key: HDFS-4491
>                 URL: https://issues.apache.org/jira/browse/HDFS-4491
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: test
>    Affects Versions: 3.0.0
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Andrey Klochkov
>         Attachments: HDFS-4491--n10.patch, HDFS-4491--n10.patch, HDFS-4491--n11.patch,
HDFS-4491--n11.patch, HDFS-4491--n12.patch, HDFS-4491--n2.patch, HDFS-4491--n3.patch, HDFS-4491--n4.patch,
HDFS-4491--n5.patch, HDFS-4491--n6.patch, HDFS-4491--n7.patch, HDFS-4491--n8.patch, HDFS-4491--n9.patch,
HDFS-4491.patch
>
>
> Parallel execution of HDFS tests in multiple forks. See HADOOP-9287 for details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message