hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "alan wootton (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-214) ClusterTestDFS fails
Date Tue, 23 May 2006 22:19:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-214?page=comments#action_12413015 ] 

alan wootton commented on HADOOP-214:

Here's the latest on this issue.

I have a new patch that fixes the two main DFS tests. I think it would be crazy for us to
move onward without having good DFS tests. 

There is a 'name server' hack in the patch now. The requires the system property "sun.net.spi.nameservice.provider"
to be set to "dns,mine".
It is not good enough to just call: 


in the code unless you can do it BEFORE anyone else initializes InetAddress.java. JUnit is
bad this way so the property must be set in the ant script that calls JUnit. This is in the
latest patch.

If you want to call the unit test from eclipse then you will need to pass 
to the jvm. Or, you might define " localhost0" through " localhost15" in
your hosts file and get the same result.

Speaking of build.xml, The definition of the 'test-cluster' target tries to run ANYTHING in
the test directory as a unit test - this generates lots of failure messages (e.g. RandomDatum
is not a TestCase so it fails). I changed it so that the test-cluster target only runs files
of the form "ClusterTest*.java". ToCpp.java, FromCpp.java, and DistributedFSCheck.java are
NOT being run. To include them we should either make a new ant target (preferred), or change
their name to start with ClusterTest. Or change the ant script.

I changed several constants in the ClusterTest*.java tests to make them run faster. I divided
all the timeouts by 10. I removed the 'already existing' file creation test from ClusterTestDFSNamespaceLogging
because it fails by timing out and that takes 3 minutes. When we can override LEASE_PERIOD
then we can put that test back in. The whole cluster-test target now runs in 4 min.

ClusterTestDFSNamespaceLogging and ClusterTestDFS have VERY similar DFS simulations. As does
MiniDfsCluster. I think someone needs to factor-out the dfs sim from all three of these tests.

> ClusterTestDFS fails
> --------------------
>          Key: HADOOP-214
>          URL: http://issues.apache.org/jira/browse/HADOOP-214
>      Project: Hadoop
>         Type: Bug

>   Components: dfs
>     Versions: 0.2.1
>  Environment: local workstation (windows) 
>     Reporter: alan wootton
>  Attachments: fix_clustertestdfs.patch
> The dfs unit tests, from the ant target 'cluster' have been failing. (ClusterTestDFSNamespaceLogging,
ClusterTestDFS). I don't know if anyone but me cares about these tests, but I do. I would
like to write better tests for dns. I think we all need that.
> They have been partially broken since  "test.dfs.same.host.targets.allowed" went away
and replication ceased for these tests. 
> They got really broken when NameNode stopped automatically formatting itself .
> Since they seem to be ignored, I took the liberty of changing how they work.
> The main thing is, you must put this into your hosts file:
>       localhost0
>       localhost1
>       localhost2
>       localhost3
>       localhost4
>       localhost5
>       localhost6
>       localhost7
>       localhost8
>       localhost9
>       localhost10
>       localhost11
>       localhost12
>       localhost13
>       localhost14
>       localhost15
> This way you can start DataNodes, and TaskTrackers (up to 16 of them) with unique hostnames.
> Also, I changed all the places that used to call InetAddress.getLocalHost().getHostName()
to get it from a new method in Configuration (this issue is the same as http://issues.apache.org/jira/browse/HADOOP-197

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message