hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2209) MiniDFS cluster improvements
Date Wed, 27 Jul 2011 15:55:09 GMT
MiniDFS cluster improvements

                 Key: HDFS-2209
                 URL: https://issues.apache.org/jira/browse/HDFS-2209
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: test
    Affects Versions:
            Reporter: Steve Loughran
            Assignee: Steve Loughran
            Priority: Minor

I've been deploying MiniDFSCluster for some testing, and while using it/looking through the
code I made some notes of where there are issues and improvement opportunities. This is mostly
minor as its a test tool, but a risk of synchronization problems is there and does need addressing;
the rest are all feature creep. 

Field {{nameNode}} should be marked as volatile as the shutdown operation can be in a different
thread than startup. Best of all, 
add synchronized methods to set and get the field, as well as shutdown.

The data dir is set from from System Properties.
    base_dir = new File(System.getProperty("test.build.data", "build/test/data"), "dfs/");
    data_dir = new File(base_dir, "data");
This is done in {{formatDataNodeDirs()}} {{corruptBlockOnDataNode()}} and  the constructor.

Improvement: have a test property in the conf file, and only read the system property if this
is unset. This will enable
 multiple MiniDFSClusters to come up in the same JVM, and handle shutdown/startup race conditions
better, and avoid the
 "java.io.IOException: Cannot lock storage build/test/data/dfs/name1. The directory is already
locked." messages

Messages should log to the commons logging and not {{System.err}} and {{System.out}}. This
enables containers to catch and stream better, 
and include more diagnostics such as timestamp and thread Id

Class could benefit from a method to return the FS URI, rather than just the FS. This currently
has to be worked around with some tricks involving a cached configuration

{{waitActive()}} could get confused if "localhost" maps to an IPv6 address. Better to ask
for as the hostname; Junit
test runs may need to be set up to force in IPv4 too.

{{injectBlocks}} has a spelling error in the IOException, "SumulatedFSDataset" is the correct

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message