hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2209) Make MiniDFS easier to embed in other apps
Date Sat, 08 Oct 2011 13:16:30 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123483#comment-13123483

Hudson commented on HDFS-2209:

Integrated in Hadoop-Mapreduce-0.23-Build #40 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/40/])
    HDFS-2209 datanode connection failure logging
HDFS-2209. Make MiniDFS easier to embed in other apps.

stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180354
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java

stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180078
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMiniDFSCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDeleteBlockPool.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java

> Make MiniDFS easier to embed in other apps
> ------------------------------------------
>                 Key: HDFS-2209
>                 URL: https://issues.apache.org/jira/browse/HDFS-2209
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: test
>    Affects Versions:
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>             Fix For: 0.23.0, 0.24.0
>         Attachments: HDFS-2209.patch, HDFS-2209.patch, HDFS-2209.patch, HDFS-2209.patch,
>   Original Estimate: 1h
>          Time Spent: 1.5h
>  Remaining Estimate: 2h
> I've been deploying MiniDFSCluster for some testing, and while using it/looking through
the code I made some notes of where there are issues and improvement opportunities. This is
mostly minor as its a test tool, but a risk of synchronization problems is there and does
need addressing; the rest are all feature creep. 
> Field {{nameNode}} should be marked as volatile as the shutdown operation can be in a
different thread than startup. Best of all, 
> add synchronized methods to set and get the field, as well as shutdown.
> The data dir is set from from System Properties.
> {code}
>     base_dir = new File(System.getProperty("test.build.data", "build/test/data"), "dfs/");
>     data_dir = new File(base_dir, "data");
> {code}
> This is done in {{formatDataNodeDirs()}} {{corruptBlockOnDataNode()}} and  the constructor.
> Improvement: have a test property in the conf file, and only read the system property
if this is unset. This will enable
>  multiple MiniDFSClusters to come up in the same JVM, and handle shutdown/startup race
conditions better, and avoid the
>  "java.io.IOException: Cannot lock storage build/test/data/dfs/name1. The directory is
already locked." messages
> Messages should log to the commons logging and not {{System.err}} and {{System.out}}.
This enables containers to catch and stream better, 
> and include more diagnostics such as timestamp and thread Id
> Class could benefit from a method to return the FS URI, rather than just the FS. This
currently has to be worked around with some tricks involving a cached configuration
> {{waitActive()}} could get confused if "localhost" maps to an IPv6 address. Better to
ask for as the hostname; Junit
> test runs may need to be set up to force in IPv4 too.
> {{injectBlocks}} has a spelling error in the IOException, "SumulatedFSDataset" is the
correct spelling

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message