hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline
Date Wed, 03 Mar 2010 09:49:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840555#action_12840555
] 

Hadoop QA commented on HDFS-826:
--------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12437700/Replicable4.txt
  against trunk revision 916902.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit
warnings.

    +1 core tests.  The patch passed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/console

This message is automatically generated.

> Allow a mechanism for an application to detect that datanode(s)  have died in the write
pipeline
> ------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-826
>                 URL: https://issues.apache.org/jira/browse/HDFS-826
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: Replicable4.txt, ReplicableHdfs.txt, ReplicableHdfs2.txt, ReplicableHdfs3.txt
>
>
> HDFS does not replicate the last block of the file that is being currently written to
by an application. Every datanode death in the write pipeline decreases the reliability of
the last block of the currently-being-written block. This situation can be improved if the
application can be notified of a datanode death in the write pipeline. Then, the application
can decide what is the right course of action to be taken on this event.
> In our use-case, the application can close the file on the first datanode death, and
start writing to a newly created file. This ensures that the reliability guarantee of a block
is close to 3 at all time.
> One idea is to make DFSOutoutStream. write() throw an exception if the number of datanodes
in the write pipeline fall below minimum.replication.factor that is set on the client (this
is backward compatible).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message