hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6867) For DFSOutputStream, do pipeline recovery for a single block in the background
Date Fri, 12 Sep 2014 21:17:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132121#comment-14132121

Zhe Zhang commented on HDFS-6867:

[~cmccabe] Thanks for the suggestion. I think Future is a good idea. To further simplify the
design I have made {{RecoveryWorker}} more _stateless_ -- instead of keep running until full
strength replication is built, it finishes as soon as at least one new DN is found. Then {{DataStreamer}}
tries to recover the pipeline, and fires another {{RecoveryWorker}} if the strength is still
not full.

bq. I think to do this right, we'll need to modify an existing NameNode RPC like updatePipeline,
or perhaps add a new RPC, that just gives us a new DataNode suitable for our pipeline, but
does not change the state of the block on the NN. Basically, this means breaking updatePipeline
into two separate calls... one to find a good node (that RecoveryWorker will call), and *another
to actually update the block to be in RBR (that DataStreamer will call)*.
In the second call (highlighted above) the NN needs to recalculate the good DN to use right?
Since the mission of the {{RecoveryWorker}} is to provide hint to the {{DataStreamer}} suggesting
good timings to trigger the real recovery process, I think we do need a new RPC, which is
a lightweight _peek_ call to indicate (with reasonable accuracy) whether the recovery will
succeed. E.g., the NN can quickly verify that the number of live DNs is least larger than
{{nodes.length}} in the current pipeline. 

> For DFSOutputStream, do pipeline recovery for a single block in the background
> ------------------------------------------------------------------------------
>                 Key: HDFS-6867
>                 URL: https://issues.apache.org/jira/browse/HDFS-6867
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>            Reporter: Colin Patrick McCabe
>            Assignee: Zhe Zhang
>         Attachments: HDFS-6867-20140827-2.patch, HDFS-6867-20140827-3.patch, HDFS-6867-20140827.patch,
HDFS-6867-20140828-1.patch, HDFS-6867-20140828-2.patch, HDFS-6867-20140910.patch, HDFS-6867-20140911.patch,
HDFS-6867-design-20140820.pdf, HDFS-6867-design-20140821.pdf, HDFS-6867-design-20140822.pdf,
HDFS-6867-design-20140827.pdf, HDFS-6867-design-20140910.pdf
> For DFSOutputStream, we should be able to do pipeline recovery in the background, while
the user is continuing to write to the file.  This is especially useful for long-lived clients
that write to an HDFS file slowly. 

This message was sent by Atlassian JIRA

View raw message