hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8254) In StripedDataStreamer, it is hard to tolerate datanode failure in the leading streamer
Date Wed, 17 Jun 2015 02:39:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589204#comment-14589204

Tsz Wo Nicholas Sze commented on HDFS-8254:

Thanks Zhe for the review.

>   ... Can we add some Javadocs for the class and methods, and ideally, a design description
on the JIRA?

Sure, will add the following javadoc.
   * This class is designed for multiple threads to share a
   * {@link MultipleBlockingQueue}. Initially, the queue is empty. The earliest
   * thread calling poll populates entries to the queue and the other threads
   * will wait for it. Once the entries are populated, all the threads can poll
   * their entries.
   * @param <T> the queue entry type.

> ... It also looks like we could run into a race condition ...

As Walter explained, there is no race condition.

> DFSStripedOutputStream already has the schema ...

The JIRA is for replacing leading streamer implementation.  For the other issues, let's fix
them in separated JIRAs.

> New methods in DataStreamer could use some Javadoc.

Do you mean callUpdateBlockForPipeline and callUpdatePipeline?  These internal APIs are both
short and straightforward.  I believe javadoc is not needed.

>  class Coordinator could be renamed to something like StreamersCoordinator, just to be
more specific.

The class actually is DFSStripedOutputStream.Coordinator.

> The Javadoc of StripedBlockUtil#checkBlocks should say that it checks the two blocks
are in the same block group

Will fix it.

> In StripedDataStreamer, it is hard to tolerate datanode failure in the leading streamer
> ---------------------------------------------------------------------------------------
>                 Key: HDFS-8254
>                 URL: https://issues.apache.org/jira/browse/HDFS-8254
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h8254_20150526.patch, h8254_20150526b.patch
> StripedDataStreamer javadoc is shown below.
> {code}
>  * The StripedDataStreamer class is used by {@link DFSStripedOutputStream}.
>  * There are two kinds of StripedDataStreamer, leading streamer and ordinary
>  * stream. Leading streamer requests a block group from NameNode, unwraps
>  * it to located blocks and transfers each located block to its corresponding
>  * ordinary streamer via a blocking queue.
> {code}
> Leading streamer is the streamer with index 0.  When the datanode of the leading streamer
fails, the other steamers cannot continue since no one will request a block group from NameNode

This message was sent by Atlassian JIRA

View raw message