hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
Date Mon, 14 Sep 2015 22:58:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744443#comment-14744443

Zhe Zhang commented on HDFS-9040:

bq. The reason I prefer not to do locateFollowingBlock in DFSOutputStream is, DFSOutputStream
is async with DataStreamer
bq. Yeah, I can understand your concern. In the replication mechanism, the async implementation
matches the single write pipeline model, and the datastreamer can handle its failure perfectly.
But with 9 streamers in parallel, we need to 1) sync all the streamers when writing a new
block, and 2) stop all the streamers and assign them with new GS when failure happens. Thus
I think we'd better add some sync code in DFSStripedOutputStream. Also in this way it becomes
easier to calculate block length and set/reset external error state.
Very good discussion here. Jing's patch leaves the behavior of non-EC {{DFSOutputStream}}
and {{DataStreamer}} unchanged: the streamer is still in charge of locating following blocks.
I think we should probably change that as well so that {{OutputStream}} and streamer have
consistent roles under both contiguous and striped layouts.

bq. Currently the fastest streamer also has to wait for other streamers before requesting
a following block group from NN, so I think we may not feel the writing speed becomes slow.
Considering the buffer in {{DFSOutputStream}}, the above is only partially true. Performance-wise
it still makes sense to decouple {{locateFollowingBlock}} from the main {{DFSOutputStream}}
thread. How about starting a separate thread to allocate new block?

> Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
> -------------------------------------------------------------------------------------------
>                 Key: HDFS-9040
>                 URL: https://issues.apache.org/jira/browse/HDFS-9040
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>         Attachments: HDFS-9040.00.patch, HDFS-9040.001.wip.patch, HDFS-9040.02.bgstreamer.patch
> The general idea is to simplify error handling logic.
> Proposal 1:
> A BlockGroupDataStreamer to communicate with NN to allocate/update block, and StripedDataStreamer
s only have to stream blocks to DNs.
> Proposal 2:
> See below the [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388]
from [~jingzhao].

This message was sent by Atlassian JIRA

View raw message