hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
Date Mon, 28 Sep 2015 21:13:09 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934036#comment-14934036
] 

Jing Zhao commented on HDFS-9040:
---------------------------------

Thanks for the review, Zhe! For {{getFollowingBlock}}, currently we have:
# The {{DFSStripedOutputStream}} fetches the new block from NN when it receives the first
chunk of data for the new block. This is before we enqueue the first packet for the new block.
# The data streamer does not call {{getFollowingBlock}} until its data queue is no longer
empty.

Thus I think when a data streamer calls {{getFollowingBlock}}, the new block should already
be ready in the queue. Therefore {{poll}} here should be safe. Besides, if we make a mistake
here, poll can give us a NPE so it may be easier for debug.

I will update the patch to address your comments about {{callUpdatePipeline}} and {{updatePipelineInternal}}.

> Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-9040
>                 URL: https://issues.apache.org/jira/browse/HDFS-9040
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Jing Zhao
>         Attachments: HDFS-9040-HDFS-7285.002.patch, HDFS-9040-HDFS-7285.003.patch, HDFS-9040-HDFS-7285.004.patch,
HDFS-9040-HDFS-7285.005.patch, HDFS-9040-HDFS-7285.005.patch, HDFS-9040-HDFS-7285.006.patch,
HDFS-9040.00.patch, HDFS-9040.001.wip.patch, HDFS-9040.02.bgstreamer.patch
>
>
> The general idea is to simplify error handling logic.
> -Proposal 1:-
> -A BlockGroupDataStreamer to communicate with NN to allocate/update block, and StripedDataStreamer
s only have to stream blocks to DNs.-
> Proposal 2:
> See below the [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388]
from [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message