hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9040) Erasure coding: A BlockGroupDataStreamer to rule all internal blocks streamers
Date Thu, 10 Sep 2015 13:25:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738730#comment-14738730
] 

Walter Su commented on HDFS-9040:
---------------------------------

bq. The role of BlockGroupDataStreamer looks similar (or closely related) to Coordinator.
Should we make it part of Coordinator?
it's not like that. The old way requires swtiching streamers. With BGDataStreamer, the data
flow becomes
DFSStripedOutputStream --> BGDataStreamer --> StripedDataStreamer(s)
bq. I think a challenge is whether / how to reuse the existing DFSOutputStream / DataStreamer
code. It doesn't seem easy to avoid refactoring them.
bq. Implementing a block group writer is a better solution but you have to copy a lot of existing
code to the new class, which may be hard to be...
Thanks for sharing the information. I'll try my best to avoid that. I'm still working on this.
Will update a demo soon.

> Erasure coding: A BlockGroupDataStreamer to rule all internal blocks streamers
> ------------------------------------------------------------------------------
>
>                 Key: HDFS-9040
>                 URL: https://issues.apache.org/jira/browse/HDFS-9040
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Walter Su
>
> A {{BlockGroupDataStreamer}} to communicate with NN to allocate/update block, and {{StripedDataStreamer}}
s only have to stream blocks to DNs. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message