hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Bo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
Date Fri, 18 Sep 2015 05:39:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14805014#comment-14805014
] 

Li Bo commented on HDFS-9040:
-----------------------------

Thanks for the deep discussion here. One point confused me:
bq. Only based on internal block lengths we cannot identify the failure
For a block not belong to the last block group, if its length is smaller than BLOCK_SIZE,
we can conclude it’s corrupt, otherwise it is a good block, right? For the last block group,
we can calculate the length of each block via the file length, so if a block doesn’t satisfy
the required length, then we can conclude it’s corrupt. The precondition is, if <=NUM_PARITY
streamers fail, we ignore their failures and treat all blocks of this block group are written
correctly.

Each block in a block group only has one replica,  and if we can judge a block corrupt or
not, it may not necessary to bump the GS.

Any other points that require bumping the GS?

> Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-9040
>                 URL: https://issues.apache.org/jira/browse/HDFS-9040
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>         Attachments: HDFS-9040-HDFS-7285.002.patch, HDFS-9040-HDFS-7285.003.patch, HDFS-9040.00.patch,
HDFS-9040.001.wip.patch, HDFS-9040.02.bgstreamer.patch
>
>
> The general idea is to simplify error handling logic.
> Proposal 1:
> A BlockGroupDataStreamer to communicate with NN to allocate/update block, and StripedDataStreamer
s only have to stream blocks to DNs.
> Proposal 2:
> See below the [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388]
from [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message