hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "GAO Rui (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7864) Erasure Coding: Update safemode calculation for striped blocks
Date Thu, 05 Mar 2015 06:46:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348258#comment-14348258
] 

GAO Rui commented on HDFS-7864:
-------------------------------

[~jingzhao]]
{quote}
The current code's calculation is thus inconsistent: each striped block is only counted as
1 expected block, while each of its member block may increase the number of received blocks
by 1.
{quote}
In above description, it looks like that DataNode will send information of both StripedBlock
and it's member blocks in re. I have checked BlockManager.java, BlockInfoStriped.java and
other related source codes. 
In BlockReport, DataNode send StorageBlockReport[ ] to NameNode. StorageBlockReport[ ] contains
Maps of <DataNodeStrotage, long[ ]> , this long[ ] array contains block Id (or StripedBlock
id) which are stored in DataNodeStrotage. The method getStoredBlock(Block block) in class
BlockManager.java can deal with StripedBlock. I am not sure wether DataNode will send information
of the StripedBlock's member blocks as well as the StripedBlock. And, how can a DataNode send
a report about a StripedBlock while only one of its member blocks could be stored in this
DataNode?  This question is based on the my image of EC logical process below:

 For example, a file (name:F, size: FS) is going to be stored in to HDFS with EC(striping
and Reed­Solomon (6,3) schema). 
If FS=128*6MB, F is truncated into 1024*6 slices(name: s1,s2,……s1024*6; each size is 128kb)
. So at last, F is stored in 6 date blocks (name: b1,b2…..b6) with 3 parity blocks(name:
pb1,pb2,pb3). And, b1 is consist of s1,s7,s13…….  ;  b2 is consist of s2,s8,s14……….
     So, in FSImage of HDFS, file F is a INodeFile with one BlockGroup, and this BlockGroup
is consist of b1,b2,b3,b4,b5,b6,pb1,pb2,pb3.
     Did I get the correct image of EC logical process? Please let me know if I have any misunderstanding.



> Erasure Coding: Update safemode calculation for striped blocks
> --------------------------------------------------------------
>
>                 Key: HDFS-7864
>                 URL: https://issues.apache.org/jira/browse/HDFS-7864
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: GAO Rui
>
> We need to update the safemode calculation for striped blocks. Specifically, each striped
block now consists of multiple data/parity blocks stored in corresponding DataNodes. The current
code's calculation is thus inconsistent: each striped block is only counted as 1 expected
block, while each of its member block may increase the number of received blocks by 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message