hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7864) Erasure Coding: Update safemode calculation for striped blocks
Date Thu, 05 Mar 2015 21:55:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349509#comment-14349509

Jing Zhao commented on HDFS-7864:

Hi [~demongaorui], thanks for the comments.

In our current implementation, DataNodes does not know any difference between a contiguous
block and a block belonging to a striped blockgroup. For example, let's say a file consists
of one striped block: b1 with block id 0x8000000000000000L. For a 6+3 encoding, b1 includes
6 data blocks and 3 parity blocks, with block id 0x8000000000000000L, 0x8000000000000001L,
0x8000000000000002L, ..., 0x8000000000000008L, accordingly. DataNodes then only see these
data blocks and parity blocks, and the concept of striped block (which is actually a block
group) is only known by the NameNode (and maybe client). In FSImage, for the striped block,
we can still treat it as a normal block except we need to store its number of data/parity

This is mainly to save NameNode memory. More details are in "NameNode Memory Usage Reduction"
section in the [design doc | https://issues.apache.org/jira/secure/attachment/12697210/HDFSErasureCodingDesign-20150206.pdf].

> Erasure Coding: Update safemode calculation for striped blocks
> --------------------------------------------------------------
>                 Key: HDFS-7864
>                 URL: https://issues.apache.org/jira/browse/HDFS-7864
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: GAO Rui
> We need to update the safemode calculation for striped blocks. Specifically, each striped
block now consists of multiple data/parity blocks stored in corresponding DataNodes. The current
code's calculation is thus inconsistent: each striped block is only counted as 1 expected
block, while each of its member block may increase the number of received blocks by 1.

This message was sent by Atlassian JIRA

View raw message