Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 5 Mar 2015 21:55:38 +0000 (UTC)
From: "Jing Zhao (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12778667.1425267748000.15212.1425592538871@Atlassian.JIRA>
In-Reply-To: <JIRA.12778667.1425267748000@Atlassian.JIRA>
References: <JIRA.12778667.1425267748000@Atlassian.JIRA>
 <JIRA.12778667.1425267748687@arcas>
Subject: [jira] [Commented] (HDFS-7864) Erasure Coding: Update safemode
 calculation for striped blocks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349509#comment-14349509 ] 

Jing Zhao commented on HDFS-7864:
---------------------------------

Hi [~demongaorui], thanks for the comments.

In our current implementation, DataNodes does not know any difference between a contiguous block and a block belonging to a striped blockgroup. For example, let's say a file consists of one striped block: b1 with block id 0x8000000000000000L. For a 6+3 encoding, b1 includes 6 data blocks and 3 parity blocks, with block id 0x8000000000000000L, 0x8000000000000001L, 0x8000000000000002L, ..., 0x8000000000000008L, accordingly. DataNodes then only see these data blocks and parity blocks, and the concept of striped block (which is actually a block group) is only known by the NameNode (and maybe client). In FSImage, for the striped block, we can still treat it as a normal block except we need to store its number of data/parity blocks.

This is mainly to save NameNode memory. More details are in "NameNode Memory Usage Reduction" section in the [design doc | https://issues.apache.org/jira/secure/attachment/12697210/HDFSErasureCodingDesign-20150206.pdf].

> Erasure Coding: Update safemode calculation for striped blocks
> --------------------------------------------------------------
>
>                 Key: HDFS-7864
>                 URL: https://issues.apache.org/jira/browse/HDFS-7864
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: GAO Rui
>
> We need to update the safemode calculation for striped blocks. Specifically, each striped block now consists of multiple data/parity blocks stored in corresponding DataNodes. The current code's calculation is thus inconsistent: each striped block is only counted as 1 expected block, while each of its member block may increase the number of received blocks by 1.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)