Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Wed, 30 Sep 2015 15:58:16 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12752604.1415064353000.114167.1443628696354@Atlassian.JIRA>
In-Reply-To: <JIRA.12752604.1415064353000@Atlassian.JIRA>
References: <JIRA.12752604.1415064353000@Atlassian.JIRA>
 <JIRA.12752604.1415064353981@arcas>
Subject: [jira] [Commented] (HDFS-7339) Representing striped block groups in
 NameNode with hierarchical naming protocol
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HDFS-7339?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14937=
062#comment-14937062 ]=20

Hudson commented on HDFS-7339:
------------------------------

FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See [https://builds.apache.=
org/job/Hadoop-Hdfs-trunk/2379/])
HDFS-7339. Allocating and persisting block groups in NameNode. (zhezhang: r=
ev bc2833b1c91e107d090619d755c584f6eae82327)
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/serv=
er/namenode/INodeFile.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/serv=
er/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/serv=
er/blockmanagement/SequentialBlockGroupIdGenerator.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/serv=
er/blockmanagement/BlockIdManager.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSC=
onfigKeys.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/serv=
er/namenode/TestAddBlockgroup.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/serv=
er/blockmanagement/SequentialBlockIdGenerator.java


> Representing striped block groups in NameNode with hierarchical naming pr=
otocol
> -------------------------------------------------------------------------=
------
>
>                 Key: HDFS-7339
>                 URL: https://issues.apache.org/jira/browse/HDFS-7339
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-7339-001.patch, HDFS-7339-002.patch, HDFS-7339-=
003.patch, HDFS-7339-004.patch, HDFS-7339-005.patch, HDFS-7339-006.patch, H=
DFS-7339-007.patch, HDFS-7339-008.patch, Meta-striping.jpg, NN-stripping.jp=
g
>
>
> All erasure codec operations center around the concept of _block group_; =
they are formed in initial encoding and looked up in recoveries and convers=
ions. A lightweight class {{BlockGroup}} is created to record the original =
and parity blocks in a coding group, as well as a pointer to the codec sche=
ma (pluggable codec schemas will be supported in HDFS-7337). With the strip=
ing layout, the HDFS client needs to operate on all blocks in a {{BlockGrou=
p}} concurrently. Therefore we propose to extend a file=E2=80=99s inode to =
switch between _contiguous_ and _striping_ modes, with the current mode rec=
orded in a binary flag. An array of BlockGroups (or BlockGroup IDs) is adde=
d, which remains empty for =E2=80=9Ctraditional=E2=80=9D HDFS files with co=
ntiguous block layout.
> The NameNode creates and maintains {{BlockGroup}} instances through the n=
ew {{ECManager}} component; the attached figure has an illustration of the =
architecture. As a simple example, when a {_Striping+EC_} file is created a=
nd written to, it will serve requests from the client to allocate new {{Blo=
ckGroups}} and store them under the {{INodeFile}}. In the current phase, {{=
BlockGroups}} are allocated both in initial online encoding and in the conv=
ersion from replication to EC. {{ECManager}} also facilitates the lookup of=
 {{BlockGroup}} information for block recovery work.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)