Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 16 Jan 2015 19:43:35 +0000 (UTC)
From: "Zhe Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12752604.1415064353000.104486.1421437415455@Atlassian.JIRA>
In-Reply-To: <JIRA.12752604.1415064353000@Atlassian.JIRA>
References: <JIRA.12752604.1415064353000@Atlassian.JIRA>
 <JIRA.12752604.1415064353981@arcas>
Subject: [jira] [Updated] (HDFS-7339) Allocating and persisting block groups
 in NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HDFS-7339?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhe Zhang updated HDFS-7339:
----------------------------
    Attachment:     (was: HDFS-7339-003.patch)

> Allocating and persisting block groups in NameNode
> --------------------------------------------------
>
>                 Key: HDFS-7339
>                 URL: https://issues.apache.org/jira/browse/HDFS-7339
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-7339-001.patch, HDFS-7339-002.patch, Meta-strip=
ing.jpg, NN-stripping.jpg
>
>
> All erasure codec operations center around the concept of _block group_; =
they are formed in initial encoding and looked up in recoveries and convers=
ions. A lightweight class {{BlockGroup}} is created to record the original =
and parity blocks in a coding group, as well as a pointer to the codec sche=
ma (pluggable codec schemas will be supported in HDFS-7337). With the strip=
ing layout, the HDFS client needs to operate on all blocks in a {{BlockGrou=
p}} concurrently. Therefore we propose to extend a file=E2=80=99s inode to =
switch between _contiguous_ and _striping_ modes, with the current mode rec=
orded in a binary flag. An array of BlockGroups (or BlockGroup IDs) is adde=
d, which remains empty for =E2=80=9Ctraditional=E2=80=9D HDFS files with co=
ntiguous block layout.
> The NameNode creates and maintains {{BlockGroup}} instances through the n=
ew {{ECManager}} component; the attached figure has an illustration of the =
architecture. As a simple example, when a {_Striping+EC_} file is created a=
nd written to, it will serve requests from the client to allocate new {{Blo=
ckGroups}} and store them under the {{INodeFile}}. In the current phase, {{=
BlockGroups}} are allocated both in initial online encoding and in the conv=
ersion from replication to EC. {{ECManager}} also facilitates the lookup of=
 {{BlockGroup}} information for block recovery work.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)