hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7839) Erasure coding: implement facilities in NameNode to create and manage EC zones
Date Mon, 02 Mar 2015 18:13:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343517#comment-14343517

Zhe Zhang commented on HDFS-7839:

Thanks Kai! The updated description is more accurate.

bq. Is there a little bit conflict or overlap between this and HDFS-7349 ?
Good catch. I read Vinay's patch again and found it has PB implementation as well. I actually
planned to use this JIRA to support a simplest EC zone so we can have an initial PoC prototype.
HDFS-7349 investigates the most suitable set of commands. So maybe we can keep the simple
PB and NN RPC implementation under this JIRA; HDFS-7349 can extend it later.
bq. So would you add to include some hard coded values like NumDataBlocks, NumParityBlocks
and etc
Where do you suggest we add those? In the XAttr?
bq. For the new operation to create an EC zone, we need two parameters, selected EC schema
(from predefined list) and the target folder.
I planned to leave the schema parameter to HDFS-7866 but we can certainly add a place holder
(a short type) here
bq. Better to move the logic of isStrippedFile out of the manager into FSDirectory.
{{FSDirectory}} does have a {{isStrippedFile}} method. Per Andrew's suggestion, I'm consolidating
all actual XAttr operations in the manager class
bq. For consistency, ErasureCodingManager ==> ECZoneManager.
{{ErasureCodingManager}} was named following the convention of {{EncryptionZoneManager}}.

Addressed both minor points in the updated patch (will upload after soliciting more feedbacks).

I verified that nested encryption zones are not supported. We should just disallow it at this
stage as well. An encryption zone on root is allowed though. Maybe we should allow it here
as well?

> Erasure coding: implement facilities in NameNode to create and manage EC zones
> ------------------------------------------------------------------------------
>                 Key: HDFS-7839
>                 URL: https://issues.apache.org/jira/browse/HDFS-7839
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-7839-000.patch, HDFS-7839-001.patch
> As a quick first step to facilitate initial development and testing, HDFS-7347 added
EC configuration in file header as one storage policy. We have discussed and [concluded |https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14296210&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14296210]
that EC configurations should be part of XAttr. This JIRA aims to add the basic EC XAttr structure.
HDFS-7337 will add configurable and pluggable schema info.
> To summarize, this will focus on providing relevant facilities in NameNode side to create
and manage EC zones. EC zone will associate, reference and store the needed schema name as
XAttr. EC schema information will be persisted in NameNode elsewhere centrally.

This message was sent by Atlassian JIRA

View raw message