hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8059) Erasure coding: revisit how to store EC schema and cellSize in NameNode
Date Fri, 31 Jul 2015 02:18:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648644#comment-14648644
] 

Haohui Mai commented on HDFS-8059:
----------------------------------

Sorry for the late reply.

bq. Well, we keep replication in the namespace and use it in the block layer; how is this
any different?

I'm surprised you think that way. There are long efforts on separating the block manager out
of the NN due to scalability concerns (which can be dated back since HDFS-2106). While we're
not there yet but it is harmful to make new design choices that are contracted with the basic
principle.

bq. I assume you will solve the replication issue somehow, and the same solution should work
for EC schema.

Just quickly skim through the patch in this jira I think the current solution looks reasonable.
For what needs to be done in trunk I'm putting up a  patch in  HDFS-8823.

> Erasure coding: revisit how to store EC schema and cellSize in NameNode
> -----------------------------------------------------------------------
>
>                 Key: HDFS-8059
>                 URL: https://issues.apache.org/jira/browse/HDFS-8059
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7285
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>         Attachments: HDFS-8059.001.patch
>
>
> Move {{dataBlockNum}} and {{parityBlockNum}} from BlockInfoStriped to INodeFile, and
store them in {{FileWithStripedBlocksFeature}}.
> Ideally these two nums are the same for all striped blocks in a file, and store them
in BlockInfoStriped will waste NN memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message