hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "SammiChen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11152) Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors
Date Thu, 19 Jan 2017 10:23:26 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829676#comment-15829676

SammiChen commented on HDFS-11152:

Hi [~andrew.wang]] I did some investigation. It seems EC policy ID is not widely used.  For
1. For directory, EC policy is saved as XAttr using policy name(policy name doesn't include
policy ID). When directory is saved to FsImage, all its XAttrs are saved including EC policy
2. For file, EC policy ID is saved in head. When file is saved to FsImage, EC policy ID is
saved in replication field. 
3. The {{setErasureCodingPolicy}} API, user will provide a {{ErasureCodingPolicy}} parameter,
currently only the schema and cell size is used to check if it's a legitimate EC policy, policy
ID is not involved in parameter checking. Because currently {{setErasureCodingPolicy}} can
only applies to directory. And only policy name is saved on directory. I will enforce the
policy ID check  in HDFS-10996 implementation which specify per-file policy when create file.

4. File read/write.  Double checked {{DFSStripedInputStream}} and {{DFSStripedOutputStream}},
policy ID is not used at all. The schema and cellsize are used. 

So, later I plan to upload a new patch, adding Precondition checks on FsImage file loading
only.  Because other spaces use policy name, schema and cell size. 

> Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected
> ---------------------------------------------------------------------------------------------
>                 Key: HDFS-11152
>                 URL: https://issues.apache.org/jira/browse/HDFS-11152
>             Project: Hadoop HDFS
>          Issue Type: Task
>    Affects Versions: 3.0.0-alpha1
>            Reporter: SammiChen
>            Assignee: SammiChen
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-11152-v1.patch
> This task will change erasure coding policy ID number starting from 1 instead of current
0, to avoid some potential unexpected errors in codes since 0 is default value for integer

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message