hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12918) NameNode fails to start after upgrade - Missing state in ECPolicy Proto
Date Wed, 13 Dec 2017 06:59:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288591#comment-16288591
] 

Xiao Chen edited comment on HDFS-12918 at 12/13/17 6:58 AM:
------------------------------------------------------------

We have an upgrade incompatible fix landed in 3.0 at e565b5277d5b890dad107fe85e295a3907e4bfc1.
The fix is necessary and it verifies the EC Policy state when loading FSImage. This issue
is nothing to do with the default value for the ECPolicyState field in the ErasureCodingPolicyProto.
While the ECPolicyState field is optional in ECPolocyProto message for over the wire communications,
but its mandatory in FSImage for the EC files. I hope the upgrade incompatible changes before
the 3.0 GA are ok. Please let me know if you have other thoughts. 


was (Author: manojg):
We have an upgrade incompatible fix landed in 3.0 at e565b5277d5b890dad107fe85e295a3907e4bfc1.
The fix is necessary and it verifies the EC Policy state when loading FSImage. This issue
is nothing to do with the default value for the ECPolicyState field in the ErasureCodingPolicyProto.
While the ECPolicyState field is optional in ECPolocyProto message for over the wire communications,
but its mandatory in FSImage for the EC files. I hope the upgrade incompatible changes before
the C6 GA are ok. Please let me know if you have other thoughts. 

> NameNode fails to start after upgrade - Missing state in ECPolicy Proto 
> ------------------------------------------------------------------------
>
>                 Key: HDFS-12918
>                 URL: https://issues.apache.org/jira/browse/HDFS-12918
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 3.0.0-beta1
>            Reporter: Zach Amsden
>            Assignee: Manoj Govindassamy
>            Priority: Critical
>
> According to documentation and code comments, the default setting for erasure coding
policy is disabled:
> /** Policy is disabled. It's policy default state. */
>  DISABLED(1),
> However, HDFS-12258 appears to have incorrectly set the policy state in the protobuf
to enabled:
> {code:java}
>  message ErasureCodingPolicyProto {
>     ooptional string name = 1;
>     optional ECSchemaProto schema = 2;
>     optional uint32 cellSize = 3;
>     required uint32 id = 4; // Actually a byte - only 8 bits used
>  + optional ErasureCodingPolicyState state = 5 [default = ENABLED];
>   }
> {code}
> This means the parameter can't actually be optional, it must always be included, and
existing serialized data without this optional field will be incorrectly interpreted as having
erasure coding enabled.
> This unnecessarily breaks compatibility and will require existing HDFS installations
that store metadata in protobufs to require reformatting.
> It looks like a simple mistake that was overlooked in code review.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message