hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zach Amsden (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12918) EC Policy defaults incorrectly to enabled in protobufs
Date Wed, 13 Dec 2017 00:45:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288537#comment-16288537
] 

Zach Amsden commented on HDFS-12918:
------------------------------------

Maybe that is the real bug then.  I got this exception when upgrading an existing HDFS cluster
- reformatting was required:


{noformat}
Failed to load image from FSImageFile(file=/data/2/dfs/nn/current/fsimage_0000000000008728887,
cpktTxId=0000000000008728887)
java.lang.IllegalArgumentException: Missing state field in ErasureCodingPolicy proto
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.convertErasureCodingPolicyInfo(PBHelperClient.java:2973)
	at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadErasureCodingSection(FSImageFormatProtobuf.java:386)
	at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:298)
	at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:188)
	at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:227)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:928)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:912)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:785)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:719)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:317)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1072)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:704)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
{noformat}


> EC Policy defaults incorrectly to enabled in protobufs
> ------------------------------------------------------
>
>                 Key: HDFS-12918
>                 URL: https://issues.apache.org/jira/browse/HDFS-12918
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Zach Amsden
>            Assignee: Manoj Govindassamy
>            Priority: Critical
>
> According to documentation and code comments, the default setting for erasure coding
policy is disabled:
> /** Policy is disabled. It's policy default state. */
>  DISABLED(1),
> However, HDFS-12258 appears to have incorrectly set the policy state in the protobuf
to enabled:
> {code:java}
>  message ErasureCodingPolicyProto {
>     ooptional string name = 1;
>     optional ECSchemaProto schema = 2;
>     optional uint32 cellSize = 3;
>     required uint32 id = 4; // Actually a byte - only 8 bits used
>  + optional ErasureCodingPolicyState state = 5 [default = ENABLED];
>   }
> {code}
> This means the parameter can't actually be optional, it must always be included, and
existing serialized data without this optional field will be incorrectly interpreted as having
erasure coding enabled.
> This unnecessarily breaks compatibility and will require existing HDFS installations
that store metadata in protobufs to require reformatting.
> It looks like a simple mistake that was overlooked in code review.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message