Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 7 Nov 2016 15:42:58 +0000 (UTC)
From: "Ewan Higgs (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12996973.1471015657000.210589.1478533378488@Atlassian.JIRA>
In-Reply-To: <JIRA.12996973.1471015657000@Atlassian.JIRA>
References: <JIRA.12996973.1471015657000@Atlassian.JIRA> <JIRA.12996973.1471015657079@arcas>
Subject: [jira] [Updated] (HDFS-10759) Change fsimage bool isStriped from
 boolean to an enum
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 07 Nov 2016 15:43:00 -0000


     [ https://issues.apache.org/jira/browse/HDFS-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ewan Higgs updated HDFS-10759:
------------------------------
    Attachment: HDFS-10759.0002.patch

Attached is HDFS-10759.0002.patch which will make this change backwards compatible to the existing {{boolean isStriped}} in the {{fsimage}}. This is done by making the {{BlockType}} enum work with {{CONTIGUOUS=0, STRIPED=1}} as this mirrors the boolean behaviour. 

As both the boolean and the enum are optional, they are usually left out of the wire protocol. If they are explicitly set then they will have the same semantic values on the wire now.

Given the following messages:
{code}
/**                                                                             
 * Types of recognized blocks.                                                 
 */                                                                            
enum BlockTypeProto {                                                           
    CONTIGUOUS = 0;                                                            
    STRIPED = 1;                                                               
}   
 
  message INodeFile {                                                          
    optional uint32 replication = 1;                                           
    optional uint64 modificationTime = 2;                                      
    optional uint64 accessTime = 3;                                            
    optional uint64 preferredBlockSize = 4;                                    
    optional fixed64 permission = 5;                                           
    repeated BlockProto blocks = 6;                                            
    optional FileUnderConstructionFeature fileUC = 7;                          
    optional AclFeatureProto acl = 8;                                           
    optional XAttrFeatureProto xAttrs = 9;                                     
    optional uint32 storagePolicyID = 10;                                      
    optional BlockTypeProto blockType = 11;                                     
  }                                                                            
                                                                                
  /* ehiggs - old style using bool blockType */                                 
  message INodeFileOld {                                                       
    optional uint32 replication = 1;                                           
    optional uint64 modificationTime = 2;                                      
    optional uint64 accessTime = 3;                                            
    optional uint64 preferredBlockSize = 4;                                    
    optional fixed64 permission = 5;                                           
    repeated BlockProto blocks = 6;                                            
    optional FileUnderConstructionFeature fileUC = 7;                          
    optional AclFeatureProto acl = 8;                                          
    optional XAttrFeatureProto xAttrs = 9;                                     
    optional uint32 storagePolicyID = 10;                                      
    optional bool isStriped = 11;                                              
  }  
{code}

We can then see that these serialise as the same values:

{code}
In [1]: import fsimage_pb2
 
In [2]: f_enum = fsimage_pb2.INodeSection.INodeFile()
 
In [3]: f_bool = fsimage_pb2.INodeSection.INodeFileOld()
 
In [4]: f_enum.SerializeToString() # Wire format of entirely optional message. Empty!
Out[4]: ''
 
In [5]: f_bool.SerializeToString() # With a bool, everything is still empty. No surprises.
Out[5]: ''
 
In [6]: f_enum.blockType = 0
 
In [7]: f_bool.isStriped = False
 
In [8]: f_enum.SerializeToString() # wire format of explicit BlockType.CONTIGUOUS
Out[8]: 'X\x00'
 
In [9]: f_bool.SerializeToString() # wire format of explicit false
Out[9]: 'X\x00'
 
In [10]: f_enum.blockType = 1 # Set explicity to STRIPED
 
In [11]: f_bool.isStriped = True # turn isStriped to True.
 
In [12]: f_enum.SerializeToString()
Out[12]: 'X\x01'
 
In [13]: f_bool.SerializeToString()
Out[13]: 'X\x01'
{code}

> Change fsimage bool isStriped from boolean to an enum
> -----------------------------------------------------
>
>                 Key: HDFS-10759
>                 URL: https://issues.apache.org/jira/browse/HDFS-10759
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 3.0.0-alpha1, 3.0.0-beta1, 3.0.0-alpha2
>            Reporter: Ewan Higgs
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-10759.0001.patch, HDFS-10759.0002.patch
>
>
> The new erasure coding project has updated the protocol for fsimage such that the {{INodeFile}} has a boolean '{{isStriped}}'. I think this is better as an enum or integer since a boolean precludes any future block types. 
> For example:
> {code}
> enum BlockType {
>   CONTIGUOUS = 0,
>   STRIPED = 1,
> }
> {code}
> We can also make this more robust to future changes where there are different block types supported in a staged rollout.  Here, we would use {{UNKNOWN_BLOCK_TYPE}} as the first value since this is the default value. See [here|http://androiddevblog.com/protocol-buffers-pitfall-adding-enum-values/] for more discussion.
> {code}
> enum BlockType {
>   UNKNOWN_BLOCK_TYPE = 0,
>   CONTIGUOUS = 1,
>   STRIPED = 2,
> }
> {code}
> But I'm not convinced this is necessary since there are other enums that don't use this approach.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org