hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manoj Govindassamy (JIRA)" <j...@apache.org>
Subject [jira] (HDFS-11382) Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage
Date Tue, 31 Jan 2017 18:58:51 GMT
Manoj Govindassamy created HDFS-11382:

             Summary: Persist Erasure Coding Policy ID in a new optional field in INodeFile
in FSImage
                 Key: HDFS-11382
                 URL: https://issues.apache.org/jira/browse/HDFS-11382
             Project: Hadoop HDFS
          Issue Type: Task
          Components: hdfs
    Affects Versions: 3.0.0-alpha1
            Reporter: Manoj Govindassamy
            Assignee: Manoj Govindassamy

For Erasure Coded files, replication field in INodeFile message is re-used  to store the EC
Policy ID. 

  private static INodesInPath addFile(
      FSDirectory fsd, INodesInPath existing, byte[] localName,
      PermissionStatus permissions, short replication, long preferredBlockSize,
      String clientName, String clientMachine)
      throws IOException {
    .. .. ..
    try {
      ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp.
          getErasureCodingPolicy(fsd.getFSNamesystem(), existing);
      if (ecPolicy != null) {
        replication = ecPolicy.getId();   <===
      final BlockType blockType = ecPolicy != null?
          BlockType.STRIPED : BlockType.CONTIGUOUS;
      INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions,
          modTime, modTime, replication, preferredBlockSize, blockType);
      newNode.toUnderConstruction(clientName, clientMachine);
      newiip = fsd.addINode(existing, newNode, permissions.getPermission());

With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly getting the
EC ID from the replication field and then uses the right Policy to construct the blocks.
      ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ?
          ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) :

The original intention was to re-use the replication field so the in-memory representation
would be compact. But, this isn't necessary for the on-disk representation. replication is
an optional field, and if we add another optional field for the EC policy, it won't be any
extra space.

Also, we need to make sure to have the appropriate asserts in place to make sure both fields
aren't set for the same INodeField.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message