hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
Date Mon, 16 Mar 2015 22:31:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364100#comment-14364100
] 

Jing Zhao edited comment on HDFS-7827 at 3/16/15 10:31 PM:
-----------------------------------------------------------

Currently we still have non-protobuf image for offline processing. Thus this change can still
be helpful especially for storage usage analysis.

Some comments on the current patch:
# After updating {{writeINodeUnderConstruction}}, we also need to update {{readINodeUnderConstruction}}
accordingly.
# We also need a unit test to cover the under construction file with striped blocks.
# Instead of the following code, we can simply use "out.writeBoolean(cons.isWithStripedBlocks())".
Whether we keep using "EC_STORAGE_POLICY_ID" is still under discussion.
{code}
+    if (cons.isStriped()){
+      out.writeByte(HdfsConstants.EC_STORAGE_POLICY_ID);
+    }else{
+      out.writeByte(0);
+    }
{code}
# Please correct the coding format according to the project conventions.

Besides, as [~wheat9] pointed out, we also need to update WebImageViewer for file size calculation.
Please open a separate jira to do this.


was (Author: jingzhao):
Currently we still have non-protobuf image for offline processing. Thus this change can still
be helpful especially for storage usage analysis.

Some comments on the current patch:
# After updating {{writeINodeUnderConstruction}}, we also need to update {{readINodeUnderConstruction}}
accordingly.
# We also need a unit test to cover the under construction file with striped blocks.
# Instead of the following code, we can simply use "out.writeBoolean(cons.isWithStripedBlocks())".
Whether we keep using "EC_STORAGE_POLICY_ID" is still under construction.
{code}
+    if (cons.isStriped()){
+      out.writeByte(HdfsConstants.EC_STORAGE_POLICY_ID);
+    }else{
+      out.writeByte(0);
+    }
{code}
# Please correct the coding format according to the project conventions.

Besides, as [~wheat9] pointed out, we also need to update WebImageViewer for file size calculation.
Please open a separate jira to do this.

> Erasure Coding: support striped blocks in non-protobuf fsimage
> --------------------------------------------------------------
>
>                 Key: HDFS-7827
>                 URL: https://issues.apache.org/jira/browse/HDFS-7827
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Hui Zheng
>         Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch, HDFS-7827.002.patch
>
>
> HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should
also add this support to the non-protobuf fsimage since it is still used for use cases like
offline image processing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message