hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-8347) Erasure Coding: whether to use the same chunkSize in decoding with the value in encoding
Date Tue, 12 May 2015 12:18:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539446#comment-14539446
] 

Yi Liu edited comment on HDFS-8347 at 5/12/15 12:17 PM:
--------------------------------------------------------

yes, the {{chunkSize}} is hardcode in future, and for decoding, {{cellSize}} is needed.  I
agree to return cellSize in HdfsFileStatus for client side,  for striped blocks reconstuction
in data, I will   send it together with striped block reconstruction command in my follow-on
JIRA for striped block reconstruction.


was (Author: hitliuyi):
yes, the {{chunkSize}} is hardcode in future, and for decoding, {{cellSize}} is needed.  I
agree to return cellSize in HdfsFileStatus for client side,  for striped blocks reconstuction
in data, I will   send it together with striped block reconstruction command in my follow-on
JIRA for striped block reconstruction.
 If the coder is linear, we can decode at any size, and we don't need to padding inputs to
chunksize, if the coder is not linear, the inputs need to padding to chunksize, then do decode.
I create HADOOP-11961 to add {{isLinear}} interface to erasure codec, and HDFS-8376 to update
calculation for last cell size.

> Erasure Coding: whether to use the same chunkSize in decoding with the value in encoding
> ----------------------------------------------------------------------------------------
>
>                 Key: HDFS-8347
>                 URL: https://issues.apache.org/jira/browse/HDFS-8347
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>
> Currently decode buffersize for Datanode striped block reconstruction is configurable
and can be less or larger than chunksize, it may cause issue for Hitchhiker which may require
encode/decode using same buffersize.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message