hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8481) Erasure coding: remove workarounds in client side stripped blocks recovering
Date Sat, 30 May 2015 00:10:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565691#comment-14565691
] 

Kai Zheng commented on HDFS-8481:
---------------------------------

Good discussion here, thanks!
bq. In that case we cannot reuse the source buffers I guess? Then do we need to expose this
information in the decoder?
Good catch Jing! Yes in this case we can't reuse the source buffers here as they need to be
passed to caller/applications without being changed. I'm planning to re-implement the Java
coders in HADOOP-12041 and related, when done it's possible to ensure the input buffers not
to be affected. Benefits of doing this in coder layer: 1) a more clear contract between coder
and caller in more general sense for the inputs; 2) concrete coder may have specific tweak
to optimize in the aspect, ideally no input data copying at all, worst, make the copy, but
all transparent to callers; 3) allow new coders (LRC, HH) to be layered on other primitive
coders (RS, XOR) more easily. So for now let's forget the source buffers reusing here and
we can do it in future, but do it for output buffers now if easy?

> Erasure coding: remove workarounds in client side stripped blocks recovering
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-8481
>                 URL: https://issues.apache.org/jira/browse/HDFS-8481
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8481-HDFS-7285.00.patch, HDFS-8481-HDFS-7285.01.patch, HDFS-8481-HDFS-7285.02.patch,
HDFS-8481-HDFS-7285.03.patch
>
>
> After HADOOP-11847 and related fixes, we should be able to properly calculate decoded
contents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message