hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8481) Erasure coding: remove workarounds in client side stripped blocks recovering
Date Sat, 30 May 2015 00:26:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565708#comment-14565708

Kai Zheng commented on HDFS-8481:

bq. it is beneficial to accumulate multiple of them before sending to decode. Kai Zheng Could
probably suggest a threshold size.
In pure coder's point of view, yes it's good to have larger cell size. It's not clear yet
in this case because the bottleneck might not be in the computation, instead in network traffic
and data copying stuffs? My suggestion would be, if the accumulation is already available
then we could have a default threshold value like 4MB but allowing it to be configurable in
future; otherwise leave the accumulation optimization for future consideration at all. I would
prefer not to do the accumulation in coder caller layer because it's hard. If it's good to
have then we may do it in coder layer in one place, like having a {{BufferedRawErasureCoder}}
layered on existing raw coders, transparent to callers.

> Erasure coding: remove workarounds in client side stripped blocks recovering
> ----------------------------------------------------------------------------
>                 Key: HDFS-8481
>                 URL: https://issues.apache.org/jira/browse/HDFS-8481
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8481-HDFS-7285.00.patch, HDFS-8481-HDFS-7285.01.patch, HDFS-8481-HDFS-7285.02.patch,
HDFS-8481-HDFS-7285.03.patch, HDFS-8481-HDFS-7285.04.patch
> After HADOOP-11847 and related fixes, we should be able to properly calculate decoded

This message was sent by Atlassian JIRA

View raw message