hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
Date Tue, 16 Feb 2016 05:54:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148127#comment-15148127
] 

Kai Zheng commented on HDFS-9603:
---------------------------------

Thanks for the discussion, [~zhz] and [~kaisasak]!
bq. Could you help clarify if Hitchhiker needs the ECBlockGroup? Or only LRC potentially needs
it?
Not sure if better to make the major change to re-implement HH in raw coder level. I'll look
at the algorithm and codes closely later, and check with Rui how to make the small fix for
the mentioned issue to use the HH coder with ECBlockGroup. I thought we'll need the high level
abstract ion eventually, not just LRC, but also for other advanced algorithms, though RS is
the right one we need for the near future.
bq. With LRC looking unlikely to be implemented before the release, I think we should focus
on ...
Sounds good to me. The priority of this issue and the aspect should be low. I guess it's not
bad if someone would work on  it, unless conflicted. Note ErasureCoder can be used and co-exist
meanwhile with raw coders.
One thing to add is, HDFS-8430 and the family, for striped file checksum.
bq. It is reasonable to keep using raw coders for now because hflush/hsync is important to
reliable operation to HDFS.
Is there anything overhead to use ErasureCoder for the support of hflush/hsync? I thought
they're not related. hflush/hsync support should not couple with any specific codec/coder.

> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---------------------------------------------------------------
>
>                 Key: HDFS-9603
>                 URL: https://issues.apache.org/jira/browse/HDFS-9603
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Rui Li
>            Assignee: Rui Li
>         Attachments: HDFS-9603.1.patch, HDFS-9603.2.patch, HDFS-9603.3.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block group.
Currently however, we directly use {{RawErasureCoder}} to do the work, e.g. in {{DFSStripedOutputStream}}.
This task attempts to encapsulate {{RawErasureCoder}} to comply with the design.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message