hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
Date Fri, 12 Feb 2016 00:30:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143778#comment-15143778
] 

Zhe Zhang commented on HDFS-9603:
---------------------------------

Thanks Rui for the work. With this JIRA and HDFS-8020, I think we should revisit the {{ECBlockGroup}}
=> {{ECBlock}} => {{ECChunk}}, and {{ErasureCodingStep}} abstractions. Also pinging
[~lewuathe] for the discussion.

IIUC the purpose of these abstractions (instead of directly using raw coders) is to support
more advanced algorithms such as LRC and Hitchhiker. [~drankye] Could you help clarify if
Hitchhiker needs the {{ECBlockGroup}}? Or only LRC potentially needs it? If HH don't need
it, I suggest we keep using raw coders for now. We should still make the change to create
different coders (RS vs. XOR) based on configuration.

Right now, without a clear need for the {{ECBlockGroup}} abstraction, we are basically creating
a {{ECBlockGroup}} from the input buffers ({{prepareBlkGroup}}), just for the purpose of taking
out the buffers again later ({{encode}}).

I'm also suggesting the above to prepare for the forthcoming release of EC (either 2.9 or
3.0). With LRC looking unlikely to be implemented before the release, I think we should focus
on:
# Supporting multiple EC policies
# hflush / hsync
# Fully integrate ISA-L coder, new Java coder, and HH coder. Ideally with more performance
tests on HH.

Thoughts?


> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---------------------------------------------------------------
>
>                 Key: HDFS-9603
>                 URL: https://issues.apache.org/jira/browse/HDFS-9603
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Rui Li
>            Assignee: Rui Li
>         Attachments: HDFS-9603.1.patch, HDFS-9603.2.patch, HDFS-9603.3.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block group.
Currently however, we directly use {{RawErasureCoder}} to do the work, e.g. in {{DFSStripedOutputStream}}.
This task attempts to encapsulate {{RawErasureCoder}} to comply with the design.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message