hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library
Date Tue, 15 Mar 2016 23:20:33 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196461#comment-15196461

Kai Zheng commented on HADOOP-11540:

Hi [~cmccabe],

Thanks for being so quick for the review and the helpful comments!

* For doEncodeByConvertingToDirectBuffers, your point makes sense as we don't have all the
codes be presented here so far. I thought I should have placed the function into AbstractNativeRawEncoder,
because only native coders will need do that. We'll have another native coder for the XOR
codec, and it will extend AbstractNativeRawEncoder and need the function too. So how about
moving doEncodeByConvertingToDirectBuffers to AbstractNativeRawEncoder, similarly for doDecodeByConvertingToDirectBuffers?

* For  initOutputs and resetBuffer, good catch! About this I initially thought as you suggested,
instead of having initOutputs, just letting concrete coders to override resetBuffer, which
would be most flexible. Then I realized for Java coders, a default behavior can be provided
and used; for native coders, we can avoid having it because at the beginning of the encode()
call the native coder can {{memset}} the output buffers directly. If instead the native coder
has to provide resetBuffer, then a JNI function has to be added, which will be called some
times to reset output buffers. Considering the overhead in both implementation and extra JNI
calls, I used the initOutputs() approach.

As I'm not sure about how would you like this clarifying, please kindly let me know if this
works for you or not. Thanks.

> Raw Reed-Solomon coder using Intel ISA-L library
> ------------------------------------------------
>                 Key: HADOOP-11540
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11540
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7285
>            Reporter: Zhe Zhang
>            Assignee: Kai Zheng
>         Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, HADOOP-11540-v2.patch,
HADOOP-11540-v4.patch, HADOOP-11540-v5.patch, HADOOP-11540-with-11996-codes.patch, Native
Erasure Coder Performance - Intel ISAL-v1.pdf
> This is to provide RS codec implementation using Intel ISA-L library for encoding and

This message was sent by Atlassian JIRA

View raw message