hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11887) Introduce Intel ISA-L erasure coding library for the native support
Date Tue, 14 Jul 2015 08:29:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14626033#comment-14626033
] 

Kai Zheng commented on HADOOP-11887:
------------------------------------

Hi [~cmccabe], 
I have some understanding about the following two major questions and need to discuss with
you further.
bq. I can see you're using -Drequire.erasurecode, -Derasurecode.prefix, etc. for the build
parameters. This might create confusion, though. Users might ask if erasure encoding is disabled
if they build without the -Derasurecode... options. Given that the name of the library is
ISA-L, maybe we can have -Drequire.ISAL, -DISAL.prefix, etc.? Seems more straightforward.
I totally got your concern. I'm using the general name {{erasurecode}} instead of {{isal}}
directly because I wish the overall work won't couple with ISA-L too tightly. In future other
native libraries like Jerasure or even hardware based ones could also be supported as well
without too much change. I'm thinking that the native APIs defined in {{erasure_code.h}} should
be general enough so other native libraries could also be easily mapped to it, thus when building,
other libraries could also be passed to via the mentioned options. Note {{require.erasurecode}}
is used to enable it, and if enabled, {{erasurecode.prefix}} should be specified to provide
the library place; If not enabled (by default for now), the building should go as normally,
and the result won't contain any erasure code related symbols. The logic is similar to existing
codes like for snappy library.
bq. I think we should avoid adding a new test program for this, and instead add this functionality
to checknative.
You found another place I need to change. Yes I need to add an entry for erasrue code in the
tool. The question here is, I'm wondering if it can serve the purpose of the new tool here,
because executing of {{hadoop checknative}} may need some configuration or tweak to make it
work, the new tool can directly run just after it's out, so can be used in maven unit tests
cleanly. I understand introducing a new tool just for ONE native test may be too heavy, if
you agree, maybe we could go simple, say, if no native library is available, the native test
program could just exit with a warning message? Do we need more native tests anyway in future?
If so the checking with the new tool may sound more reasonable.

Would you let know your thoughts? Thanks again.

> Introduce Intel ISA-L erasure coding library for the native support
> -------------------------------------------------------------------
>
>                 Key: HADOOP-11887
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11887
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HADOOP-11887-v1.patch, HADOOP-11887-v2.patch, HADOOP-11887-v3.patch,
HADOOP-11887-v4.patch
>
>
> This is to introduce Intel ISA-L erasure coding library for the native support, via dynamic
loading mechanism (dynamic module, like *.so in *nix and *.dll on Windows).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message