hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "luhuichun (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
Date Fri, 30 Jun 2017 02:37:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

luhuichun updated HDFS-12046:
    Attachment:     (was: ISA-L CRC Performance Report using intel ISA-L.pdf)

> Hadoop CRC implementation using Intel ISA-L library
> ---------------------------------------------------
>                 Key: HDFS-12046
>                 URL: https://issues.apache.org/jira/browse/HDFS-12046
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: luhuichun
>            Assignee: luhuichun
> Intel ISA-L open source library provides set of highly optimized functions for RAID,
erasure code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l.
HDFS-EC has already integrated ISA-L and added the necessary building options support for
Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L,
performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop
native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native
CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage
is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have
to maintain their own bunch of ASM codes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message