hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "luhuichun (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
Date Thu, 29 Jun 2017 04:14:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

luhuichun updated HDFS-12046:
    Summary: Hadoop CRC implementation using Intel ISA-L library  (was: Hadoop CRC implementation
using Intel ISAL library)

> Hadoop CRC implementation using Intel ISA-L library
> ---------------------------------------------------
>                 Key: HDFS-12046
>                 URL: https://issues.apache.org/jira/browse/HDFS-12046
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: luhuichun
>            Assignee: luhuichun
>         Attachments: ISA-L CRC Performance Report using intel ISA-L.pdf
> Intel ISA-L open source library provides set of highly optimized functions for RAID,
erasure code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l.
HDFS-EC has already integrated ISA-L and added the necessary building options support for
Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L,
performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop
native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native
CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage
is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have
to maintain their own bunch of ASM codes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message