hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "SammiChen (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
Date Tue, 24 Oct 2017 07:51:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

SammiChen reassigned HDFS-12046:

    Assignee: SammiChen  (was: luhuichun)

> Hadoop CRC implementation using Intel ISA-L library
> ---------------------------------------------------
>                 Key: HDFS-12046
>                 URL: https://issues.apache.org/jira/browse/HDFS-12046
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: luhuichun
>            Assignee: SammiChen
>         Attachments: HDFS-12046-001.patch, ISA-L CRC Performance Report using intel ISA-L.pdf
> Intel ISA-L open source library provides set of highly optimized functions for RAID,
erasure code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l.
HDFS-EC has already integrated ISA-L and added the necessary building options support for
Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L,
performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop
native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native
CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage
is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have
to maintain their own bunch of ASM codes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message