hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "luhuichun (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-12046) Hadoop CRC implementation using Intel ISAL-CRC library
Date Tue, 27 Jun 2017 03:32:00 GMT
luhuichun created HDFS-12046:

             Summary: Hadoop CRC implementation using Intel ISAL-CRC library
                 Key: HDFS-12046
                 URL: https://issues.apache.org/jira/browse/HDFS-12046
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: luhuichun
            Assignee: luhuichun

Intel ISA-L open source library provides set of highly optimized functions for RAID, erasure
code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l.
HDFS-EC has already integrated ISA-L and added the necessary building options support for
Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L,
performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop
native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native
CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage
is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have
to maintain their own bunch of ASM codes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message