hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "luhuichun (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
Date Thu, 29 Jun 2017 04:14:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

luhuichun updated HDFS-12046:
-----------------------------
    Summary: Hadoop CRC implementation using Intel ISA-L library  (was: Hadoop CRC implementation
using Intel ISAL library)

> Hadoop CRC implementation using Intel ISA-L library
> ---------------------------------------------------
>
>                 Key: HDFS-12046
>                 URL: https://issues.apache.org/jira/browse/HDFS-12046
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: luhuichun
>            Assignee: luhuichun
>         Attachments: ISA-L CRC Performance Report using intel ISA-L.pdf
>
>
> Intel ISA-L open source library provides set of highly optimized functions for RAID,
erasure code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l.
HDFS-EC has already integrated ISA-L and added the necessary building options support for
Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L,
performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop
native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native
CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage
is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have
to maintain their own bunch of ASM codes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message