hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10331) Use java.util.zip.CRC32 for checksum in java8 or above
Date Tue, 26 Apr 2016 20:44:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258885#comment-15258885

Tsz Wo Nicholas Sze commented on HDFS-10331:

It seems that we are already using java.util.zip.CRC32 for Java 7 or above, including Java
  public static Checksum newCrc32() {
    return Shell.isJava7OrAbove()? new CRC32(): new PureJavaCrc32();

> Use java.util.zip.CRC32 for checksum in java8 or above
> ------------------------------------------------------
>                 Key: HDFS-10331
>                 URL: https://issues.apache.org/jira/browse/HDFS-10331
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, hdfs, hdfs-client
>    Affects Versions: 2.6.0
>            Reporter: He Tianyi
> In java8, performance of intrinsic CRC32 has been dramatically improved.
> See: https://bugs.openjdk.java.net/browse/JDK-7088419
> I carried an in-memory benchmark of throughput, on a server with two E5-2630 v2 cpus,
> java7  java.util.zip.CRC32: 0.81GB/s
> hdfs DataChecksum, native: 1.46GB/s
> java8  java.util.zip.CRC32: 2.39GB/s
> hdfs DataChecksum, CRC32 on java8: 2.39GB/s
> IMHO I think we could either:
> A) provide a configuration for user to switch CRC32 implementations;
> or B) On java8 or above, always use intrinsic CRC32.

This message was sent by Atlassian JIRA

View raw message