hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Masatake Iwasaki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11765) Fix:Performance regression due to incorrect use of DataChecksum
Date Tue, 09 May 2017 09:03:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002334#comment-16002334

Masatake Iwasaki commented on HDFS-11765:

+1. I saw no relevant test failures in hadoop-hdfs. 

bq. BTW, the default checksum type is CRC32C in hadoop, which means HDFS will use PureJava
algorithm when we didn't let it use native.

Sure. Discussion about further optimization should be done in another JIRA. 

> Fix:Performance regression due to incorrect use of DataChecksum
> ---------------------------------------------------------------
>                 Key: HDFS-11765
>                 URL: https://issues.apache.org/jira/browse/HDFS-11765
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: native, performance
>    Affects Versions: 2.8.0, 3.0.0-alpha1
>            Reporter: LiXin Ge
>            Assignee: LiXin Ge
>         Attachments: HDFS-11765.patch
> Recently I have upgraded my Hadoop version from 2.6 to 3.0, and I find that the write
performance decreased by 13%. After some days comparative analysis, It's seems introduced
by HADOOP-10865. 
> Since James Thomas have done the work that native checksum can run against byte[] arrays
instead of just against byte buffers, we may use native method preferential because it runs
faster than others.
> [~szetszwo] and [~iwasakims] could you take a look at this to see if  it make bad effect
on your benchmark test? [~tlipcon] could you help to see if I have make mistakes in this patch?
> thanks!

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message