hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7333) Performance improvement in PureJavaCrc32
Date Fri, 27 May 2011 02:15:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040032#comment-13040032
] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7333:
------------------------------------------------

> ... I wonder if you use -XX:+AggressiveOpts on the earlier JVM if it would have the same
performance increases?

By comparing the following with (2.1), it seems both columns have some better numbers but
the patch still degrades the performance.

-----
h3. 2.3) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_05-b13 with -XX:+AggressiveOpts
$ java -XX:+AggressiveOpts -cp hadoop-common-0.23.0-SNAPSHOT.jar:hadoop-common-test-0.23.0-SNAPSHOT.jar
 org.apache.hadoop.util.TestPureJavaCrc32\$PerformanceTest
java.version = 1.6.0_05
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_05-b13
java.vm.version = 10.0-b19
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = i386
os.name = Linux
os.version = 2.6.18-53.1.13.el5

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     9.699 |        111.729 |   108.432 |
|          2 |    18.007 |        139.071 |   138.762 |
|          4 |    35.060 |        217.921 |   223.770 |
|          8 |    64.308 |        220.957 |   241.875 |
|         16 |   108.495 |        290.035 |   308.596 |
|         32 |   166.368 |        365.532 |   379.159 |
|         64 |   226.203 |        424.410 |   423.369 |
|        128 |   275.945 |        463.763 |   449.295 |
|        256 |   311.077 |        485.665 |   465.597 |
|        512 |   331.712 |        498.074 |   473.644 |
|       1024 |   340.871 |        504.308 |   475.610 |
|       2048 |   346.745 |        507.910 |   478.512 |
|       4096 |   351.109 |        508.981 |   480.016 |
|       8192 |   352.892 |        509.891 |   480.836 |
|      16384 |   350.394 |        508.394 |   480.766 |
|      32768 |   350.816 |        500.573 |   479.659 |
|      65536 |   348.351 |        500.105 |   473.086 |
|     131072 |   352.889 |        499.988 |   480.385 |
|     262144 |   352.796 |        499.779 |   480.421 |
|     524288 |   353.542 |        499.895 |   480.432 |
|    1048576 |   353.364 |        499.754 |   480.367 |
|    2097152 |   352.447 |        499.452 |   480.190 |
|    4194304 |   352.423 |        499.187 |   479.956 |
|    8388608 |   351.037 |        493.194 |   475.243 |
|   16777216 |   350.760 |        492.260 |   474.271 |

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc."
I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods
in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store
the final result back into the data member, it results in slightly more efficient hotspot
codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest"
which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety
of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10%
speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded
in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate
stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message