hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7333) Performance improvement in PureJavaCrc32
Date Thu, 26 May 2011 17:27:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039806#comment-13039806
] 

Todd Lipcon commented on HADOOP-7333:
-------------------------------------

I was able to reproduce a similar performance gain on one of our test boxes (Xeon E5540 @
2.53ghz)

Without patch:

|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |    15.644 |        167.530 |
|          2 |    28.458 |        167.751 |
|          4 |    56.510 |        269.064 |
|          8 |    94.573 |        440.632 |
|         16 |   144.602 |        458.159 |
|         32 |   197.831 |        557.829 |
|         64 |   246.404 |        618.492 |
|        128 |   275.926 |        630.191 |
|        256 |   294.701 |        646.540 |
|        512 |   306.519 |        660.747 |
|       1024 |   308.296 |        666.046 |
|       2048 |   315.532 |        675.444 |
|       4096 |   314.267 |        666.920 |
|       8192 |   314.343 |        669.222 |
|      16384 |   314.655 |        674.295 |
|      32768 |   314.940 |        665.232 |
|      65536 |   314.991 |        670.965 |
|     131072 |   314.950 |        678.721 |
|     262144 |   318.098 |        665.080 |
|     524288 |   317.110 |        661.923 |
|    1048576 |   315.832 |        672.078 |
|    2097152 |   313.902 |        669.421 |
|    4194304 |   315.350 |        670.617 |
|    8388608 |   316.019 |        673.716 |
|   16777216 |   316.324 |        657.439 |

With patch:

|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |    15.801 |        150.324 |
|          2 |    29.117 |        159.762 |
|          4 |    60.321 |        261.519 |
|          8 |   100.779 |        423.568 |
|         16 |   153.917 |        458.597 |
|         32 |   208.510 |        564.227 |
|         64 |   249.382 |        644.973 |
|        128 |   281.078 |        714.154 |
|        256 |   298.333 |        737.455 |
|        512 |   308.425 |        750.980 |
|       1024 |   313.149 |        759.572 |
|       2048 |   315.489 |        764.234 |
|       4096 |   316.663 |        767.175 |
|       8192 |   317.716 |        768.538 |
|      16384 |   315.930 |        758.694 |
|      32768 |   317.824 |        763.277 |
|      65536 |   317.246 |        764.298 |
|     131072 |   317.927 |        765.582 |
|     262144 |   317.217 |        750.438 |
|     524288 |   318.005 |        759.085 |
|    1048576 |   318.100 |        763.027 |
|    2097152 |   318.050 |        762.100 |
|    4194304 |   317.742 |        757.370 |
|    8388608 |   316.887 |        753.327 |
|   16777216 |   315.039 |        750.667 |

Our most common use of CRC32 in DFS is with 512 byte chunk size, where we see about a 20%
speedup in this test run. For big chunks, looks like about 15% - this is closer to the MR
spill checksumming case.

So, +1 pending Hudson results.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc."
I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods
in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store
the final result back into the data member, it results in slightly more efficient hotspot
codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest"
which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety
of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10%
speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded
in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate
stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message