hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10674) Rewrite the PureJavaCrc32 loop for performance improvement
Date Mon, 09 Jun 2014 23:15:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14025912#comment-14025912
] 

Tsz Wo Nicholas Sze commented on HADOOP-10674:
----------------------------------------------

Interesting, the result is difference in Java 7.  java.util.zip.CRC32 is the best in Java
7

java.version = 1.7.0_45
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.7.0_45-b18
java.vm.version = 24.45-b08
java.vm.vendor = Oracle Corporation
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.7
java.specification.version = 1.7
os.arch = x86_64
os.name = Mac OS X
os.version = 10.9.3

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 | % diff || PureJavaCrc32new | % diff | % diff ||
|          1 |    10.463 |        145.591 | 1291.5% |           156.814 | 1398.7% |   7.7%
|
|          2 |    22.984 |        267.600 | 1064.3% |           286.817 | 1147.9% |   7.2%
|
|          4 |    42.437 |        338.265 | 697.1% |           338.419 | 697.5% |   0.0% |
|          8 |    77.802 |        781.078 | 903.9% |           638.547 | 720.7% | -18.2% |
|         16 |   146.058 |       1086.145 | 643.6% |           863.587 | 491.3% | -20.5% |
|         32 |   256.917 |       1301.898 | 406.7% |          1030.223 | 301.0% | -20.9% |
|         64 |   449.421 |       1455.786 | 223.9% |          1164.038 | 159.0% | -20.0% |
|        128 |   753.642 |       1557.765 | 106.7% |          1246.300 |  65.4% | -20.0% |
|        256 |  1307.296 |       1532.365 |  17.2% |          1245.948 |  -4.7% | -18.7% |
|        512 |  1811.799 |       1598.921 | -11.7% |          1492.607 | -17.6% |  -6.6% |
|       1024 |  2279.771 |       1570.552 | -31.1% |          1551.045 | -32.0% |  -1.2% |
|       2048 |  2541.397 |       1639.197 | -35.5% |          1535.380 | -39.6% |  -6.3% |
|       4096 |  2733.641 |       1648.720 | -39.7% |          1622.368 | -40.7% |  -1.6% |
|       8192 |  2806.648 |       1630.906 | -41.9% |          1643.027 | -41.5% |   0.7% |
|      16384 |  2892.917 |       1629.971 | -43.7% |          1646.938 | -43.1% |   1.0% |
|      32768 |  2944.763 |       1596.667 | -45.8% |          1661.647 | -43.6% |   4.1% |
|      65536 |  2961.181 |       1636.034 | -44.8% |          1656.229 | -44.1% |   1.2% |
|     131072 |  2743.073 |       1627.546 | -40.7% |          1663.785 | -39.3% |   2.2% |
|     262144 |  2979.100 |       1661.431 | -44.2% |          1659.321 | -44.3% |  -0.1% |
|     524288 |  2966.809 |       1627.857 | -45.1% |          1641.636 | -44.7% |   0.8% |
|    1048576 |  2987.165 |       1645.096 | -44.9% |          1596.109 | -46.6% |  -3.0% |
|    2097152 |  2876.469 |       1588.168 | -44.8% |          1651.272 | -42.6% |   4.0% |
|    4194304 |  2884.637 |       1623.130 | -43.7% |          1633.612 | -43.4% |   0.6% |
|    8388608 |  2956.257 |       1654.131 | -44.0% |          1643.217 | -44.4% |  -0.7% |
|   16777216 |  2951.417 |       1645.519 | -44.2% |          1243.503 | -57.9% | -24.4% |


> Rewrite the PureJavaCrc32 loop for performance improvement
> ----------------------------------------------------------
>
>                 Key: HADOOP-10674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10674
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: performance, util
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: c10674_20140609.patch
>
>
> Below are some performance improvement opportunities performance improvement in PureJavaCrc32.
> - eliminate "off += 8; len -= 8;"
> - replace T8_x_start with hard coded constants
> - eliminate c0 - c7 local variables
> In my machine, there are 30% to 50% improvement for most of the cases.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message