hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10674) Rewrite the PureJavaCrc32 loop for performance improvement
Date Tue, 10 Jun 2014 05:56:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026142#comment-14026142
] 

Rajesh Balamohan commented on HADOOP-10674:
-------------------------------------------

Posting the results for 1.8 (even though it might take lot of time for Hadoop to move to 1.8).


java.version = 1.8.0_05
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.8.0_05-b13
java.vm.version = 25.5-b02
java.vm.vendor = Oracle Corporation
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.8
java.specification.version = 1.8
os.arch = x86_64
os.name = Mac OS X
os.version = 10.9.2

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 | % diff || PureJavaCrc32new | % diff ||
|          1 |   136.199 |        184.501 |  35.5% |           195.550 |   6.0% |
|          2 |   261.167 |        283.589 |   8.6% |           277.812 |  -2.0% |
|          4 |   316.930 |        325.149 |   2.6% |           323.370 |  -0.5% |
|          8 |   350.402 |        839.801 | 139.7% |           630.399 | -24.9% |
|         16 |   371.638 |       1191.862 | 220.7% |           845.571 | -29.1% |
|         32 |   451.213 |       1400.300 | 210.3% |           958.945 | -31.5% |
|         64 |   810.445 |       1556.288 |  92.0% |          1133.285 | -27.2% |
|        128 |  1366.747 |       1581.594 |  15.7% |          1189.381 | -24.8% |
|        256 |  2526.898 |       1497.479 | -40.7% |          1292.694 | -13.7% |
|        512 |  4407.562 |       1591.387 | -63.9% |          1485.212 |  -6.7% |
|       1024 |  6661.116 |       1633.674 | -75.5% |          1576.161 |  -3.5% |
|       2048 |  8702.155 |       1639.995 | -81.2% |          1599.360 |  -2.5% |
|       4096 | 10258.054 |       1609.719 | -84.3% |          1652.210 |   2.6% |
|       8192 | 11365.654 |       1655.651 | -85.4% |          1646.133 |  -0.6% |
|      16384 | 12126.954 |       1592.555 | -86.9% |          1679.129 |   5.4% |
|      32768 | 12155.745 |       1654.580 | -86.4% |          1655.308 |   0.0% |
|      65536 | 12411.519 |       1641.699 | -86.8% |          1665.409 |   1.4% |
|     131072 | 12425.978 |       1652.551 | -86.7% |          1568.704 |  -5.1% |
|     262144 | 11402.102 |       1600.620 | -86.0% |          1670.975 |   4.4% |
|     524288 | 11499.416 |       1618.737 | -85.9% |          1656.208 |   2.3% |
|    1048576 | 12458.633 |       1639.512 | -86.8% |          1411.573 | -13.9% |
|    2097152 | 12368.345 |       1552.908 | -87.4% |          1634.488 |   5.3% |
|    4194304 | 11476.733 |       1581.028 | -86.2% |          1642.247 |   3.9% |
|    8388608 | 11798.322 |       1543.582 | -86.9% |          1634.050 |   5.9% |
|   16777216 | 11103.400 |       1641.573 | -85.2% |          1580.520 |  -3.7% |

With  1.7.0_51

java.version = 1.7.0_51
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.7.0_51-b13
java.vm.version = 24.51-b03
java.vm.vendor = Oracle Corporation
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.7
java.specification.version = 1.7
os.arch = x86_64
os.name = Mac OS X
os.version = 10.9.2

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 | % diff || PureJavaCrc32new | % diff ||
|          1 |    10.496 |        150.871 | 1337.4% |           182.493 |  21.0% |
|          2 |    21.612 |        284.191 | 1215.0% |           283.723 |  -0.2% |
|          4 |    43.344 |        323.760 | 646.9% |           318.677 |  -1.6% |
|          8 |    79.073 |        818.739 | 935.4% |           632.961 | -22.7% |
|         16 |   145.119 |       1186.438 | 717.6% |           837.318 | -29.4% |
|         32 |   255.357 |       1403.678 | 449.7% |          1009.559 | -28.1% |
|         64 |   499.140 |       1507.390 | 202.0% |          1128.529 | -25.1% |
|        128 |   890.676 |       1596.368 |  79.2% |          1185.855 | -25.7% |
|        256 |  1623.995 |       1528.194 |  -5.9% |          1258.926 | -17.6% |
|        512 |  3097.399 |       1591.901 | -48.6% |          1449.768 |  -8.9% |
|       1024 |  4218.714 |       1585.198 | -62.4% |          1588.188 |   0.2% |
|       2048 |  7154.835 |       1637.164 | -77.1% |          1623.995 |  -0.8% |
|       4096 |  9198.045 |       1639.722 | -82.2% |          1450.458 | -11.5% |
|       8192 | 10574.143 |       1629.141 | -84.6% |          1682.528 |   3.3% |
|      16384 | 11368.683 |       1591.268 | -86.0% |          1670.931 |   5.0% |
|      32768 | 11695.906 |       1625.149 | -86.1% |          1615.529 |  -0.6% |
|      65536 | 11958.146 |       1620.048 | -86.5% |          1676.622 |   3.5% |
|     131072 | 12567.501 |       1649.230 | -86.9% |          1599.780 |  -3.0% |
|     262144 | 12415.131 |       1583.394 | -87.2% |          1698.063 |   7.2% |
|     524288 | 12389.894 |       1652.252 | -86.7% |          1685.808 |   2.0% |
|    1048576 | 11980.532 |       1658.224 | -86.2% |          1673.268 |   0.9% |
|    2097152 | 11981.653 |       1655.779 | -86.2% |          1682.860 |   1.6% |
|    4194304 | 12092.584 |       1647.340 | -86.4% |          1628.892 |  -1.1% |
|    8388608 | 12014.267 |       1545.241 | -87.1% |          1621.628 |   4.9% |
|   16777216 | 10965.476 |       1646.705 | -85.0% |          1606.345 |  -2.5% |




> Rewrite the PureJavaCrc32 loop for performance improvement
> ----------------------------------------------------------
>
>                 Key: HADOOP-10674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10674
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: performance, util
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: c10674_20140609.patch, c10674_20140609b.patch
>
>
> Below are some performance improvement opportunities performance improvement in PureJavaCrc32.
> - eliminate "off += 8; len -= 8;"
> - replace T8_x_start with hard coded constants
> - eliminate c0 - c7 local variables
> In my machine, there are 30% to 50% improvement for most of the cases.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message