hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
Date Mon, 13 Nov 2017 10:50:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249373#comment-16249373
] 

ASF GitHub Bot commented on HADOOP-15033:
-----------------------------------------

GitHub user dchuyko opened a pull request:

    https://github.com/apache/hadoop/pull/291

    HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above

    I added isJava9OrAbove() flag in Shell which is statically initialized using version string
parsing.
    If it is true, DataChecksum creates zip implementation instead of PureJavaCrc32C. It is
created using constructor method handle that's looked up also if it's 9 or above.
    Signature check should ignore java.lang.invoke to let method handles code pass build.
    Performance test now outputs extra column for 9 and above with results of zip CRC32C.
    
    Results on Cavium ThunderX server (AArch64) show ~4x improvement and are listed below.
The picture is about the same on x86.
    
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |    32 |  1 |     157.6 |     160.1 |   1.6% |     110.3 | -31.1% |       98.9 | -10.3%
|
    |    32 |  2 |     171.2 |     184.2 |   7.6% |     107.0 | -41.9% |      112.9 |   5.5%
|
    |    32 |  4 |     182.1 |     176.0 |  -3.3% |     104.0 | -40.9% |      113.3 |   8.9%
|
    |    32 |  8 |     179.3 |     173.0 |  -3.6% |     102.1 | -41.0% |      111.5 |   9.3%
|
    |    32 | 16 |      95.0 |      91.3 |  -3.9% |      53.4 | -41.5% |       57.5 |   7.7%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |    64 |  1 |     326.5 |     324.6 |  -0.6% |     137.4 | -57.7% |      144.7 |   5.3%
|
    |    64 |  2 |     298.8 |     293.8 |  -1.7% |     133.5 | -54.6% |      139.5 |   4.5%
|
    |    64 |  4 |     279.3 |     273.3 |  -2.2% |     128.4 | -53.0% |      136.5 |   6.3%
|
    |    64 |  8 |     269.8 |     267.2 |  -0.9% |     125.9 | -52.9% |      133.6 |   6.1%
|
    |    64 | 16 |     140.1 |     139.3 |  -0.6% |      65.6 | -52.9% |       69.3 |   5.6%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |   128 |  1 |     666.6 |     668.6 |   0.3% |     185.0 | -72.3% |      208.7 |  12.8%
|
    |   128 |  2 |     656.3 |     616.9 |  -6.0% |     184.5 | -70.1% |      207.7 |  12.6%
|
    |   128 |  4 |     636.7 |     599.7 |  -5.8% |     181.8 | -69.7% |      203.6 |  12.0%
|
    |   128 |  8 |     628.3 |     592.3 |  -5.7% |     180.6 | -69.5% |      202.2 |  12.0%
|
    |   128 | 16 |     378.1 |     345.9 |  -8.5% |      93.1 | -73.1% |      108.4 |  16.4%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |   256 |  1 |     906.4 |     776.1 | -14.4% |     207.9 | -73.2% |      239.2 |  15.0%
|
    |   256 |  2 |     882.9 |     833.0 |  -5.6% |     206.6 | -75.2% |      238.6 |  15.5%
|
    |   256 |  4 |     837.9 |     801.0 |  -4.4% |     204.2 | -74.5% |      235.9 |  15.5%
|
    |   256 |  8 |     806.1 |     776.7 |  -3.7% |     201.7 | -74.0% |      225.6 |  11.8%
|
    |   256 | 16 |     493.6 |     456.0 |  -7.6% |     107.8 | -76.4% |      121.3 |  12.5%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |   512 |  1 |    1016.3 |     850.4 | -16.3% |     214.6 | -74.8% |      248.7 |  15.9%
|
    |   512 |  2 |     969.7 |     894.4 |  -7.8% |     212.1 | -76.3% |      241.5 |  13.8%
|
    |   512 |  4 |     894.0 |     829.4 |  -7.2% |     207.4 | -75.0% |      232.1 |  11.9%
|
    |   512 |  8 |     886.8 |     831.8 |  -6.2% |     205.2 | -75.3% |      238.6 |  16.3%
|
    |   512 | 16 |     518.3 |     525.4 |   1.4% |     111.1 | -78.9% |      124.8 |  12.4%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |  1024 |  1 |    1120.0 |    1130.5 |   0.9% |     219.1 | -80.6% |      258.0 |  17.8%
|
    |  1024 |  2 |    1054.9 |     983.9 |  -6.7% |     217.5 | -77.9% |      254.8 |  17.1%
|
    |  1024 |  4 |     956.8 |     766.0 | -19.9% |     213.0 | -72.2% |      241.1 |  13.2%
|
    |  1024 |  8 |     835.5 |     864.9 |   3.5% |     209.6 | -75.8% |      247.2 |  18.0%
|
    |  1024 | 16 |     545.0 |     540.5 |  -0.8% |     110.1 | -79.6% |      131.8 |  19.7%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |  2048 |  1 |    1189.2 |    1190.5 |   0.1% |     223.3 | -81.2% |      263.3 |  17.9%
|
    |  2048 |  2 |    1116.4 |    1027.5 |  -8.0% |     220.6 | -78.5% |      260.0 |  17.9%
|
    |  2048 |  4 |    1009.9 |     931.0 |  -7.8% |     215.3 | -76.9% |      253.3 |  17.6%
|
    |  2048 |  8 |     950.4 |     881.0 |  -7.3% |     213.2 | -75.8% |      251.5 |  17.9%
|
    |  2048 | 16 |     570.7 |     521.7 |  -8.6% |     111.7 | -78.6% |      132.1 |  18.3%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |  4096 |  1 |    1264.7 |     991.2 | -21.6% |     225.7 | -77.2% |      267.6 |  18.6%
|
    |  4096 |  2 |    1161.3 |    1023.4 | -11.9% |     222.2 | -78.3% |      263.2 |  18.5%
|
    |  4096 |  4 |    1037.7 |     931.6 | -10.2% |     217.4 | -76.7% |      256.2 |  17.9%
|
    |  4096 |  8 |    1010.8 |     925.2 |  -8.5% |     214.7 | -76.8% |      255.4 |  18.9%
|
    |  4096 | 16 |     626.8 |     575.2 |  -8.2% |     115.4 | -79.9% |      129.8 |  12.5%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    |  8192 |  1 |    1264.7 |    1264.0 |  -0.1% |     225.2 | -82.2% |      269.3 |  19.6%
|
    |  8192 |  2 |    1182.1 |    1077.8 |  -8.8% |     222.1 | -79.4% |      264.6 |  19.1%
|
    |  8192 |  4 |    1056.2 |     968.4 |  -8.3% |     217.3 | -77.6% |      257.6 |  18.6%
|
    |  8192 |  8 |    1015.0 |     890.3 | -12.3% |     216.0 | -75.7% |      255.1 |  18.1%
|
    |  8192 | 16 |     616.7 |     571.2 |  -7.4% |     113.1 | -80.2% |      135.5 |  19.8%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    | 16384 |  1 |    1226.4 |    1116.2 |  -9.0% |     221.6 | -80.1% |      262.9 |  18.6%
|
    | 16384 |  2 |    1152.5 |    1080.3 |  -6.3% |     219.0 | -79.7% |      259.1 |  18.3%
|
    | 16384 |  4 |     895.7 |     976.4 |   9.0% |     214.5 | -78.0% |      253.1 |  18.0%
|
    | 16384 |  8 |     985.8 |     961.2 |  -2.5% |     211.0 | -78.0% |      252.2 |  19.5%
|
    | 16384 | 16 |     567.6 |     565.2 |  -0.4% |     110.0 | -80.5% |      133.6 |  21.5%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    | 32768 |  1 |    1107.3 |     913.8 | -17.5% |     217.2 | -76.2% |      257.1 |  18.3%
|
    | 32768 |  2 |    1026.4 |     998.0 |  -2.8% |     214.2 | -78.5% |      252.2 |  17.7%
|
    | 32768 |  4 |     942.1 |     908.1 |  -3.6% |     209.9 | -76.9% |      246.8 |  17.6%
|
    | 32768 |  8 |     896.2 |     798.5 | -10.9% |     206.2 | -74.2% |      245.3 |  19.0%
|
    | 32768 | 16 |     549.0 |     504.9 |  -8.0% |     109.6 | -78.3% |      127.7 |  16.5%
|
    |  bpc  | #T ||      Zip ||     ZipC | % diff || PureJava | % diff || PureJavaC | % diff
|
    | 65536 |  1 |    1101.6 |    1078.4 |  -2.1% |     217.2 | -79.9% |      256.3 |  18.0%
|
    | 65536 |  2 |    1026.1 |     846.7 | -17.5% |     213.7 | -74.8% |      253.0 |  18.4%
|
    | 65536 |  4 |     936.5 |     841.9 | -10.1% |     210.1 | -75.0% |      246.7 |  17.4%
|
    | 65536 |  8 |     899.7 |     886.5 |  -1.5% |     207.9 | -76.5% |      242.9 |  16.8%
|
    | 65536 | 16 |     497.8 |     511.7 |   2.8% |     107.1 | -79.1% |      124.2 |  16.0%
|

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dchuyko/hadoop HADOOP-15033

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/hadoop/pull/291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #291
    
----
commit 2f70c6eb65a433fb6a82f5ef2efa6db7480cb0f9
Author: tp <dmitry.chuyko@bell-sw.com>
Date:   2017-11-13T10:38:21Z

    HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above

----


> Use java.util.zip.CRC32C for Java 9 and above
> ---------------------------------------------
>
>                 Key: HADOOP-15033
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15033
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: performance, util
>            Reporter: Dmitry Chuyko
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more aeefective than any pure Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible with method
handle on 9 to instances implementing Checksum in runtime.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message