Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 35958200D3D for ; Mon, 13 Nov 2017 11:50:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 33D9F160BF3; Mon, 13 Nov 2017 10:50:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 53766160BE4 for ; Mon, 13 Nov 2017 11:50:06 +0100 (CET) Received: (qmail 81782 invoked by uid 500); 13 Nov 2017 10:50:05 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 81771 invoked by uid 99); 13 Nov 2017 10:50:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Nov 2017 10:50:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A04D01A06BE for ; Mon, 13 Nov 2017 10:50:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id q9vzhjyeamC5 for ; Mon, 13 Nov 2017 10:50:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id CEEDF5FC9D for ; Mon, 13 Nov 2017 10:50:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 9DA81E0ECF for ; Mon, 13 Nov 2017 10:50:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 50240240DA for ; Mon, 13 Nov 2017 10:50:00 +0000 (UTC) Date: Mon, 13 Nov 2017 10:50:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 13 Nov 2017 10:50:07 -0000 [ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249373#comment-16249373 ] ASF GitHub Bot commented on HADOOP-15033: ----------------------------------------- GitHub user dchuyko opened a pull request: https://github.com/apache/hadoop/pull/291 HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above I added isJava9OrAbove() flag in Shell which is statically initialized using version string parsing. If it is true, DataChecksum creates zip implementation instead of PureJavaCrc32C. It is created using constructor method handle that's looked up also if it's 9 or above. Signature check should ignore java.lang.invoke to let method handles code pass build. Performance test now outputs extra column for 9 and above with results of zip CRC32C. Results on Cavium ThunderX server (AArch64) show ~4x improvement and are listed below. The picture is about the same on x86. | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 32 | 1 | 157.6 | 160.1 | 1.6% | 110.3 | -31.1% | 98.9 | -10.3% | | 32 | 2 | 171.2 | 184.2 | 7.6% | 107.0 | -41.9% | 112.9 | 5.5% | | 32 | 4 | 182.1 | 176.0 | -3.3% | 104.0 | -40.9% | 113.3 | 8.9% | | 32 | 8 | 179.3 | 173.0 | -3.6% | 102.1 | -41.0% | 111.5 | 9.3% | | 32 | 16 | 95.0 | 91.3 | -3.9% | 53.4 | -41.5% | 57.5 | 7.7% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 64 | 1 | 326.5 | 324.6 | -0.6% | 137.4 | -57.7% | 144.7 | 5.3% | | 64 | 2 | 298.8 | 293.8 | -1.7% | 133.5 | -54.6% | 139.5 | 4.5% | | 64 | 4 | 279.3 | 273.3 | -2.2% | 128.4 | -53.0% | 136.5 | 6.3% | | 64 | 8 | 269.8 | 267.2 | -0.9% | 125.9 | -52.9% | 133.6 | 6.1% | | 64 | 16 | 140.1 | 139.3 | -0.6% | 65.6 | -52.9% | 69.3 | 5.6% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 128 | 1 | 666.6 | 668.6 | 0.3% | 185.0 | -72.3% | 208.7 | 12.8% | | 128 | 2 | 656.3 | 616.9 | -6.0% | 184.5 | -70.1% | 207.7 | 12.6% | | 128 | 4 | 636.7 | 599.7 | -5.8% | 181.8 | -69.7% | 203.6 | 12.0% | | 128 | 8 | 628.3 | 592.3 | -5.7% | 180.6 | -69.5% | 202.2 | 12.0% | | 128 | 16 | 378.1 | 345.9 | -8.5% | 93.1 | -73.1% | 108.4 | 16.4% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 256 | 1 | 906.4 | 776.1 | -14.4% | 207.9 | -73.2% | 239.2 | 15.0% | | 256 | 2 | 882.9 | 833.0 | -5.6% | 206.6 | -75.2% | 238.6 | 15.5% | | 256 | 4 | 837.9 | 801.0 | -4.4% | 204.2 | -74.5% | 235.9 | 15.5% | | 256 | 8 | 806.1 | 776.7 | -3.7% | 201.7 | -74.0% | 225.6 | 11.8% | | 256 | 16 | 493.6 | 456.0 | -7.6% | 107.8 | -76.4% | 121.3 | 12.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 512 | 1 | 1016.3 | 850.4 | -16.3% | 214.6 | -74.8% | 248.7 | 15.9% | | 512 | 2 | 969.7 | 894.4 | -7.8% | 212.1 | -76.3% | 241.5 | 13.8% | | 512 | 4 | 894.0 | 829.4 | -7.2% | 207.4 | -75.0% | 232.1 | 11.9% | | 512 | 8 | 886.8 | 831.8 | -6.2% | 205.2 | -75.3% | 238.6 | 16.3% | | 512 | 16 | 518.3 | 525.4 | 1.4% | 111.1 | -78.9% | 124.8 | 12.4% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 1024 | 1 | 1120.0 | 1130.5 | 0.9% | 219.1 | -80.6% | 258.0 | 17.8% | | 1024 | 2 | 1054.9 | 983.9 | -6.7% | 217.5 | -77.9% | 254.8 | 17.1% | | 1024 | 4 | 956.8 | 766.0 | -19.9% | 213.0 | -72.2% | 241.1 | 13.2% | | 1024 | 8 | 835.5 | 864.9 | 3.5% | 209.6 | -75.8% | 247.2 | 18.0% | | 1024 | 16 | 545.0 | 540.5 | -0.8% | 110.1 | -79.6% | 131.8 | 19.7% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 2048 | 1 | 1189.2 | 1190.5 | 0.1% | 223.3 | -81.2% | 263.3 | 17.9% | | 2048 | 2 | 1116.4 | 1027.5 | -8.0% | 220.6 | -78.5% | 260.0 | 17.9% | | 2048 | 4 | 1009.9 | 931.0 | -7.8% | 215.3 | -76.9% | 253.3 | 17.6% | | 2048 | 8 | 950.4 | 881.0 | -7.3% | 213.2 | -75.8% | 251.5 | 17.9% | | 2048 | 16 | 570.7 | 521.7 | -8.6% | 111.7 | -78.6% | 132.1 | 18.3% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 4096 | 1 | 1264.7 | 991.2 | -21.6% | 225.7 | -77.2% | 267.6 | 18.6% | | 4096 | 2 | 1161.3 | 1023.4 | -11.9% | 222.2 | -78.3% | 263.2 | 18.5% | | 4096 | 4 | 1037.7 | 931.6 | -10.2% | 217.4 | -76.7% | 256.2 | 17.9% | | 4096 | 8 | 1010.8 | 925.2 | -8.5% | 214.7 | -76.8% | 255.4 | 18.9% | | 4096 | 16 | 626.8 | 575.2 | -8.2% | 115.4 | -79.9% | 129.8 | 12.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 8192 | 1 | 1264.7 | 1264.0 | -0.1% | 225.2 | -82.2% | 269.3 | 19.6% | | 8192 | 2 | 1182.1 | 1077.8 | -8.8% | 222.1 | -79.4% | 264.6 | 19.1% | | 8192 | 4 | 1056.2 | 968.4 | -8.3% | 217.3 | -77.6% | 257.6 | 18.6% | | 8192 | 8 | 1015.0 | 890.3 | -12.3% | 216.0 | -75.7% | 255.1 | 18.1% | | 8192 | 16 | 616.7 | 571.2 | -7.4% | 113.1 | -80.2% | 135.5 | 19.8% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 16384 | 1 | 1226.4 | 1116.2 | -9.0% | 221.6 | -80.1% | 262.9 | 18.6% | | 16384 | 2 | 1152.5 | 1080.3 | -6.3% | 219.0 | -79.7% | 259.1 | 18.3% | | 16384 | 4 | 895.7 | 976.4 | 9.0% | 214.5 | -78.0% | 253.1 | 18.0% | | 16384 | 8 | 985.8 | 961.2 | -2.5% | 211.0 | -78.0% | 252.2 | 19.5% | | 16384 | 16 | 567.6 | 565.2 | -0.4% | 110.0 | -80.5% | 133.6 | 21.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 32768 | 1 | 1107.3 | 913.8 | -17.5% | 217.2 | -76.2% | 257.1 | 18.3% | | 32768 | 2 | 1026.4 | 998.0 | -2.8% | 214.2 | -78.5% | 252.2 | 17.7% | | 32768 | 4 | 942.1 | 908.1 | -3.6% | 209.9 | -76.9% | 246.8 | 17.6% | | 32768 | 8 | 896.2 | 798.5 | -10.9% | 206.2 | -74.2% | 245.3 | 19.0% | | 32768 | 16 | 549.0 | 504.9 | -8.0% | 109.6 | -78.3% | 127.7 | 16.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 65536 | 1 | 1101.6 | 1078.4 | -2.1% | 217.2 | -79.9% | 256.3 | 18.0% | | 65536 | 2 | 1026.1 | 846.7 | -17.5% | 213.7 | -74.8% | 253.0 | 18.4% | | 65536 | 4 | 936.5 | 841.9 | -10.1% | 210.1 | -75.0% | 246.7 | 17.4% | | 65536 | 8 | 899.7 | 886.5 | -1.5% | 207.9 | -76.5% | 242.9 | 16.8% | | 65536 | 16 | 497.8 | 511.7 | 2.8% | 107.1 | -79.1% | 124.2 | 16.0% | You can merge this pull request into a Git repository by running: $ git pull https://github.com/dchuyko/hadoop HADOOP-15033 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #291 ---- commit 2f70c6eb65a433fb6a82f5ef2efa6db7480cb0f9 Author: tp Date: 2017-11-13T10:38:21Z HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above ---- > Use java.util.zip.CRC32C for Java 9 and above > --------------------------------------------- > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util > Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more aeefective than any pure Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org