hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LiXin Ge (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14424) Add CRC32C performance test.
Date Wed, 14 Jun 2017 09:54:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048982#comment-16048982
] 

LiXin Ge commented on HADOOP-14424:
-----------------------------------

Thanks for your advice, [~iwasakims].
I'm done to combine the two performance functions into one test, some interface and functions
was modified to support both Crc32 and Crc32C. 
It's only print diff to one of the results now and a part of the output of Crc32PerformanceTest
is:
Direct Buffer():
{quote}
...
|  bpc  | #T ||      Zip || PureJava | % diff || PureJavaC | % diff ||   Native | % diff ||
 NativeC | % diff |
|    32 |  1 |     154.5 |     357.2 | 131.2% |      349.2 |  -2.2% |     783.3 | 124.3% |
   3146.0 | 301.6% |
|    32 |  2 |     154.3 |     354.0 | 129.4% |      350.2 |  -1.1% |     794.8 | 126.9% |
   1710.1 | 115.2% |
|    32 |  4 |      98.4 |     294.0 | 198.8% |      339.6 |  15.5% |     782.2 | 130.3% |
   2615.0 | 234.3% |
|    32 |  8 |     112.1 |     231.7 | 106.6% |      199.8 | -13.7% |     708.4 | 254.5% |
   1833.0 | 158.7% |
|    32 | 16 |      65.0 |     137.2 | 111.2% |      144.2 |   5.1% |     442.9 | 207.1% |
   1107.7 | 150.1% |
...
{quote}

Non-direct Buffer:
{quote}
...
|  bpc  | #T ||      Zip || PureJava | % diff || PureJavaC | % diff ||  NativeC | % diff |
|    32 |  1 |     161.3 |     402.0 | 149.3% |      396.9 |  -1.3% |    3028.9 | 663.1% |
|    32 |  2 |     158.3 |     403.1 | 154.6% |      396.4 |  -1.7% |    1697.1 | 328.1% |
|    32 |  4 |     156.8 |     402.7 | 156.8% |      394.7 |  -2.0% |    1898.1 | 380.9% |
|    32 |  8 |     157.4 |     402.8 | 155.9% |      392.7 |  -2.5% |    1897.4 | 383.2% |
|    32 | 16 |      91.4 |     224.1 | 145.2% |      219.9 |  -1.9% |    1202.8 | 446.9% |
...
{quote}

> Add CRC32C performance test.
> ----------------------------
>
>                 Key: HADOOP-14424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14424
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: common
>    Affects Versions: 3.0.0-alpha2
>            Reporter: LiXin Ge
>            Assignee: LiXin Ge
>            Priority: Minor
>              Labels: test
>         Attachments: HADOOP-14424.002.patch, HADOOP-14424.patch
>
>
> The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new test to
compare Crc32C chunked verification implementations.
> This test is based on Crc32PerformanceTest, what I have done in this test is:
> 1.CRC32C performance test.
> 2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it from this test.
> 3.User can choose either direct buffer or non-directly buffer to run this test manually.
> 4.Take use of verifyChunkedSumsByteArray for native to support non-directly native test.
> The test result in my environment is:
> [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
>                  java.version = 1.8.0_111
>             java.runtime.name = Java(TM) SE Runtime Environment
>          java.runtime.version = 1.8.0_111-b14
>               java.vm.version = 25.111-b14
>                java.vm.vendor = Oracle Corporation
>                  java.vm.name = Java HotSpot(TM) 64-Bit Server VM
> java.vm.specification.version = 1.8
>    java.specification.version = 1.8
>                       os.arch = amd64
>                       os.name = Linux
>                    os.version = 2.6.33.20
> Data Length = 64 MB
> Trials      = 5
> Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)
> |  bpc  | #T || PureJava ||   Native | % diff |
> |    32 |  1 |     394.0 |    4156.2 | 954.9% |
> |    32 |  2 |     400.5 |    3679.7 | 818.7% |
> |    32 |  4 |     401.8 |    2657.0 | 561.3% |
> |    32 |  8 |     389.1 |    1633.8 | 319.9% |
> |    32 | 16 |     222.2 |    1116.3 | 402.5% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |    64 |  1 |     465.0 |    5931.0 | 1175.5% |
> |    64 |  2 |     468.8 |    1839.2 | 292.3% |
> |    64 |  4 |     460.4 |    2968.3 | 544.7% |
> |    64 |  8 |     452.4 |    1925.7 | 325.6% |
> |    64 | 16 |     246.9 |    1291.8 | 423.3% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   128 |  1 |     522.0 |    6147.8 | 1077.6% |
> |   128 |  2 |     366.0 |    4758.5 | 1200.2% |
> |   128 |  4 |     307.8 |    3265.1 | 960.8% |
> |   128 |  8 |     283.6 |    2092.2 | 637.6% |
> |   128 | 16 |     219.9 |    1226.1 | 457.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   256 |  1 |     550.7 |    3177.6 | 477.0% |
> |   256 |  2 |     538.6 |    1933.2 | 258.9% |
> |   256 |  4 |     427.2 |    3278.1 | 667.3% |
> |   256 |  8 |     420.8 |    2272.3 | 440.0% |
> |   256 | 16 |     294.0 |    1311.2 | 346.0% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   512 |  1 |     553.4 |    3690.4 | 566.9% |
> |   512 |  2 |     455.6 |    4974.1 | 991.7% |
> |   512 |  4 |     494.2 |    3406.4 | 589.2% |
> |   512 |  8 |     431.4 |    2257.0 | 423.2% |
> |   512 | 16 |     316.3 |    1272.0 | 302.2% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  1024 |  1 |     566.1 |    3520.0 | 521.8% |
> |  1024 |  2 |     508.7 |    4437.4 | 772.3% |
> |  1024 |  4 |     520.7 |    3422.6 | 557.4% |
> |  1024 |  8 |     501.8 |    2124.7 | 323.4% |
> |  1024 | 16 |     340.6 |    1305.0 | 283.2% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  2048 |  1 |     535.1 |    5438.5 | 916.4% |
> |  2048 |  2 |     537.3 |    4668.3 | 768.8% |
> |  2048 |  4 |     529.2 |    2417.2 | 356.7% |
> |  2048 |  8 |     485.1 |    2249.8 | 363.8% |
> |  2048 | 16 |     334.3 |    1265.7 | 278.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  4096 |  1 |     563.0 |    7264.0 | 1190.1% |
> |  4096 |  2 |     538.8 |    5681.4 | 954.4% |
> |  4096 |  4 |     528.9 |    3107.6 | 487.5% |
> |  4096 |  8 |     521.8 |    2168.9 | 315.6% |
> |  4096 | 16 |     336.5 |    1256.5 | 273.4% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  8192 |  1 |     521.1 |    6842.4 | 1213.1% |
> |  8192 |  2 |     543.0 |    5510.2 | 914.8% |
> |  8192 |  4 |     526.1 |    3759.5 | 614.6% |
> |  8192 |  8 |     480.1 |    2254.1 | 369.6% |
> |  8192 | 16 |     336.4 |    1213.3 | 260.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 16384 |  1 |     521.7 |    3695.7 | 608.3% |
> | 16384 |  2 |     540.8 |    5610.5 | 937.4% |
> | 16384 |  4 |     539.1 |    3313.0 | 514.6% |
> | 16384 |  8 |     518.7 |    2198.4 | 323.8% |
> | 16384 | 16 |     337.2 |    1237.8 | 267.1% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 32768 |  1 |     532.1 |    3696.0 | 594.6% |
> | 32768 |  2 |     541.4 |    4647.4 | 758.4% |
> | 32768 |  4 |     536.6 |    2325.1 | 333.3% |
> | 32768 |  8 |     518.8 |    2036.9 | 292.6% |
> | 32768 | 16 |     337.8 |    1232.0 | 264.7% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 65536 |  1 |     527.7 |    8240.0 | 1461.4% |
> | 65536 |  2 |     561.9 |    5472.3 | 873.9% |
> | 65536 |  4 |     536.5 |    3725.2 | 594.4% |
> | 65536 |  8 |     482.8 |    2040.2 | 322.5% |
> | 65536 | 16 |     335.9 |    1240.4 | 269.3% |
> Elapsed 77.4s



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message