hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14424) Add CRC32C performance test.
Date Wed, 31 May 2017 00:28:04 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Wang updated HADOOP-14424:
---------------------------------
    Fix Version/s:     (was: 3.0.0-alpha2)

> Add CRC32C performance test.
> ----------------------------
>
>                 Key: HADOOP-14424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14424
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: common
>    Affects Versions: 3.0.0-alpha2
>            Reporter: LiXin Ge
>            Assignee: LiXin Ge
>            Priority: Minor
>              Labels: test
>         Attachments: HADOOP-14424.patch
>
>
> The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new test to
compare Crc32C chunked verification implementations.
> This test is based on Crc32PerformanceTest, what I have done in this test is:
> 1.CRC32C performance test.
> 2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it from this test.
> 3.User can choose either direct buffer or non-directly buffer to run this test manually.
> 4.Take use of verifyChunkedSumsByteArray for native to support non-directly native test.
> The test result in my environment is:
> [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
>                  java.version = 1.8.0_111
>             java.runtime.name = Java(TM) SE Runtime Environment
>          java.runtime.version = 1.8.0_111-b14
>               java.vm.version = 25.111-b14
>                java.vm.vendor = Oracle Corporation
>                  java.vm.name = Java HotSpot(TM) 64-Bit Server VM
> java.vm.specification.version = 1.8
>    java.specification.version = 1.8
>                       os.arch = amd64
>                       os.name = Linux
>                    os.version = 2.6.33.20
> Data Length = 64 MB
> Trials      = 5
> Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)
> |  bpc  | #T || PureJava ||   Native | % diff |
> |    32 |  1 |     394.0 |    4156.2 | 954.9% |
> |    32 |  2 |     400.5 |    3679.7 | 818.7% |
> |    32 |  4 |     401.8 |    2657.0 | 561.3% |
> |    32 |  8 |     389.1 |    1633.8 | 319.9% |
> |    32 | 16 |     222.2 |    1116.3 | 402.5% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |    64 |  1 |     465.0 |    5931.0 | 1175.5% |
> |    64 |  2 |     468.8 |    1839.2 | 292.3% |
> |    64 |  4 |     460.4 |    2968.3 | 544.7% |
> |    64 |  8 |     452.4 |    1925.7 | 325.6% |
> |    64 | 16 |     246.9 |    1291.8 | 423.3% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   128 |  1 |     522.0 |    6147.8 | 1077.6% |
> |   128 |  2 |     366.0 |    4758.5 | 1200.2% |
> |   128 |  4 |     307.8 |    3265.1 | 960.8% |
> |   128 |  8 |     283.6 |    2092.2 | 637.6% |
> |   128 | 16 |     219.9 |    1226.1 | 457.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   256 |  1 |     550.7 |    3177.6 | 477.0% |
> |   256 |  2 |     538.6 |    1933.2 | 258.9% |
> |   256 |  4 |     427.2 |    3278.1 | 667.3% |
> |   256 |  8 |     420.8 |    2272.3 | 440.0% |
> |   256 | 16 |     294.0 |    1311.2 | 346.0% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   512 |  1 |     553.4 |    3690.4 | 566.9% |
> |   512 |  2 |     455.6 |    4974.1 | 991.7% |
> |   512 |  4 |     494.2 |    3406.4 | 589.2% |
> |   512 |  8 |     431.4 |    2257.0 | 423.2% |
> |   512 | 16 |     316.3 |    1272.0 | 302.2% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  1024 |  1 |     566.1 |    3520.0 | 521.8% |
> |  1024 |  2 |     508.7 |    4437.4 | 772.3% |
> |  1024 |  4 |     520.7 |    3422.6 | 557.4% |
> |  1024 |  8 |     501.8 |    2124.7 | 323.4% |
> |  1024 | 16 |     340.6 |    1305.0 | 283.2% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  2048 |  1 |     535.1 |    5438.5 | 916.4% |
> |  2048 |  2 |     537.3 |    4668.3 | 768.8% |
> |  2048 |  4 |     529.2 |    2417.2 | 356.7% |
> |  2048 |  8 |     485.1 |    2249.8 | 363.8% |
> |  2048 | 16 |     334.3 |    1265.7 | 278.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  4096 |  1 |     563.0 |    7264.0 | 1190.1% |
> |  4096 |  2 |     538.8 |    5681.4 | 954.4% |
> |  4096 |  4 |     528.9 |    3107.6 | 487.5% |
> |  4096 |  8 |     521.8 |    2168.9 | 315.6% |
> |  4096 | 16 |     336.5 |    1256.5 | 273.4% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  8192 |  1 |     521.1 |    6842.4 | 1213.1% |
> |  8192 |  2 |     543.0 |    5510.2 | 914.8% |
> |  8192 |  4 |     526.1 |    3759.5 | 614.6% |
> |  8192 |  8 |     480.1 |    2254.1 | 369.6% |
> |  8192 | 16 |     336.4 |    1213.3 | 260.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 16384 |  1 |     521.7 |    3695.7 | 608.3% |
> | 16384 |  2 |     540.8 |    5610.5 | 937.4% |
> | 16384 |  4 |     539.1 |    3313.0 | 514.6% |
> | 16384 |  8 |     518.7 |    2198.4 | 323.8% |
> | 16384 | 16 |     337.2 |    1237.8 | 267.1% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 32768 |  1 |     532.1 |    3696.0 | 594.6% |
> | 32768 |  2 |     541.4 |    4647.4 | 758.4% |
> | 32768 |  4 |     536.6 |    2325.1 | 333.3% |
> | 32768 |  8 |     518.8 |    2036.9 | 292.6% |
> | 32768 | 16 |     337.8 |    1232.0 | 264.7% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 65536 |  1 |     527.7 |    8240.0 | 1461.4% |
> | 65536 |  2 |     561.9 |    5472.3 | 873.9% |
> | 65536 |  4 |     536.5 |    3725.2 | 594.4% |
> | 65536 |  8 |     482.8 |    2040.2 | 322.5% |
> | 65536 | 16 |     335.9 |    1240.4 | 269.3% |
> Elapsed 77.4s



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message