hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Masatake Iwasaki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14424) Add CRC32C performance test.
Date Fri, 09 Jun 2017 14:05:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044468#comment-16044468
] 

Masatake Iwasaki commented on HADOOP-14424:
-------------------------------------------

Thanks for working on this, [~GeLiXin].

It would be nice if duplicate code in Crc32PerformanceTest and Crc32CPerformanceTest. I think
it is ok to just add test cases for CRC32C to existing Crc32PerformanceTest. In that case,
displaying diffs to all of previous benchmark results could be noisy. It should be omitted
or just print diff to one of the results.
{noformat}
//compare result with previous
for(BenchResult p : previous) {
          final double diff = (result.mbps - p.mbps) / p.mbps * 100;
          printCell(String.format("%5.1f%%", diff), diffStr.length(), out);
        }
        previous.add(result);
      }
{noformat}

> Add CRC32C performance test.
> ----------------------------
>
>                 Key: HADOOP-14424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14424
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: common
>    Affects Versions: 3.0.0-alpha2
>            Reporter: LiXin Ge
>            Assignee: LiXin Ge
>            Priority: Minor
>              Labels: test
>         Attachments: HADOOP-14424.patch
>
>
> The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new test to
compare Crc32C chunked verification implementations.
> This test is based on Crc32PerformanceTest, what I have done in this test is:
> 1.CRC32C performance test.
> 2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it from this test.
> 3.User can choose either direct buffer or non-directly buffer to run this test manually.
> 4.Take use of verifyChunkedSumsByteArray for native to support non-directly native test.
> The test result in my environment is:
> [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
>                  java.version = 1.8.0_111
>             java.runtime.name = Java(TM) SE Runtime Environment
>          java.runtime.version = 1.8.0_111-b14
>               java.vm.version = 25.111-b14
>                java.vm.vendor = Oracle Corporation
>                  java.vm.name = Java HotSpot(TM) 64-Bit Server VM
> java.vm.specification.version = 1.8
>    java.specification.version = 1.8
>                       os.arch = amd64
>                       os.name = Linux
>                    os.version = 2.6.33.20
> Data Length = 64 MB
> Trials      = 5
> Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)
> |  bpc  | #T || PureJava ||   Native | % diff |
> |    32 |  1 |     394.0 |    4156.2 | 954.9% |
> |    32 |  2 |     400.5 |    3679.7 | 818.7% |
> |    32 |  4 |     401.8 |    2657.0 | 561.3% |
> |    32 |  8 |     389.1 |    1633.8 | 319.9% |
> |    32 | 16 |     222.2 |    1116.3 | 402.5% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |    64 |  1 |     465.0 |    5931.0 | 1175.5% |
> |    64 |  2 |     468.8 |    1839.2 | 292.3% |
> |    64 |  4 |     460.4 |    2968.3 | 544.7% |
> |    64 |  8 |     452.4 |    1925.7 | 325.6% |
> |    64 | 16 |     246.9 |    1291.8 | 423.3% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   128 |  1 |     522.0 |    6147.8 | 1077.6% |
> |   128 |  2 |     366.0 |    4758.5 | 1200.2% |
> |   128 |  4 |     307.8 |    3265.1 | 960.8% |
> |   128 |  8 |     283.6 |    2092.2 | 637.6% |
> |   128 | 16 |     219.9 |    1226.1 | 457.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   256 |  1 |     550.7 |    3177.6 | 477.0% |
> |   256 |  2 |     538.6 |    1933.2 | 258.9% |
> |   256 |  4 |     427.2 |    3278.1 | 667.3% |
> |   256 |  8 |     420.8 |    2272.3 | 440.0% |
> |   256 | 16 |     294.0 |    1311.2 | 346.0% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |   512 |  1 |     553.4 |    3690.4 | 566.9% |
> |   512 |  2 |     455.6 |    4974.1 | 991.7% |
> |   512 |  4 |     494.2 |    3406.4 | 589.2% |
> |   512 |  8 |     431.4 |    2257.0 | 423.2% |
> |   512 | 16 |     316.3 |    1272.0 | 302.2% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  1024 |  1 |     566.1 |    3520.0 | 521.8% |
> |  1024 |  2 |     508.7 |    4437.4 | 772.3% |
> |  1024 |  4 |     520.7 |    3422.6 | 557.4% |
> |  1024 |  8 |     501.8 |    2124.7 | 323.4% |
> |  1024 | 16 |     340.6 |    1305.0 | 283.2% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  2048 |  1 |     535.1 |    5438.5 | 916.4% |
> |  2048 |  2 |     537.3 |    4668.3 | 768.8% |
> |  2048 |  4 |     529.2 |    2417.2 | 356.7% |
> |  2048 |  8 |     485.1 |    2249.8 | 363.8% |
> |  2048 | 16 |     334.3 |    1265.7 | 278.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  4096 |  1 |     563.0 |    7264.0 | 1190.1% |
> |  4096 |  2 |     538.8 |    5681.4 | 954.4% |
> |  4096 |  4 |     528.9 |    3107.6 | 487.5% |
> |  4096 |  8 |     521.8 |    2168.9 | 315.6% |
> |  4096 | 16 |     336.5 |    1256.5 | 273.4% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> |  8192 |  1 |     521.1 |    6842.4 | 1213.1% |
> |  8192 |  2 |     543.0 |    5510.2 | 914.8% |
> |  8192 |  4 |     526.1 |    3759.5 | 614.6% |
> |  8192 |  8 |     480.1 |    2254.1 | 369.6% |
> |  8192 | 16 |     336.4 |    1213.3 | 260.6% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 16384 |  1 |     521.7 |    3695.7 | 608.3% |
> | 16384 |  2 |     540.8 |    5610.5 | 937.4% |
> | 16384 |  4 |     539.1 |    3313.0 | 514.6% |
> | 16384 |  8 |     518.7 |    2198.4 | 323.8% |
> | 16384 | 16 |     337.2 |    1237.8 | 267.1% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 32768 |  1 |     532.1 |    3696.0 | 594.6% |
> | 32768 |  2 |     541.4 |    4647.4 | 758.4% |
> | 32768 |  4 |     536.6 |    2325.1 | 333.3% |
> | 32768 |  8 |     518.8 |    2036.9 | 292.6% |
> | 32768 | 16 |     337.8 |    1232.0 | 264.7% |
> |  bpc  | #T || PureJava ||   Native | % diff |
> | 65536 |  1 |     527.7 |    8240.0 | 1461.4% |
> | 65536 |  2 |     561.9 |    5472.3 | 873.9% |
> | 65536 |  4 |     536.5 |    3725.2 | 594.4% |
> | 65536 |  8 |     482.8 |    2040.2 | 322.5% |
> | 65536 | 16 |     335.9 |    1240.4 | 269.3% |
> Elapsed 77.4s



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message