hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
Date Fri, 14 Aug 2009 02:20:14 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated HADOOP-6166:

    Attachment: graph.r

Attaching PDF of the concurrency benchmark from HADOOP-5318 run on all of these implementations.
The first page also includes java.util.zip.CRC32 to verify that the benchmark script really
is testing something and that we've fixed the problem originally reported in that ticket.
The second page of the PDF is the same graph after dropping the super-slow one.

I've normalized all of the results as percentages of the very fastest. It looks like they're
all within 5-10% of each other for this benchmark, and with the error bars it's pretty obvious
that there is no clear winner for the 1-byte-write case at any concurrency level. If anything
Crc32_6_6 seems to come out a little ahead, but really barely discernible.

I'd upload the script I used to run the benchmarks, but I had to add a silly little patch
to Hadoop to be able to switch between implementations easily, so it's a hassle to run.

For reference, this system is a Nehalem box running 64-bit JDK 1.6.0u14, 8 cores w/ hyperthreading
(16 logical).

> Improve PureJavaCrc32
> ---------------------
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt,
c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r,
> Got some ideas to improve CRC32 calculation.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message