hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32 calculator
Date Wed, 15 Jul 2009 23:45:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731747#action_12731747
] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6148:
------------------------------------------------

Since the benchmark program is very useful, we may combine TestCrc32Performance with TestPureJavaCrc32.
 Then, the benchmark can be executed by something like "java TestPureJavaCrc32" in the future.

Todd, could you post a new patch?

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt,
hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt,
PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java,
PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java,
TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time
in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total
of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message