hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
Date Tue, 11 Aug 2009 18:00:14 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741993#action_12741993
] 

Scott Carey commented on HADOOP-6166:
-------------------------------------

We probably want to have Todd's concurrency test from HADOOP-5318 run to make sure the larger
lookup table doesn't slow things down under concurrency.

We might also want to try the old four at a time code for 4 <= len < 8.

We should also confirm the results on one of the other systems we tested in the past. I won't
be able to do that for a couple days, but it should be easy then.
How many other variants did you try? Intel's C code does some strange things to group CRC's
by 3. Maybe something like the below is worth testing if it hasn't already been done haven't
already:


{code}+public class Crc32_8_8e extends Crc32Base {
  public void update(byte[] b, int off, int len) {
    while(len > 7) {
      int c0 = b[off++] ^ crc;
      int c1 = b[off++] ^ (crc >>>= 8);
      int c2 = b[off++] ^ (crc >>>= 8);
      int c3 = b[off++] ^ (crc >>>= 8);
     
      crc = T8_7[c0 & 0xff] ^ T8_6[c1 & 0xff] ^ T8_5[c2 & 0xff] ^ T8_4[c3 &
0xff]; // three xors

      crc ^= T8_3[b[off++] & 0xff] ^ T8_2[b[off++] & 0xff]; // two xors

      crc ^= T8_1[b[off++] & 0xff] ^ T8_0[b[off++] & 0xff]; // two xors

      len -= 8;
    }
    while(len > 0) {
      crc = (crc >>> 8) ^ T8_0[(crc ^ b[off++]) & 0xff];
      len--;
    }
  }
}{code}


> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt,
c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message