hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
Date Mon, 10 Aug 2009 22:46:14 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090810.patch

>> I think there may be some improvement possible in the 8_8 case in how the 9 XORs
at the end are done. ...
>Yeah, we should try.

c6166_20090810.patch: tried various xor schemes for 8_8 and 16_16.

Seems Crc32_8_8d is the best choice.  I will generate a patch to replace current PureJavaCrc32
unless someone would like to run some benchmarks.

- Linux
java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d
|| Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          8 |    49.617 |        186.277 |    245.369 |     258.905 |     229.094 |     246.095
|      217.684 |       209.102 |       225.104 |       216.537 |
|         16 |    83.794 |        260.552 |    254.068 |     253.335 |     240.414 |     269.863
|      268.293 |       280.991 |       248.725 |       272.588 |
|         32 |   129.851 |        311.725 |    319.791 |     317.253 |     300.914 |     325.145
|      294.224 |       306.928 |       294.640 |       294.838 |
|         64 |   175.566 |        348.701 |    357.292 |     359.353 |     345.997 |     370.976
|      325.410 |       349.629 |       343.592 |       346.502 |
|        128 |   217.394 |        369.562 |    386.217 |     384.014 |     371.929 |     392.099
|      339.749 |       382.753 |       378.737 |       381.048 |
|        256 |   246.060 |        381.159 |    403.724 |     401.239 |     389.683 |     403.455
|      346.794 |       394.901 |       397.287 |       398.093 |
|        512 |   261.941 |        385.598 |    412.253 |     405.512 |     395.790 |     412.212
|      350.763 |       409.029 |       408.213 |       407.630 |
|       1024 |   271.043 |        390.318 |    408.391 |     408.592 |     398.981 |     413.389
|      352.213 |       414.423 |       414.213 |       413.736 |
|       2048 |   275.942 |        391.870 |    412.933 |     411.371 |     401.784 |     417.020
|      351.803 |       418.063 |       416.982 |       415.269 |
|       4096 |   280.018 |        393.816 |    412.645 |     411.781 |     402.632 |     418.353
|      353.348 |       420.783 |       416.263 |       416.360 |
|       8192 |   281.432 |        394.699 |    414.998 |     410.618 |     401.145 |     417.090
|      351.428 |       420.008 |       415.680 |       415.281 |
|      16384 |   279.658 |        391.788 |    413.657 |     411.491 |     402.747 |     418.249
|      349.888 |       419.431 |       414.399 |       414.312 |
|      32768 |   279.788 |        391.665 |    408.767 |     410.097 |     403.396 |     417.542
|      349.473 |       418.467 |       409.388 |       412.525 |
|      65536 |   280.168 |        391.557 |    411.674 |     410.447 |     404.971 |     419.271
|      350.824 |       419.866 |       412.099 |       413.847 |
|     131072 |   281.196 |        393.222 |    411.687 |     411.539 |     404.931 |     418.101
|      350.692 |       418.924 |       412.873 |       413.763 |
|     262144 |   281.874 |        392.158 |    411.668 |     411.660 |     405.089 |     419.301
|      350.236 |       418.776 |       412.855 |       413.918 |
|     524288 |   281.905 |        392.967 |    411.713 |     410.524 |     404.851 |     418.481
|      350.890 |       419.835 |       412.881 |       412.677 |
|    1048576 |   281.864 |        393.155 |    411.589 |     410.697 |     405.352 |     417.905
|      350.737 |       419.861 |       412.734 |       413.808 |
|    2097152 |   281.269 |        392.916 |    410.495 |     411.322 |     405.298 |     419.084
|      349.765 |       419.540 |       412.517 |       413.538 |
|    4194304 |   280.904 |        388.513 |    408.313 |     410.383 |     404.432 |     418.168
|      349.982 |       418.628 |       411.404 |       412.493 |
|    8388608 |   279.946 |        389.306 |    407.485 |     407.420 |     401.088 |     415.597
|      347.824 |       415.479 |       408.459 |       409.290 |
|   16777216 |   279.517 |        388.903 |    407.068 |     406.760 |     400.404 |     415.171
|      347.468 |       415.047 |       408.089 |       408.576 |

- Windows
java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||     CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d
|| Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          8 |    30.582 |        165.684 |    222.712 |     239.031 |     192.301 |     192.855
|      121.520 |       118.156 |       121.212 |       119.522 |
|         16 |    54.720 |        215.567 |    237.607 |     232.193 |     230.172 |     215.288
|      269.855 |       279.729 |       254.891 |       251.049 |
|         32 |    93.370 |        250.110 |    290.161 |     300.234 |     272.414 |     266.170
|      288.042 |       293.170 |       273.936 |       278.459 |
|         64 |   142.510 |        271.193 |    317.292 |     331.031 |     293.492 |     298.169
|      314.142 |       310.036 |       309.833 |       315.505 |
|        128 |   193.336 |        285.876 |    340.127 |     355.663 |     302.081 |     319.978
|      330.662 |       318.949 |       326.466 |       333.227 |
|        256 |   237.137 |        292.450 |    349.322 |     366.198 |     308.711 |     328.690
|      335.694 |       323.855 |       328.766 |       342.473 |
|        512 |   266.200 |        293.008 |    352.708 |     370.587 |     307.753 |     330.876
|      335.443 |       320.305 |       338.866 |       342.883 |
|       1024 |   275.926 |        292.009 |    347.310 |     367.795 |     305.459 |     332.800
|      328.240 |       291.745 |       270.544 |       338.249 |
|       2048 |   285.404 |        282.665 |    338.805 |     356.377 |     300.794 |     333.204
|      334.669 |       317.972 |       327.810 |       347.768 |
|       4096 |   295.626 |        296.351 |    353.792 |     374.937 |     310.589 |     337.225
|      338.851 |       323.519 |       335.347 |       349.135 |
|       8192 |   299.106 |        297.335 |    352.159 |     371.486 |     310.073 |     334.584
|      336.462 |       320.744 |       340.072 |       343.119 |
|      16384 |   298.202 |        295.121 |    353.141 |     371.399 |     311.208 |     347.209
|      340.819 |       321.550 |       341.870 |       349.779 |
|      32768 |   301.017 |        294.772 |    355.933 |     374.572 |     311.367 |     354.163
|      335.798 |       320.324 |       344.780 |       351.859 |
|      65536 |   305.349 |        297.090 |    355.641 |     377.599 |     310.232 |     354.429
|      337.837 |       324.306 |       345.244 |       350.304 |
|     131072 |   305.814 |        297.094 |    356.518 |     380.588 |     310.183 |     354.515
|      339.324 |       319.657 |       344.429 |       347.836 |
|     262144 |   303.674 |        294.812 |    352.616 |     377.256 |     306.132 |     349.959
|      330.275 |       319.694 |       339.153 |       347.561 |
|     524288 |   306.440 |        296.660 |    352.876 |     371.992 |     308.378 |     353.454
|      339.735 |       322.835 |       344.267 |       350.561 |
|    1048576 |   303.807 |        297.499 |    351.199 |     375.630 |     307.785 |     350.496
|      338.899 |       321.673 |       342.487 |       350.763 |
|    2097152 |   302.701 |        295.785 |    352.618 |     376.966 |     306.529 |     351.045
|      335.405 |       318.857 |       340.192 |       349.454 |
|    4194304 |   300.318 |        293.945 |    345.822 |     373.705 |     306.829 |     348.913
|      334.297 |       318.580 |       339.454 |       346.533 |
|    8388608 |   299.847 |        293.259 |    348.667 |     373.015 |     305.414 |     338.622
|      333.332 |       317.337 |       339.085 |       346.144 |
|   16777216 |   298.157 |        293.100 |    349.116 |     372.502 |     306.273 |     348.192
|      334.259 |       316.192 |       336.934 |       347.545 |

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt,
c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message