hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
Date Wed, 12 Aug 2009 05:07:14 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090811.patch

c6166_20090811.patch: added Crc32_8_8e and deleted some old classes.

> We probably want to have Todd's concurrency test from HADOOP-5318 run to make sure the
larger lookup table doesn't slow things down under concurrency.
Todd, could you help running the test?

> We might also want to try the old four at a time code for 4 <= len < 8.
What do you mean exactly?

> We should also confirm the results on one of the other systems we tested in the past.
I won't be able to do that for a couple days, but it should be easy then.
Hope you could find some time to do it soon.

> How many other variants did you try? Intel's C code does some strange things to group
CRC's by 3. ...
Many others.  You know, there are many combinations.  I also tried different table sizes as
shown before.

Included Crc32_8_8e below.  Crc32_8_8d still seems the best choice.


- java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d
|| Crc32_8_8e || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          1 |     7.554 |         71.591 |     79.205 |     103.936 |      80.965 |     
79.185 |      80.812 |       84.215 |        80.733 |        80.732 |        84.860 |
|          2 |    14.768 |        104.753 |    110.837 |     110.771 |     110.798 |     115.222
|     110.820 |      119.278 |       110.844 |       110.836 |       119.207 |
|          4 |    27.150 |        177.142 |    119.587 |     114.780 |     120.051 |     128.626
|     117.005 |      125.412 |       115.439 |       120.409 |       125.486 |
|          8 |    49.921 |        193.631 |    239.096 |     238.506 |     228.868 |     248.667
|     244.446 |      217.270 |       209.170 |       209.149 |       215.505 |
|         16 |    83.886 |        259.453 |    261.683 |     254.016 |     240.835 |     258.250
|     247.261 |      267.441 |       279.439 |       276.864 |       271.520 |
|         32 |   128.960 |        312.202 |    323.233 |     319.924 |     301.175 |     322.962
|     320.275 |      295.428 |       302.402 |       294.960 |       295.872 |
|         64 |   177.529 |        349.882 |    362.037 |     364.918 |     347.768 |     365.013
|     358.567 |      326.303 |       352.837 |       345.517 |       347.880 |
|        128 |   217.577 |        370.920 |    387.254 |     386.534 |     372.472 |     391.180
|     386.159 |      339.850 |       383.044 |       373.716 |       381.058 |
|        256 |   245.685 |        382.041 |    403.693 |     401.793 |     390.270 |     406.037
|     402.777 |      347.134 |       400.956 |       388.941 |       399.331 |
|        512 |   263.143 |        385.666 |    411.380 |     407.443 |     396.506 |     413.556
|     411.803 |      350.844 |       411.293 |       397.335 |       409.011 |
|       1024 |   271.941 |        390.055 |    406.949 |     406.830 |     399.163 |     415.963
|     424.095 |      352.875 |       417.516 |       401.168 |       414.229 |
|       2048 |   276.881 |        392.684 |    412.759 |     411.123 |     402.143 |     418.730
|     424.565 |      353.111 |       418.855 |       402.870 |       415.696 |
|       4096 |   279.541 |        393.738 |    418.844 |     413.271 |     403.645 |     419.106
|     424.975 |      353.582 |       421.574 |       403.834 |       417.058 |
|       8192 |   280.308 |        392.859 |    417.096 |     412.115 |     403.015 |     417.989
|     422.254 |      352.011 |       419.664 |       402.378 |       415.385 |
|      16384 |   280.420 |        393.006 |    415.629 |     409.566 |     403.498 |     418.107
|     420.676 |      350.423 |       418.062 |       401.090 |       412.468 |
|      32768 |   280.488 |        392.172 |    410.737 |     411.129 |     403.873 |     417.434
|     413.839 |      349.526 |       418.633 |       401.736 |       412.341 |
|      65536 |   281.809 |        393.565 |    411.490 |     412.331 |     405.347 |     419.190
|     414.866 |      350.781 |       418.473 |       401.637 |       412.297 |
|     131072 |   281.994 |        393.634 |    411.533 |     412.281 |     405.387 |     419.620
|     414.802 |      350.869 |       418.350 |       401.634 |       412.346 |
|     262144 |   282.106 |        393.732 |    411.339 |     411.562 |     404.172 |     416.299
|     413.072 |      349.416 |       416.190 |       400.395 |       410.680 |
|     524288 |   281.194 |        392.195 |    409.839 |     410.759 |     403.954 |     417.950
|     413.178 |      350.104 |       418.596 |       401.175 |       412.137 |
|    1048576 |   282.160 |        393.700 |    411.316 |     412.339 |     405.165 |     419.694
|     414.608 |      351.029 |       416.836 |       401.927 |       412.122 |
|    2097152 |   281.584 |        393.264 |    410.322 |     405.048 |     402.028 |     414.893
|     406.574 |      328.270 |       410.558 |       393.239 |       403.796 |
|    4194304 |   274.062 |        385.751 |    409.093 |     408.604 |     402.332 |     419.176
|     412.265 |      350.039 |       415.385 |       400.332 |       410.738 |
|    8388608 |   279.870 |        385.793 |    406.788 |     407.685 |     400.830 |     415.537
|     409.612 |      344.396 |       412.684 |       397.534 |       406.201 |
|   16777216 |   279.585 |        389.025 |    405.902 |     407.222 |     400.414 |     414.680
|     409.184 |      346.930 |       413.039 |       396.057 |       406.953 |

- java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d
|| Crc32_8_8e || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          1 |     4.669 |         62.483 |     67.995 |      64.360 |      64.700 |     
64.430 |      51.386 |       61.391 |        62.175 |        68.944 |        62.271 |
|          2 |     9.959 |         79.342 |     88.566 |      79.881 |      91.326 |     
88.695 |      81.976 |       89.846 |        83.984 |        91.166 |        84.918 |
|          4 |    19.448 |        136.996 |    119.941 |     129.939 |     120.992 |     119.898
|     132.349 |      118.682 |       116.076 |       122.505 |       114.090 |
|          8 |    36.518 |        168.433 |    240.284 |     219.896 |     213.865 |     212.471
|     209.605 |      115.325 |       135.040 |       145.009 |       139.316 |
|         16 |    66.157 |        233.146 |    272.227 |     258.220 |     250.948 |     245.771
|     253.216 |      313.174 |       321.156 |       289.904 |       285.301 |
|         32 |   111.982 |        283.081 |    327.881 |     326.083 |     311.988 |     300.416
|     299.799 |      331.685 |       338.723 |       316.987 |       319.500 |
|         64 |   169.448 |        315.802 |    364.829 |     368.059 |     348.219 |     340.445
|     323.045 |      360.273 |       357.086 |       352.497 |       359.832 |
|        128 |   227.468 |        335.743 |    384.770 |     394.813 |     373.775 |     368.234
|     338.247 |      379.686 |       363.607 |       372.900 |       382.183 |
|        256 |   274.901 |        348.319 |    395.674 |     410.081 |     386.566 |     388.908
|     345.215 |      385.142 |       368.920 |       383.446 |       392.874 |
|        512 |   307.317 |        353.626 |    401.379 |     422.212 |     392.543 |     399.756
|     348.747 |      391.548 |       372.486 |       389.674 |       400.035 |
|       1024 |   325.610 |        358.707 |    408.000 |     424.190 |     390.858 |     386.952
|     349.863 |      392.606 |       373.485 |       393.192 |       403.376 |
|       2048 |   337.261 |        361.642 |    410.563 |     429.318 |     395.794 |     388.887
|     351.070 |      392.865 |       373.458 |       394.564 |       404.084 |
|       4096 |   346.026 |        363.077 |    408.908 |     432.169 |     402.087 |     394.633
|     351.254 |      393.764 |       372.779 |       395.751 |       405.531 |
|       8192 |   348.807 |        363.550 |    407.264 |     431.891 |     409.774 |     400.138
|     350.242 |      393.582 |       373.939 |       394.498 |       405.981 |
|      16384 |   350.030 |        361.773 |    407.574 |     433.393 |     413.786 |     399.135
|     352.191 |      391.875 |       371.887 |       396.563 |       404.991 |
|      32768 |   350.552 |        360.131 |    408.495 |     428.463 |     413.372 |     406.904
|     352.049 |      390.723 |       371.160 |       393.841 |       403.526 |
|      65536 |   350.624 |        359.343 |    407.846 |     427.477 |     411.844 |     407.847
|     351.866 |      388.578 |       371.861 |       394.314 |       403.752 |
|     131072 |   351.383 |        358.687 |    407.854 |     426.513 |     411.453 |     407.655
|     352.480 |      388.961 |       372.343 |       394.181 |       401.394 |
|     262144 |   351.508 |        359.392 |    406.755 |     427.817 |     413.534 |     407.467
|     352.457 |      389.811 |       370.843 |       396.034 |       404.051 |
|     524288 |   350.264 |        358.659 |    407.698 |     427.750 |     411.986 |     408.091
|     351.678 |      390.126 |       371.458 |       395.884 |       404.374 |
|    1048576 |   351.092 |        358.452 |    406.796 |     426.634 |     413.797 |     408.162
|     351.430 |      389.716 |       372.157 |       394.826 |       403.300 |
|    2097152 |   349.731 |        357.048 |    407.095 |     426.922 |     411.427 |     405.970
|     351.830 |      388.308 |       371.514 |       394.656 |       401.170 |
|    4194304 |   345.948 |        354.513 |    401.117 |     423.269 |     408.442 |     401.112
|     348.512 |      384.834 |       365.403 |       390.539 |       399.911 |
|    8388608 |   343.162 |        350.602 |    400.009 |     415.757 |     403.662 |     399.569
|     344.909 |      382.301 |       365.704 |       386.523 |       397.156 |
|   16777216 |   343.989 |        351.502 |    400.036 |     417.893 |     403.717 |     400.511
|     345.505 |      381.932 |       363.491 |       385.044 |       393.422 |

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt,
c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message