hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
Date Wed, 19 Aug 2009 20:32:14 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090819.patch

> I tried this before but it did not help. ...
I should be more clear: it did help for the 4-byte case but it hurt the performance in some
other cases.

c6166_20090819.patch: added Crc32_8_8b2 and Crc32_8_8d2.  See whether anyone wants to play
with them.

Here is my results.
- java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8b || Crc32_8_8b2 || Crc32_8_8d || Crc32_8_8d2
||
|          1 |     7.236 |        106.903 |      58.534 |       61.106 |     109.742 |   
   67.673 |
|          2 |    14.605 |        104.665 |     104.573 |      104.952 |     115.005 |   
  102.944 |
|          4 |    27.566 |        177.502 |     223.599 |      228.648 |     131.890 |   
  218.530 |
|          8 |    49.708 |        193.198 |     196.794 |      175.006 |     251.469 |   
  204.796 |
|         16 |    83.804 |        259.008 |     234.906 |      221.473 |     258.878 |   
  236.937 |
|         32 |   128.918 |        313.157 |     299.622 |      273.306 |     322.913 |   
  300.376 |
|         64 |   176.797 |        349.738 |     339.404 |      320.543 |     365.522 |   
  348.328 |
|        128 |   217.087 |        370.589 |     362.759 |      353.613 |     390.209 |   
  384.627 |
|        256 |   244.869 |        381.630 |     376.096 |      369.157 |     405.767 |   
  401.649 |
|        512 |   261.282 |        385.554 |     382.974 |      378.841 |     413.286 |   
  411.211 |
|       1024 |   271.205 |        390.170 |     384.064 |      384.586 |     415.817 |   
  413.505 |
|       2048 |   276.269 |        392.448 |     386.215 |      387.585 |     418.559 |   
  417.466 |
|       4096 |   279.280 |        393.668 |     387.571 |      388.814 |     418.777 |   
  419.686 |
|       8192 |   280.938 |        394.176 |     388.399 |      389.576 |     420.204 |   
  420.571 |
|      16384 |   281.385 |        393.164 |     389.518 |      388.856 |     420.236 |   
  420.392 |
|      32768 |   281.442 |        391.935 |     389.115 |      387.306 |     419.440 |   
  417.362 |
|      65536 |   281.605 |        391.986 |     389.124 |      387.297 |     419.405 |   
  417.123 |
|     131072 |   281.334 |        393.221 |     389.093 |      384.810 |     417.836 |   
  415.334 |
|     262144 |   280.363 |        391.701 |     386.141 |      385.937 |     417.907 |   
  415.452 |
|     524288 |   280.373 |        391.727 |     387.646 |      385.911 |     417.851 |   
  415.510 |
|    1048576 |   280.295 |        391.585 |     387.535 |      385.745 |     417.587 |   
  414.222 |
|    2097152 |   280.866 |        391.217 |     387.413 |      385.603 |     417.605 |   
  414.120 |
|    4194304 |   280.273 |        390.247 |     387.650 |      385.971 |     418.284 |   
  414.302 |
|    8388608 |   279.799 |        388.960 |     385.093 |      383.170 |     414.179 |   
  412.095 |
|   16777216 |   279.473 |        388.359 |     384.629 |      382.737 |     413.717 |   
  411.523 |

- java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8b || Crc32_8_8b2 || Crc32_8_8d || Crc32_8_8d2
||
|          1 |     3.744 |         49.901 |      47.314 |       49.920 |      51.063 |   
   45.306 |
|          2 |     7.397 |         78.714 |      64.757 |       63.904 |      69.411 |   
   69.742 |
|          4 |    14.129 |        118.892 |     116.559 |      122.589 |     100.686 |   
  120.545 |
|          8 |    27.169 |        141.788 |     178.545 |      179.537 |     177.252 |   
  176.913 |
|         16 |    48.864 |        197.479 |     199.851 |      206.187 |     205.112 |   
  197.922 |
|         32 |    87.972 |        242.980 |     252.241 |      243.120 |     248.911 |   
  230.985 |
|         64 |   128.459 |        262.461 |     272.747 |      284.252 |     276.611 |   
  264.473 |
|        128 |   178.257 |        265.661 |     305.895 |      303.167 |     293.562 |   
  286.069 |
|        256 |   213.740 |        280.286 |     307.318 |      324.065 |     296.782 |   
  297.760 |
|        512 |   247.829 |        287.490 |     334.125 |      332.208 |     317.107 |   
  310.138 |
|       1024 |   266.095 |        289.453 |     343.593 |      340.496 |     322.506 |   
  330.703 |
|       2048 |   284.693 |        286.926 |     328.546 |      338.577 |     323.831 |   
  324.033 |
|       4096 |   292.326 |        304.553 |     354.299 |      351.903 |     332.555 |   
  334.845 |
|       8192 |   276.416 |        286.101 |     343.856 |      336.716 |     327.936 |   
  317.006 |
|      16384 |   284.944 |        292.414 |     338.566 |      333.038 |     323.622 |   
  314.212 |
|      32768 |   275.270 |        292.939 |     339.562 |      332.072 |     332.398 |   
  294.426 |
|      65536 |   271.285 |        294.269 |     336.092 |      339.905 |     327.122 |   
  306.583 |
|     131072 |   288.692 |        290.441 |     337.711 |      312.378 |     240.516 |   
  299.113 |
|     262144 |   288.789 |        290.830 |     342.317 |      331.144 |     322.446 |   
  312.372 |
|     524288 |   277.527 |        288.056 |     337.654 |      329.802 |     326.141 |   
  301.740 |
|    1048576 |   278.305 |        288.149 |     332.875 |      328.402 |     320.754 |   
  300.196 |
|    2097152 |   278.992 |        291.338 |     324.239 |      325.872 |     324.888 |   
  294.238 |
|    4194304 |   282.476 |        281.645 |     331.034 |      332.346 |     318.677 |   
  296.442 |
|    8388608 |   283.086 |        287.335 |     328.317 |      331.580 |     325.669 |   
  306.567 |
|   16777216 |   290.031 |        296.875 |     339.540 |      332.101 |     323.918 |   
  307.348 |


> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt,
c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch,
graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf,
Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message