cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vijay (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3610) Checksum improvement for CompressedRandomAccessReader
Date Thu, 22 Dec 2011 20:53:30 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175050#comment-13175050
] 

Vijay commented on CASSANDRA-3610:
----------------------------------

I might have miss commuicated the results... here is the results without any setting. SunJDK
vs OpenJDK is the only difference.

[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ java -jar TestCRC32Performance.jar 
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1	|40.582 	|11.678 	|
| 2	|152.297 	|23.060 	|
| 4	|206.689 	|46.501 	|
| 8	|311.654 	|87.895 	|
| 16	|333.174 	|151.716 	|
| 32	|386.252 	|246.780 	|
| 64	|414.639 	|353.623 	|| PureJava	|0-64	|256.513 MB/sec|| Native	|0-64	|196.319 MB/sec|
| 128	|429.995 	|449.165 	|| PureJava	|0-128	|324.051 MB/sec|| Native	|0-128	|306.220 MB/sec|
| 256	|438.206 	|519.658 	|| PureJava	|0-256	|377.582 MB/sec|| Native	|0-256	|410.256 MB/sec|
| 512	|442.210 	|565.027 	|| PureJava	|0-512	|408.946 MB/sec|| Native	|0-512	|494.210 MB/sec|
| 1024	|453.627 	|591.082 	|| PureJava	|0-1024	|428.094 MB/sec|| Native	|0-1024	|551.725 MB/sec|
| 2048	|455.738 	|604.581 	|| PureJava	|0-2048	|436.861 MB/sec|| Native	|0-2048	|573.995 MB/sec|
| 4096	|443.916 	|610.058 	|| PureJava	|0-4096	|444.456 MB/sec|| Native	|0-4096	|595.364 MB/sec|
| 8192	|449.964 	|613.482 	|| PureJava	|0-8192	|446.008 MB/sec|| Native	|0-8192	|606.647 MB/sec|
| 16384	|447.484 	|615.583 	|| PureJava	|0-16384	|446.013 MB/sec|| Native	|0-16384	|609.534
MB/sec|
| 32768	|446.452 	|616.336 	|| PureJava	|0-32768	|447.556 MB/sec|| Native	|0-32768	|612.477
MB/sec|
| 65536	|446.351 	|616.431 	|| PureJava	|0-65536	|446.171 MB/sec|| Native	|0-65536	|612.508
MB/sec|
| 131072	|446.457 	|616.208 	|| PureJava	|0-131072	|444.713 MB/sec|| Native	|0-131072	|615.538
MB/sec|
| 262144	|445.521 	|613.368 	|| PureJava	|0-262144	|446.434 MB/sec|| Native	|0-262144	|615.559
MB/sec|
| 524288	|445.233 	|612.942 	|| PureJava	|0-524288	|444.488 MB/sec|| Native	|0-524288	|613.167
MB/sec|
| 1048576	|445.278 	|613.044 	|| PureJava	|0-1048576	|445.561 MB/sec|| Native	|0-1048576	|614.500
MB/sec|
| 2097152	|443.896 	|611.457 	|| PureJava	|0-2097152	|440.685 MB/sec|| Native	|0-2097152	|601.230
MB/sec|
| 4194304	|425.962 	|595.604 	|| PureJava	|0-4194304	|440.787 MB/sec|| Native	|0-4194304	|610.587
MB/sec|
| 8388608	|430.688 	|608.036 	|| PureJava	|0-8388608	|439.621 MB/sec|| Native	|0-8388608	|608.256
MB/sec|
| 16777216	|436.099 	|599.706 	|| PureJava	|0-16777216	|435.695 MB/sec|| Native	|0-16777216
|603.244 MB/sec|
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -jar TestCRC32Performance.jar
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1	|78.345 	|10.897 	|
| 2	|157.171 	|20.341 	|
| 4	|197.572 	|40.119 	|
| 8	|295.368 	|67.051 	|
| 16	|379.664 	|104.782 	|
| 32	|440.257 	|147.490 	|
| 64	|478.970 	|187.356 	|| PureJava	|0-64	|268.908 MB/sec|| Native	|0-64	|141.436 MB/sec|
| 128	|495.235 	|211.237 	|| PureJava	|0-128	|363.636 MB/sec|| Native	|0-128	|182.596 MB/sec|
| 256	|511.961 	|231.697 	|| PureJava	|0-256	|432.433 MB/sec|| Native	|0-256	|211.921 MB/sec|
| 512	|517.852 	|240.495 	|| PureJava	|0-512	|475.837 MB/sec|| Native	|0-512	|229.803 MB/sec|
| 1024	|516.815 	|246.001 	|| PureJava	|0-1024	|496.127 MB/sec|| Native	|0-1024	|240.152 MB/sec|
| 2048	|520.283 	|248.616 	|| PureJava	|0-2048	|509.963 MB/sec|| Native	|0-2048	|245.212 MB/sec|
| 4096	|521.950 	|249.806 	|| PureJava	|0-4096	|518.219 MB/sec|| Native	|0-4096	|247.586 MB/sec|
| 8192	|516.635 	|249.594 	|| PureJava	|0-8192	|520.342 MB/sec|| Native	|0-8192	|249.513 MB/sec|
| 16384	|522.743 	|250.757 	|| PureJava	|0-16384	|522.467 MB/sec|| Native	|0-16384	|250.493
MB/sec|
| 32768	|521.264 	|250.807 	|| PureJava	|0-32768	|520.336 MB/sec|| Native	|0-32768	|250.503
MB/sec|
| 65536	|521.192 	|250.916 	|| PureJava	|0-65536	|520.443 MB/sec|| Native	|0-65536	|250.505
MB/sec|
| 131072	|520.264 	|249.842 	|| PureJava	|0-131072	|514.098 MB/sec|| Native	|0-131072	|250.517
MB/sec|
| 262144	|519.953 	|250.719 	|| PureJava	|0-262144	|520.510 MB/sec|| Native	|0-262144	|249.682
MB/sec|
| 524288	|512.457 	|250.282 	|| PureJava	|0-524288	|516.266 MB/sec|| Native	|0-524288	|250.072
MB/sec|
| 1048576	|514.005 	|249.813 	|| PureJava	|0-1048576	|516.444 MB/sec|| Native	|0-1048576	|250.548
MB/sec|
| 2097152	|519.207 	|250.680 	|| PureJava	|0-2097152	|519.260 MB/sec|| Native	|0-2097152	|250.526
MB/sec|
| 4194304	|518.882 	|250.608 	|| PureJava	|0-4194304	|517.022 MB/sec|| Native	|0-4194304	|250.538
MB/sec|
| 8388608	|517.444 	|250.182 	|| PureJava	|0-8388608	|518.502 MB/sec|| Native	|0-8388608	|249.994
MB/sec|
| 16777216	|511.875 	|249.008 	|| PureJava	|0-16777216	|514.823 MB/sec|| Native	|0-16777216
|249.477 MB/sec|
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ java -version
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.21.b17.el5-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -versionjava
version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ 



                
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
>                 Key: CASSANDRA-3610
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.1
>         Environment: JVM
>            Reporter: Vijay
>            Assignee: Vijay
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch,
TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more
than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it
or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni
to call the OS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message