cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vijay (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3610) Checksum improvement for CompressedRandomAccessReader
Date Thu, 22 Dec 2011 20:29:32 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175035#comment-13175035
] 

Vijay commented on CASSANDRA-3610:
----------------------------------

Ooops pasted the wrong data.... the above data is without any Heap settings hence GC becomes
a bottleneck... Plz see the below :)


/usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G
-Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-jar TestCRC32Performance.jar 


||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1	|121.124 	|11.866 	|
| 2	|161.981 	|23.851 	|
| 4	|204.718 	|45.486 	|
| 8	|297.229 	|76.296 	|
| 16	|379.268 	|117.326 	|
| 32	|440.153 	|157.711 	|
| 64	|468.143 	|193.304 	|| PureJava	|0-64	|272.921 MB/sec|| Native	|0-64	|145.289 MB/sec|
| 128	|500.006 	|219.657 	|| PureJava	|0-128	|367.816 MB/sec|| Native	|0-128	|186.861 MB/sec|
| 256	|511.572 	|234.052 	|| PureJava	|0-256	|432.433 MB/sec|| Native	|0-256	|214.047 MB/sec|
| 512	|517.550 	|242.634 	|| PureJava	|0-512	|474.074 MB/sec|| Native	|0-512	|231.047 MB/sec|
| 1024	|516.994 	|246.424 	|| PureJava	|0-1024	|498.055 MB/sec|| Native	|0-1024	|241.056 MB/sec|
| 2048	|518.095 	|248.529 	|| PureJava	|0-2048	|509.960 MB/sec|| Native	|0-2048	|245.683 MB/sec|
| 4096	|522.002 	|249.755 	|| PureJava	|0-4096	|518.226 MB/sec|| Native	|0-4096	|248.062 MB/sec|
| 8192	|522.795 	|250.316 	|| PureJava	|0-8192	|520.326 MB/sec|| Native	|0-8192	|249.519 MB/sec|
| 16384	|522.521 	|250.484 	|| PureJava	|0-16384	|522.480 MB/sec|| Native	|0-16384	|250.002
MB/sec|
| 32768	|521.098 	|250.604 	|| PureJava	|0-32768	|520.349 MB/sec|| Native	|0-32768	|250.494
MB/sec|
| 65536	|520.973 	|250.837 	|| PureJava	|0-65536	|520.392 MB/sec|| Native	|0-65536	|249.063
MB/sec|
| 131072	|510.129 	|248.949 	|| PureJava	|0-131072	|516.246 MB/sec|| Native	|0-131072	|249.535
MB/sec|
| 262144	|513.534 	|249.506 	|| PureJava	|0-262144	|514.407 MB/sec|| Native	|0-262144	|250.617
MB/sec|
| 524288	|519.554 	|250.696 	|| PureJava	|0-524288	|520.402 MB/sec|| Native	|0-524288	|251.048
MB/sec|
| 1048576	|519.559 	|250.557 	|| PureJava	|0-1048576	|520.403 MB/sec|| Native	|0-1048576	|250.734
MB/sec|
| 2097152	|519.259 	|250.456 	|| PureJava	|0-2097152	|519.337 MB/sec|| Native	|0-2097152	|250.299
MB/sec|
| 4194304	|518.649 	|250.470 	|| PureJava	|0-4194304	|518.495 MB/sec|| Native	|0-4194304	|250.523
MB/sec|
| 8388608	|501.986 	|248.044 	|| PureJava	|0-8388608	|509.521 MB/sec|| Native	|0-8388608	|248.626
MB/sec|
| 16777216	|508.201 	|247.587 	|| PureJava	|0-16777216	|505.258 MB/sec|| Native	|0-16777216
|249.558 MB/sec|

                
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
>                 Key: CASSANDRA-3610
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.1
>         Environment: JVM
>            Reporter: Vijay
>            Assignee: Vijay
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more
than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it
or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni
to call the OS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message