cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prince Nana Owusu Boateng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
Date Tue, 26 Sep 2017 16:17:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181041#comment-16181041
] 

Prince Nana Owusu Boateng commented on CASSANDRA-13896:
-------------------------------------------------------

Stress profile: $CASSANDRA_HOME/tools/bin/cassandra-stress user profile=$CASSANDRA_HOME/tools/cqlstress-insanity-example.yaml
ops\(insert=1\) no-warmup cl=ONE duration=30m -mode native cql3 -pop dist=uniform\(1..2400000000\)
-node 192.168.1.246 -rate threads=125 -errors ignore
Only change to cqlstress-insanity-example.yaml is selecting SizeTieredCompactionStrategy.

Cassandra.yaml used is the default cassandra yaml for 3.10 with the following changes: Concurrent_reads=1024;
Concurrent_writes=32 Heap size= 64GB

Dataset used:  LZ4 compression; compression chunk size of 64KB; 2.8 million entries; Insanity
data

> Improving Cassandra write performance  
> ---------------------------------------
>
>                 Key: CASSANDRA-13896
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13896
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Local Write-Read Paths
>         Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs
> OS: Centos 7.3 
> Java: Oracle JDK1.8.0_121
>            Reporter: Prince Nana Owusu Boateng
>              Labels: Performance
>             Fix For: 4.x
>
>         Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot 2017-09-22
at 3.31.09 PM.png
>
>
> During our Cassandra performance testing, we see high percentage of the CPU spent in
*org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, OpOrder Group) * method.  Appears
to be high contention of the *<nextFreeOffset>* atomic Integer in write workloads. 
 This structure is used by the threads for keeping track of the region bytebuffer allocation.
 When the contention appears, adding more clients, modifications of write specific parameters
does not change write throughput performance.  Attached are the details of Java Flight Recorder
(JFR), showing hot functions and also performance results.   When we see this contention,
we still have plenty of CPU and throughput left ( *<20%*  Total average CPU utilization
and  *<11%* of the storage write total throughput).   This occurs on Cassandra 3.10.0-src
version using the Cassandra-Stress.
> Proposal:
> We will like to introduce a solution which eliminates the atomic operations on the *<nextFreeOffset>*
atomic Integer. This implementation will allow concurrent allocation of bytebuffers without
an atomic compareAndSet and incrementAndGet operations. The solution is expected to increase
overall write performance while improving CPU utilization.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message