cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)
Date Fri, 15 Sep 2017 11:13:42 GMT
Most people find 3.0 slightly slower than 2.1. The only thing that really stands out in your
email is the huge change in 95% latency - that's atypical. Are you using thrift or native
9042)?  The decrease in compression metadata offheap usage is likely due to the increased
storage efficiency of the storage engine (see Cassandra-8099).


-- 
Jeff Jirsa


> On Sep 15, 2017, at 2:37 AM, Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com>
wrote:
> 
> Hello,
>  
> we have a test (regression) environment hosted in AWS, which is used for auto deploying
our software on a daily basis and attach constant load across all deployments. Basically to
allow us to detect any regressions in our software on a daily basis.
>  
> On the Cassandra-side, this is single-node in AWS, m4.xlarge, EBS gp2, 8G heap, CMS.
The environment has also been upgraded from Cassandra 2.1.18 to 3.0.14 at a certain point
in time. Without running upgradesstables so far. We have not made any additional JVM/GC configuration
change when going from 2.1.18 to 3.0.14 on our own, thus, any self-made configuration changes
(e.g. new gen heap size) for 2.1.18 are also in place with 3.0.14.
>  
> What we see after a time-frame of ~ 7 days (so, e.g. should not be caused by some sort
of spiky compaction pattern) is an AVG increase in GC/CPU (most likely correlating):
> ·         CPU: ~ 12% => ~ 17%
> ·         GC Suspension: ~ 1,7% => 3,29%
>  
> In this environment not a big deal, but relatively we have a CPU increase of ~ 50% (with
increased GC most likely contributing). Something we have deal with when going into production
(going into larger, multi-node loadtest environments first though).
>  
> Beside the CPU/GC shift, we also monitor the following noticeable changes (don’t know
if they somehow correlate with the CPU/GC shift above):
> ·         Increased AVG Write Client Requests Latency (95th Percentile), org.apache.cassandra.metrics.ClientRequest.Latency.Write:
6,05ms => 29,2ms, but almost constant (no change in) write client request latency for our
particular keyspace only, org.apache.cassandra.metrics.Keyspace.ruxitdb.WriteLatency
> ·         Compression metadata memory usage drop, org.apache.cassandra.metrics.Keyspace.XXX.
CompressionMetadataOffHeapMemoryUsed: ~218MB => ~105MB => Good or bad? Known?
>  
> I know, looks all a bit vague, but perhaps someone else has seen something similar when
upgrading to 3.0.14 and can share their thoughts/ideas. Especially the (relative) CPU/GC increase
is something we are curious about.
>  
> Thanks a lot.
>  
> Thomas
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313

Mime
View raw message