cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gosar M <koolja...@yahoo.com.INVALID>
Subject Re: Upgrade from 2.1 to 3.11
Date Sat, 01 Sep 2018 05:06:01 GMT
Hello,
Yes we encountered the same issue. See CASSANDRA-13125, CASSANDRA-12144, and 14008
The scrub helped us, but it took almost 4-5 hrs one 1 table of 145GB per node. We are still
scrubbing our table to resolve this issue. The next step is to upgrade to next version as
well.

Thank you

   On Friday, 31 August 2018, 21:20:10 GMT-7, Mun Dega <mundega97@gmail.com> wrote:
 
 
 I think I narrowed down the constant Full GC due to accumulation of large partitions as a
result of upgrade to 3.11.And how the large partitions were produced maybe related to https://issues.apache.org/jira/browse/CASSANDRA-11887
I started to see duplicate records in one of the tables which probably expanded the size of
already semi large enough partitions to well over 300, 500MB and some cases 800MB.  My system.log
started to report Writing large partition during compaction quite frequently after the upgrade.
On CASSANDRA-11887, the suggestion is to run nodetool scrub which I will try next to see if
dup records are purged out.
Is anyone else familiar with this problem?  It says scrub fixes the duplicate record issue
but I'm not sure it would be that simple.  I think the duplicate record issue is the cause
of the problem but I don't see any open issue on this other than 11887.
On Tue, Aug 28, 2018 at 12:14 PM ZAIDI, ASAD A <az192g@att.com> wrote:


You may want to check if coincidentally you’re having expired cells in heap. GC log should
be able to tell you OR look for tombstones in system.log file. See your compactions are under
control and normal.  This may not be related to upgrade at all!

 

 

From: Pradeep Chhetri [mailto:pradeep@stashaway.com]
Sent: Tuesday, August 28, 2018 3:32 AM
To: user@cassandra.apache.org
Subject: Re: Upgrade from 2.1 to 3.11

 

You may want to try upgrading to 3.11.3 instead which has some memory leaks fixes.

 

On Tue, Aug 28, 2018 at 9:59 AM, Mun Dega <mundega97@gmail.com> wrote:


I am surprised that no one else ran into any issues with this version.  GC can't catch up
fast enough and there is constant Full GC taking place.

 

The result? unresponsive nodes makeing entire cluster unusable.

 

Any insight on this issue from anyone that is using this version would be appreciated.

 

Ma

 

On Fri, Aug 24, 2018, 04:30 Mohamadreza Rostami <mohamadrezarostami2@gmail.com> wrote:


You have very large heap,it’s take most of  cpu time in GC stage.you should in maximum
set heap on 12GB and enable row cache to your cluster become faster.

On Friday, 24 August 2018, Mun Dega <mundega97@gmail.com> wrote:


120G data

28G heap out of 48 on system

9 node cluster, RF3

 

On Thu, Aug 23, 2018, 17:19 Mohamadreza Rostami <mohamadrezarostami2@gmail.com> wrote:


Hi,

How much data do you have? How much RAM do your servers have? How much do you have a heep?

On Thu, Aug 23, 2018 at 10:14 PM Mun Dega <mundega97@gmail.com> wrote:


Hello,

 

We recently upgraded from Cassandra 2.1 to 3.11.2 on one cluster.  The process went OK including
upgradesstable but we started to experience high latency for r/w, occasional OOM and long
GC pause after.

 

For the same cluster with 2.1, we didn't have any issues like this.  We also kept server
specs, heap, all the same in post upgrade

 

Has anyone else had similar issues going to 3.11 and what are the major changes that could
have such a major setback in the new version?

 

Ma Dega






 

  
Mime
View raw message