cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arya Goudarzi <gouda...@gmail.com>
Subject Re: Cassandra mad GC
Date Fri, 17 Jan 2014 06:57:12 GMT
Ha! I missed the log line:

WARN [ReadStage:70] 2014-01-14 13:03:36,963 SliceQueryFilter.java
(line 209) Read 1001 live and 1518 tombstoned cells (see
tombstone_warn_threshold)


Seems some application code it trying to read a wide row with lots of
tombstones.


On Thu, Jan 16, 2014 at 8:28 PM, Aaron Morton <aaron@thelastpickle.com>wrote:

> c3.4xlarge
>
> long par new on a machine like this is not normal.
>
> Do you have a custom comparator or are you using triggers ?
> Do you have a data model that creates a lot of tombstones ?
>
> Try to return the settings to default and then tune from there, that
> includes returning to the default JVM GC settings. If for no other reason
> than other people will be able to offer advice.
>
> Have you changed the compaction_throughput ? Put it back if you have.
> If you have enabled multi_threaded compaction disable it.
> Consider setting concurrent_compactors to 4 or 8 to reduce compaction
> churn.
> If you have increased in_memory_compaction_limit put it back.
>
> Cassandra logs
>
> Can you provide some of the log messages from GCInspector ? How long are
> the pauses ? Is there a lot of CMS or ParNew ?
> Do you have monitoring in place ? Is CMS able to return the heap to a low
> value e.g. < 3Gb ?
>
> cpu load > 1000%
>
> Is this all from cassandra ?
> try jvmtop (https://code.google.com/p/jvmtop/) to see what cassandra
> threads are doing.
>
> It’s a lot easier to tune a system with fewer non default settings.
>
> Cheers
>
>  -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 16/01/2014, at 8:22 am, Arya Goudarzi <goudarzi@gmail.com> wrote:
>
> It is not a good idea to change settings without identifying the root
> cause. Chances are what you did masked the problem a bit for you, but the
> problem is still there, isn't it?
>
>
> On Wed, Jan 15, 2014 at 1:11 AM, Dimetrio <dimetrio@flysoft.ru> wrote:
>
>> I set G1 because GS started to work wrong(dropped messages) with standard
>> GC
>> settings.
>> In my opinion, Cassandra started to work more stable with G1 (it's getting
>> less count of timeouts now) but it's not ideally yet.
>> I just want cassandra  to works fine.
>>
>>
>>
>> --
>> View this message in context:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-mad-GC-tp7592248p7592257.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive
>> at Nabble.com.
>>
>
>
>
> --
> Cheers,
> -Arya
>
>
>


-- 
Cheers,
-Arya

Mime
View raw message