incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: GCInspector GC for ConcurrentMarkSweep running every 15 seconds
Date Tue, 18 Feb 2014 23:01:45 GMT
Sounds like you have CMSInitiatingOccupancyFraction set close to 60.
You can raise that and/or figure out how to use less heap.

On Mon, Feb 17, 2014 at 5:06 PM, John Pyeatt <john.pyeatt@singlewire.com> wrote:
> I have a 6 node cluster running on AWS. We are using m1.large instances with
> heap size set to 3G.
>
> 5 of the 6 nodes seem quite healthy. The 6th one however is running
> GCInspector GC for ConcurrentMarkSweep every 15 seconds or so. There is
> nothing going on on this box. No repairs and almost not user activity. But
> the CPU is almost continuously at 50% or more.
>
> The only message in the log at all is the
>  INFO 2014-02-17 22:58:53,429 [ScheduledTasks:1] GCInspector GC for
> ConcurrentMarkSweep: 213 ms for 1 collections, 1964940024 used; max is
> 3200253952
>  INFO 2014-02-17 22:59:07,431 [ScheduledTasks:1] GCInspector GC for
> ConcurrentMarkSweep: 250 ms for 1 collections, 1983269488 used; max is
> 3200253952
>  INFO 2014-02-17 22:59:21,522 [ScheduledTasks:1] GCInspector GC for
> ConcurrentMarkSweep: 280 ms for 1 collections, 1998214480 used; max is
> 3200253952
>  INFO 2014-02-17 22:59:36,527 [ScheduledTasks:1] GCInspector GC for
> ConcurrentMarkSweep: 305 ms for 1 collections, 2013065592 used; max is
> 3200253952
>  INFO 2014-02-17 22:59:50,529 [ScheduledTasks:1] GCInspector GC for
> ConcurrentMarkSweep: 334 ms for 1 collections, 2028069232 used; max is
> 3200253952
>
> We don't see any of these messages on the other nodes in the cluster.
>
> We are seeing similar behaviour for both our production and QA clusters.
> Production is running cassandra 1.2.9 and QA is running 1.2.13.
>
> Here are some of the cassandra settings that I would think might be
> relevant.
>
> flush_largest_memtables_at: 0.75
> reduce_cache_sizes_at: 0.85
> reduce_cache_capacity_to: 0.6
> in_memory_compaction_limit_in_mb: 64
>
> Does anyone have any ideas why we are seeing this so selectively on one box?
>
> Any cures???
> --
> John Pyeatt
> Singlewire Software, LLC
> www.singlewire.com
> ------------------
> 608.661.1184
> john.pyeatt@singlewire.com



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Mime
View raw message