cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Ferland <>
Subject Re: Cassandra stalls and dropped messages not due to GC
Date Tue, 03 Nov 2015 00:14:11 GMT

> On Nov 2, 2015, at 11:35 AM, Nate McCall <> wrote:
> Forgive me, but what is CMS?
> Sorry - ConcurrentMarkSweep garbage collector. 

Ah, my brain was trying to think in terms of something Cassandra specific. I have full GC
logging on and since moving to G1, I haven’t had any >500ms GC cycles and the >200ms
logger triggers about once every 2 minutes. I don’t intend to roll off that given positive
confirmation that the cycles seem to be working well and GC logs don’t line up with outages.
Also, the issue proved to be the same on CMS as on G1.

> No. I’ve tried some mitigations since tuning thread pool sizes and GC, but the problem
begins with only an upgrade of Cassandra. No other system packages, kernels, etc.
> From what 2.0 version did you upgrade? If it was < 2.0.7, you would need to run 'nodetool
upgradesstables'  but I'm not sure the issue would manifest that way. Otherwise, double check
the DSE release notes and upgrade guide. I've not had any issues like this going from 2.0.x
to 2.1.x on vanilla C*. 

2.0.14 or higher. I don’t recall what version of DSE I standardized on last, but probably
2.0.16. In any case, the format moved from jb to ka.

I checked into the related source code and from there gripped my logs where I’m seeing messages
like (most extreme example):

WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2015-11-02 23:10:16,478
- Out of 38 commit log syncs over the past 307s with average duration of 2087.32ms, 3 have
exceeded the configured commit interval by an average of 15515.67ms

I seem to have 3-4 32MB commit logs created per minute. In a quick experiment, I’ve run
nodetool flush just now and reduced a 5.7G directory to 58M. I’m going to flush all the
nodes and see if that’s somehow related where it’s just holding commit logs too long.
(Did I miss the configuration for maximum memtable age?)

View raw message