cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michał Łowicki <mlowi...@gmail.com>
Subject C* 2.1.2 invokes oom-killer
Date Wed, 18 Feb 2015 18:28:19 GMT
Hi,

Couple of times a day 2 out of 4 members cluster nodes are killed

root@db4:~# dmesg | grep -i oom
[4811135.792657] [ pid ]   uid  tgid total_vm      rss cpu oom_adj
oom_score_adj name
[6559049.307293] java invoked oom-killer: gfp_mask=0x201da, order=0,
oom_adj=0, oom_score_adj=0

Nodes are using 8GB heap (confirmed with *nodetool info*) and aren't using
row cache.

Noticed that couple of times a day used RSS is growing really fast within
couple of minutes and I see CPU spikes at the same time -
https://www.dropbox.com/s/khco2kdp4qdzjit/Screenshot%202015-02-18%2015.10.54.png?dl=0
.

Could be related to compaction but after compaction is finished used RSS
doesn't shrink. Output from pmap when C* process uses 50GB RAM (out of
64GB) is available on http://paste.ofcode.org/ZjLUA2dYVuKvJHAk9T3Hjb. At
the time dump was made heap usage is far below 8GB (~3GB) but total RSS is
~50GB.

Any help will be appreciated.

-- 
BR,
Michał Łowicki

Mime
View raw message