cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Lacerda <steve.lace...@datastax.com>
Subject Re: cassandra node was put down with oom error
Date Wed, 01 May 2019 14:17:59 GMT
First, you have to find out where the memory is going. So, you can use the
mbeans in jconsole or something like that. You'll have to look at different
caches and offheap in cache and metrics types. Once you've figured that
out, then you can start working on tuning things. Yes, your heap is 32G,
add about 2G for jvm classes, and other jvm stuff. Then chunk cache,
counter cache, key cache, bloom filters, and whatever else and voila the
heap is gone.

On Mon, Apr 29, 2019 at 11:58 PM yeomii999@gmail.com <yeomii999@gmail.com>
wrote:

> Hello,
>
> I'm suffering from similar problem with OSS cassandra version3.11.3.
> My cassandra cluster have been running for longer than 1 years and there
> was no problem until this year.
> The cluster is write-intensive, consists of 70 nodes, and all rows have 2
> hr TTL.
> The only change is the read consistency from QUORUM to ONE. (I cannot
> revert this change because of the read latency)
> Below is my compaction strategy.
> ```
> compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '3', 'compaction_window_unit': 'MINUTES',
> 'enabled': 'true', 'max_threshold': '32', 'min_threshold': '4',
> 'tombstone_compaction_interval': '60', 'tombstone_threshold': '0.2',
> 'unchecked_tombstone_compaction': 'false'}
> ```
> I've tried rolling restarting the cluster several times,
> but the memory usage of cassandra process always keeps going high.
> I also tried Native Memory Tracking, but it only measured less memory
> usage than the system mesaures (RSS in /proc/{cassandra-pid}/status)
>
> Is there any way that I could figure out the cause of this problem?
>
>
> On 2019/01/26 20:53:26, Jeff Jirsa <jjirsa@gmail.com> wrote:
> > You’re running DSE so the OSS list may not be much help. Datastax May
> have more insight
> >
> > In open source, the only things offheap that vary significantly are
> bloom filters and compression offsets - both scale with disk space, and
> both increase during compaction. Large STCS compaction can cause pretty
> meaningful allocations for these. Also, if you have an unusually low
> compression chunk size or a very low bloom filter FP ratio, those will be
> larger.
> >
> >
> > --
> > Jeff Jirsa
> >
> >
> > > On Jan 26, 2019, at 12:11 PM, Ayub M <hiayub@gmail.com> wrote:
> > >
> > > Cassandra node went down due to OOM, and checking the /var/log/message
> I see below.
> > >
> > > ```
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: java invoked oom-killer:
> gfp_mask=0x280da, order=0, oom_score_adj=0
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: java cpuset=/ mems_allowed=0
> > > ....
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 DMA: 1*4kB (U) 0*8kB
> 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U)
> 1*2048kB (M) 3*4096kB (M) = 15908kB
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 DMA32: 1294*4kB (UM)
> 932*8kB (UEM) 897*16kB (UEM) 483*32kB (UEM) 224*64kB (UEM) 114*128kB (UEM)
> 41*256kB (UEM) 12*512kB (UEM) 7*1024kB (UE
> > > M) 2*2048kB (EM) 35*4096kB (UM) = 242632kB
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 Normal: 5319*4kB
> (UE) 3233*8kB (UEM) 960*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 62500kB
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 hugepages_total=0
> hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Node 0 hugepages_total=0
> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: 38109 total pagecache pages
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: 0 pages in swap cache
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Swap cache stats: add 0,
> delete 0, find 0/0
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Free swap  = 0kB
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Total swap = 0kB
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: 16394647 pages RAM
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: 0 pages HighMem/MovableOnly
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: 310559 pages reserved
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ pid ]   uid  tgid
> total_vm      rss nr_ptes swapents oom_score_adj name
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ 2634]     0  2634
> 41614      326      82        0             0 systemd-journal
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ 2690]     0  2690
> 29793      541      27        0             0 lvmetad
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ 2710]     0  2710
> 11892      762      25        0         -1000 systemd-udevd
> > > .....
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [13774]     0 13774
>  459778    97729     429        0             0 Scan Factory
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14506]     0 14506
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14586]     0 14586
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14588]     0 14588
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14589]     0 14589
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14598]     0 14598
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14599]     0 14599
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14600]     0 14600
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [14601]     0 14601
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [19679]     0 19679
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [19680]     0 19680
> 21628     5340      24        0             0 macompatsvc
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ 9084]  1007  9084
> 2822449   260291     810        0             0 java
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ 8509]  1007  8509
> 17223585 14908485   32510        0             0 java
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [21877]     0 21877
>  461828    97716     318        0             0 ScanAction Mgr
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [21884]     0 21884
>  496653    98605     340        0             0 OAS Manager
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [31718]    89 31718
> 25474      486      48        0             0 pickup
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ 4891]  1007  4891
> 26999      191       9        0             0 iostat
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: [ 4957]  1007  4957
> 26999      192      10        0             0 iostat
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Out of memory: Kill process
> 8509 (java) score 928 or sacrifice child
> > > Jan 23 20:07:17 ip-xxx-xxx-xxx-xxx kernel: Killed process 8509 (java)
> total-vm:68894340kB, anon-rss:59496344kB, file-rss:137596kB, shmem-rss:0kB
> > > ```
> > >
> > > Nothing else runs on this host except dse cassandra with search and
> monitoring agents. Max heap size is set to 31g, the cassandra java process
> seems to be using ~57gb (ram is 62gb) at the time of error.
> > > So I am guess the jvm started using lots of memory and triggered oom
> error.
> > > Is my understanding correct?
> > > That this is linux triggered jvm kill as the jvm was consuming more
> than available memory?
> > >
> > > So in this case jvm was using max of 31g and remaining 26gb its using
> is non-heap memory. Normally this process takes around 42g and the fact
> that at the time of oom moment it was consuming 57g I am suspecting the
> java process to be the culprit rather than victim.
> > >
> > > At the time of issue there was no heap dump taken, I have configured
> it now. But even if heap dump was taken would it have help figure out who
> is consuming more memory. Heapdump would only dump heap memory area, what
> should be used to dump non-heapdump? Native memory tracking is one thing I
> came across.
> > > Any way to have native memory dumped when oom occurs?
> > > Whats the best way to monitor the jvm memory to diagnose oom errors?
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: user-help@cassandra.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>

-- 
Steve Lacerda | Technical Support Engineer
office +1.408.933.3102 | mobile +1. 669.377.9742
steve.lacerda@datastax.com | datastax.com
<https://twitter.com/datastax> <https://www.linkedin.com/company/datastax/>
<https://www.facebook.com/datastax> <http://feeds.feedburner.com/datastax>
<https://github.com/datastax/>
<https://www.datastax.com/accelerate>

Mime
View raw message