cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitalii Tymchyshyn <tiv...@gmail.com>
Subject Re: 1.0.6 - High CPU troubleshooting
Date Wed, 25 Jan 2012 15:09:47 GMT
Hello.

What's in the logs? It should output something like "Hey, you've got 
most of your memory used. I am going to flush some of memtables". Sorry, 
I don't remember exact spelling, but it's gong from GC, so it should be 
greppable by "GC".

25.01.12 16:26, Matthew Trinneer написав(ла):
> Hello Community,
>
> Am troubleshooting an issue with sudden and sustained high CPU on nodes in a 3 node cluster.
 This takes place when there is minimal load on the servers, and continues indefinitely until
I stop and restart a node.  All nodes (3) seem to be effected by the same issue, however it
doesn't occur simultaneous on them (although all potentially could be effected at the same
time).
>
> Here's what I've been doing to troubleshoot.
>
> * vmstat - have run on all servers and there is *no* swap going on
>
> * iostat - have run on all servers and there is no indication that the disk is in anyway
a bottleneck
>
> * nodetool - nothing is backed up.  Not using all available heap (about 2/3rds).
>
> * Have tracked it down to a specific thread (top -H).  When I find that thread's hex
equivalent in a jstack dump I see the following
>
> 	"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0000000001d48800 nid=0x4534 runnable
>
>
> Which makes me think that perhaps it's something to do with GC configuration.   Unfortunately
I'm not able to determine why this might be happening.  Any suggestions on how to continue
troubleshooting?
>
> btw - here's my jvm (just in case)
>
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
>
>
> Thanks!
>
> Matt
>


Mime
View raw message