Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of tivv00@gmail.com designates
 209.85.214.44 as permitted sender)
Message-ID: <4F201B3B.3060708@gmail.com>
Date: Wed, 25 Jan 2012 17:09:47 +0200
From: Vitalii Tymchyshyn <tivv00@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.24) Gecko/20111109 Lightning/1.0b2 Thunderbird/3.1.16
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: 1.0.6 - High CPU troubleshooting
References: <EA1556AD-86EC-4975-B7D6-A1877FFE0F24@coveritlive.com>
In-Reply-To: <EA1556AD-86EC-4975-B7D6-A1877FFE0F24@coveritlive.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hello.

What's in the logs? It should output something like "Hey, you've got 
most of your memory used. I am going to flush some of memtables". Sorry, 
I don't remember exact spelling, but it's gong from GC, so it should be 
greppable by "GC".

25.01.12 16:26, Matthew Trinneer написав(ла):
> Hello Community,
>
> Am troubleshooting an issue with sudden and sustained high CPU on nodes in a 3 node cluster.  This takes place when there is minimal load on the servers, and continues indefinitely until I stop and restart a node.  All nodes (3) seem to be effected by the same issue, however it doesn't occur simultaneous on them (although all potentially could be effected at the same time).
>
> Here's what I've been doing to troubleshoot.
>
> * vmstat - have run on all servers and there is *no* swap going on
>
> * iostat - have run on all servers and there is no indication that the disk is in anyway a bottleneck
>
> * nodetool - nothing is backed up.  Not using all available heap (about 2/3rds).
>
> * Have tracked it down to a specific thread (top -H).  When I find that thread's hex equivalent in a jstack dump I see the following
>
> 	"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0000000001d48800 nid=0x4534 runnable
>
>
> Which makes me think that perhaps it's something to do with GC configuration.   Unfortunately I'm not able to determine why this might be happening.  Any suggestions on how to continue troubleshooting?
>
> btw - here's my jvm (just in case)
>
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
>
>
> Thanks!
>
> Matt
>