incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cyril Scetbon <cyril.scet...@free.fr>
Subject Re: IllegalStateException when bootstrapping new nodes
Date Fri, 01 Nov 2013 18:04:18 GMT
We got issues with Hinted Handoff threads (2 actives) now which cause High CPU Load and a lot
of garbage collection only on our new nodes. You can see that we have a lot of Hints on our
new nodes (the 4 last ones)

node001
2.2M	/cassandra/data/system/hints/
node002
1.6M	/cassandra/data/system/hints/
node003
1.8M	/cassandra/data/system/hints/
node004
1.9M	/cassandra/data/system/hints/
node005
1.9M	/cassandra/data/system/hints/
node006
2.0M	/cassandra/data/system/hints/
node007
1.9M	/cassandra/data/system/hints/
node008
1.8M	/cassandra/data/system/hints/
node009
34G	/cassandra/data/system/hints/
node011
43G	/cassandra/data/system/hints/
node013
43G	/cassandra/data/system/hints/
node015
16G	/cassandra/data/system/hints/

I try to pause and resume hinted which made CPU return to normal load, but they are still
there and CPU seems to get high again later with same consequences (lot of garbage collections)
which seems to be again related to hints.

any idea ?
-- 
Cyril SCETBON

On 31 Oct 2013, at 23:35, Cyril Scetbon <cyril.scetbon@free.fr> wrote:

> No we're using vnodes
> -- 
> Cyril SCETBON
> 
> On 30 Oct 2013, at 20:25, Robert Coli <rcoli@eventbrite.com> wrote:
> 
>> On Wed, Oct 30, 2013 at 12:22 PM, Cyril Scetbon <cyril.scetbon@free.fr> wrote:
>> FYI, we should upgrade to the last 1.2 version (1.2.11+) in January 2014. However,
we would like to know if it's a known fixed bug or inform you about this issue if it's not.
>> 
>> Did you bootstrap multiple nodes into the same token range? That is unsupported...
>> 
>> What does "nodetool ring" say?
>> 
>> =Rob
>>  
> 


Mime
View raw message