incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Molinaro <antho...@alumni.caltech.edu>
Subject Cassandra 0.5.1 restarts slow
Date Tue, 20 Apr 2010 21:57:24 GMT
Hi,

  I have a cassandra cluster where a couple things are happening.  Every
once in a while a node will start to get backed up.  Checking tpstats I
see a very large value for ROW-MUTATION-STAGE.  Sometimes it will be able
to clear it if I give it enough time, other times the vm OOMs.  With some
nodes I also see this happen during restarts, I'll restart and have to
wait 6-12 hours for the node to not be marked as 'Down'.
I've seen
http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
and ended up with the following settings.

KeysCachedFraction            : 0.01
MemtableSizeInMB              : 100
MemtableObjectCountInMillions : 0.5
Heap                          : -Xmx5G

I only have 2 CFs in this instance and entries are small so in most cases
I hit MemtableObjectCountInMillions first and total MemtableSizeInMB is
about 60MB-120MB for the 2 CFs combined.

Anyone have any pointers on where to look next?  These are m1.large EC2
instances (I want to move to xlarge to get more memory, but haven't yet
gotten clarification on the best process for node replacement, per my
other thread).

Thanks,

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@alumni.caltech.edu>

Mime
View raw message