On 11/12/2013 05:29 AM, Aaron Morton
thanks for explanation, will try to provide some figures (but
unfortunately not from the 2.0.2).
For tombstones, do you do a lot of deletes ?
Are you doing large slices or do could you have
a lot of tombstones on the rows ?
don't really know - how can I monitor that?
Also in v2.0.2 cfstats has this
live cells per slice (last five minutes): 0.0
tombstones per slice (last five minutes): 0.0
For large slices you need to check your code. e.g. do you
anything that reads lots of columns or very large columns or
lets the user select how many columns to read?
The org.apache.cassandra.db.ArrayBackedSortedColumns in the
trace back is used during reads (.e.g.
Sorry for the confusion. I meant to say that I tried to decrease row
cache size to 384M and then to 128M and the GC times did not change
at all (still ~30% of the time).
You probably want the heap to be 4G to 8G in size,
10G will encounter longer pauses.
Also the size of the new heap may be too big
depending on the number of cores. I would recommend
I tried to decrease it first to 384M then to 128M with no
change in the behaviour. I don't really care extra memory
overhead of the cache - to be able to actual point to it
with objects, but I don't really see the reason why it
should create/delete those many objects so quickly.
Not sure what you changed to 384M.
I will get in touch with the developers and take the data from cf*
commands in a few days (I am out of office for some days).
Shows the heap growing very quickly.
This could be due to wide reads or a high write
Well, both prg01 and
prg02 receive the same load which is about ~150-250 (during
peak) read requests per seconds and 100-160 write requests
per second. The only with heap growing rapidly and GC
kicking in is on nodes with row cache enabled.
This sounds like on a row cache miss cassandra is reading the
whole row, which happens to be a wide row. I would also guess
some writes are going to the rows and they are getting
invalidated out of the row cache.
The row cache is not great for rows the update frequently
and/or wide rows.
How big are the rows ? use nodetool cfstats and nodetool
Thanks for the pointers, will get in touch.