incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiri Horky <ho...@avast.com>
Subject Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled
Date Tue, 12 Nov 2013 09:28:34 GMT
Hi,
On 11/12/2013 05:29 AM, Aaron Morton wrote:
>>> Are you doing large slices or do could you have a lot of tombstones
>>> on the rows ? 
>> don't really know - how can I monitor that?
> For tombstones, do you do a lot of deletes ? 
> Also in v2.0.2 cfstats has this 
>
> Average live cells per slice (last five minutes): 0.0
> Average tombstones per slice (last five minutes): 0.0
>
> For large slices you need to check your code. e.g. do you anything
> that reads lots of columns or very large columns or lets the user
> select how many columns to read?
>
> The org.apache.cassandra.db.ArrayBackedSortedColumns in the trace back
> is used during reads (.e.g.
> org.apache.cassandra.db.filter.SliceQueryFilter)
thanks for explanation, will try to provide some figures (but
unfortunately not from the 2.0.2).
>
>>> You probably want the heap to be 4G to 8G in size, 10G will
>>> encounter longer pauses. 
>>> Also the size of the new heap may be too big depending on the number
>>> of cores. I would recommend trying 800M
>> I tried to decrease it first to 384M then to 128M with no change in
>> the behaviour. I don't really care extra memory overhead of the cache
>> - to be able to actual point to it with objects, but I don't really
>> see the reason why it should create/delete those many objects so
>> quickly. 
> Not sure what you changed to 384M.
Sorry for the confusion. I meant to say that I tried to decrease row
cache size to 384M and then to 128M and the GC times did not change at
all (still ~30% of the time).
>
>>> Shows the heap growing very quickly. This could be due to wide reads
>>> or a high write throughput. 
>> Well, both prg01 and prg02 receive the same load which is about
>> ~150-250 (during peak) read requests per seconds and 100-160 write
>> requests per second. The only with heap growing rapidly and GC
>> kicking in is on nodes with row cache enabled.
>
> This sounds like on a row cache miss cassandra is reading the whole
> row, which happens to be a wide row. I would also guess some writes
> are going to the rows and they are getting invalidated out of the row
> cache. 
>
> The row cache is not great for rows the update frequently and/or wide
> rows. 
>
> How big are the rows ? use nodetool cfstats and nodetool cfhistograms.
I will get in touch with the developers and take the data from cf*
commands in a few days (I am out of office for some days).

Thanks for the pointers, will get in touch.

Cheers
Jiri Horky


Mime
View raw message