incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Row caching + Wide row column family == almost crashed?
Date Mon, 03 Dec 2012 20:27:41 GMT
> > Disabling row caching on this new column family has resolved the issue
> > for now, but, is there something fundamental about row caching that I am
> > missing?
What cache provider were you using ? Check the row_cache_provider setting in the yaml file.


If you were using the ConcurrentLinkedHashCacheProvider that would explain the increase in
GC activity. Try the SerializingCacheProvider , it will take the row cache off the JVM heap.


And as Ed says, it's not always  a great idea for wide rows. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 3/12/2012, at 11:07 AM, Edward Capriolo <edlinuxguru@gmail.com> wrote:

> Row cache has to store the entire row. It is a very bad option for wide rows.
> 
> On Sunday, December 2, 2012, Mike <mtheroux2@yahoo.com> wrote:
> > Hello,
> >
> > We recently hit an issue within our Cassandra based application.  We
> > have a relatively new Column Family with some very wide rows (10's of
> > thousands of columns, or more in some cases).  During a periodic
> > activity, we the range of columns to retrieve various pieces of
> > information, a segment at a time.
> >
> > We do these same queries frequently at various stages of the process,
> > and I thought the application could see a performance benefit from row
> > caching.  We have a small row cache (100MB per node) already enabled,
> > and I enabled row caching on the new column family.
> >
> > The results were very negative.  When performing range queries with a
> > limit of 200 results, for a small minority of the rows in the new column
> > family, performance plummeted.  CPU utilization on the Cassandra node
> > went through the roof, and it started chewing up memory.  Some queries
> > to this column family hung completely.
> >
> > According to the logs, we started getting frequent GCInspector
> > messages.  Cassandra started flushing the largest mem_tables due to
> > hitting the "flush_largest_memtables_at" of 75%, and scaling back the
> > key/row caches.  However, to Cassandra's credit, it did not die with an
> > OutOfMemory error.  Its measures to emergency measures to conserve
> > memory worked, and the cluster stayed up and running.  No real errors
> > showed in the logs, except for Messages getting drop, which I believe
> > was caused by what was going on with CPU and memory.
> >
> > Disabling row caching on this new column family has resolved the issue
> > for now, but, is there something fundamental about row caching that I am
> > missing?
> >
> > We are running Cassandra 1.1.2 with a 6 node cluster, with a replication
> > factor of 3.
> >
> > Thanks,
> > -Mike
> >
> >


Mime
View raw message