incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Serediuk, Adam" <Adam.Sered...@serialssolutions.com>
Subject Memory Usage During Read
Date Fri, 06 May 2011 19:25:45 GMT
We're troubleshooting a memory usage problem during batch reads. We've spent the last few days
profiling and trying different GC settings. The symptoms are that after a certain amount of
time during reads one or more nodes in the cluster will exhibit extreme memory pressure followed
by a gc storm. We've tried every possible JVM setting and different GC methods and the issue
persists. This is pointing towards something instantiating a lot of objects and keeping references
so that they can't be cleaned up.

Typically nothing is ever logged other than the GC failures however just now one of the nodes
emitted logs we've never seen before:

 INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java (line 2218) Unable to
reduce heap usage since there are no dirty column families

We have tried increasing the heap on these nodes to large values, eg 24GB and still run into
the same issue. We're running 8GB of heap normally and only one or two nodes will ever exhibit
this issue, randomly. We don't use key/row caching and our memtable sizing is 64mb/0.3. Larger
or smaller memtables make no difference in avoiding the issue. We're on 0.7.5, mmap, jna and
jdk 1.6.0_24

We've somewhat hit the wall in troubleshooting and any advice is greatly appreciated.

--
Adam

Mime
View raw message