incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: How to tune cassandra to avoid OOM
Date Mon, 23 Dec 2013 20:03:43 GMT
> Cassandra version is : apache-cassandra-1.2.4	
The latest 1.2 version is 1.2.13, you really should be on that. 

> commitlog_total_space_in_mb: 16
> commitlog_segment_size_in_mb: 16
Reducing the total commit log size to 16 MB is a very bad idea, you should return it to 4096
and the segment size to 32.

The commit log is kept on disk and has no impact on the memory footprint. Reducing the size
will cause much more disk IO. 

It’s kind of unusual to go OOM in 1.2+, but I’ve seen it happen with large number of SSTables
(30k+) and LCS. Also wide rows, or lots of tombstones, and bad queries can result in a lot
of premature tenuring. Finally custom comparators can create a lot of garbage or a low powered
CPU may not be able to keep up. 

How many cores do you have ? 

You may want to make these changes to reduce how quickly objects are tenured, also pay attention
to how low the total heap use get’s to after CMS.

JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=4" 
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=2"
JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=50”  

Hope that helps. 

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 19/12/2013, at 4:47 pm, Lee Mighdoll <lee@underneath.ca> wrote:

> I'd suggest setting some cassandra jvm parameters so that you can analyze a heap dump
and peek through the gc logs.  That'll give you some clues e.g. if the memory problem is growing
steadily or suddenly, and clues from a peek at which object are using the memory.  
> 
> -XX:+HeapDumpOnOutOfMemoryError
> 
> And if you don't want to wait six days for another failure, you can collect a heap sooner
with jmap -F.
> 
> -Xloggc:/path/to/where/to/put/the/gc.log
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure
> 
> Cheers,
> Lee
> 
> 
> 
> On Wed, Dec 18, 2013 at 6:52 PM, Shammi Jayasinghe <shammi@wso2.com> wrote:
> Hi,
> 
> 
> We are facing with a problem on Cassandra tuning. In that we have faced with following
OOM scenario[1], after running the system for 6 days. We have tuned the cassandra with following
values. These values also obtained by going through huge number of testing cycles. But still
it has gone OOM. I would like to know if someone can help on identifying tuning parameters.
> 
> In this server , we have given 6GB for the Xmx value and the total memory in the server
is 8GB. Cassandra version is : apache-cassandra-1.2.4	
> 
> Tuning parameters:
> flush_largest_memtables_at: 0.5
> reduce_cache_sizes_at: 0.85
> reduce_cache_capacity_to: 0.6
> commitlog_total_space_in_mb: 16
> commitlog_segment_size_in_mb: 16
> 
> 
> 
> As i mentioned in the above parameters ( Flush_largest_memtable_at to 0,5) , i feel that
it has not be affected to the server. Is there any way that we can check whether it is affected
as expected to the server ?
> 
> 
> 
> [1]WARN 19:16:50,355 Heap is 0.9971737408184552 full.  You may need to reduce memtable
and/or cache sizes.  Cassandra will now flush up to the two largest memtables to free up memory.
 Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra
to do this automatically
>  WARN 19:18:19,784 Flushing CFS(Keyspace='QpidKeySpace', ColumnFamily='DestinationSubscriptionsCountRow')
to relieve memory pressure
> ERROR 19:20:50,316 Exception in thread Thread[ReadStage:63,5,main]
> java.lang.OutOfMemoryError: Java heap space
>         at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
>         at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
>         at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:391)
>         at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
>         at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
>         at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:84)
>         at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73)
>         at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:370)
>         at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:325)
>         at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:151)
>         at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:48)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90)
>         at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171)
>         at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154)
>         at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:157)
>         at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
>         at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:293)
>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
>         at org.apache.cassandra.db.Table.getRow(Table.java:347)
>         at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
>         at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1052)
>  INFO 19:20:51,397 Stop listening to thrift clients
> 
> -- 
> Best Regards, 
>  
> Shammi Jayasinghe
> Associate Tech Lead
> WSO2, Inc.; http://wso2.com,
> mobile: +94 71 4493085 
> 
> 


Mime
View raw message