incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lee Mighdoll <...@underneath.ca>
Subject Re: How to tune cassandra to avoid OOM
Date Thu, 19 Dec 2013 03:47:37 GMT
I'd suggest setting some cassandra jvm parameters so that you can analyze a
heap dump and peek through the gc logs.  That'll give you some clues e.g.
if the memory problem is growing steadily or suddenly, and clues from a
peek at which object are using the memory.

-XX:+HeapDumpOnOutOfMemoryError

And if you don't want to wait six days for another failure, you can collect
a heap sooner with jmap -F.

-Xloggc:/path/to/where/to/put/the/gc.log
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure

Cheers,
Lee



On Wed, Dec 18, 2013 at 6:52 PM, Shammi Jayasinghe <shammi@wso2.com> wrote:

> Hi,
>
>
> We are facing with a problem on Cassandra tuning. In that we have faced
> with following OOM scenario[1], after running the system for 6 days. We
> have tuned the cassandra with following values. These values also obtained
> by going through huge number of testing cycles. But still it has gone OOM.
> I would like to know if someone can help on identifying tuning parameters.
>
> In this server , we have given 6GB for the Xmx value and the total memory
> in the server is 8GB. Cassandra version is : apache-cassandra-1.2.4
>
> Tuning parameters:
>
> flush_largest_memtables_at: 0.5
>
> reduce_cache_sizes_at: 0.85
>
> reduce_cache_capacity_to: 0.6
>
> commitlog_total_space_in_mb: 16
>
> commitlog_segment_size_in_mb: 16
>
>
>
> As i mentioned in the above parameters ( Flush_largest_memtable_at to 0,5)
> , i feel that it has not be affected to the server. Is there any way that
> we can check whether it is affected as expected to the server ?
>
>
>
> [1]WARN 19:16:50,355 Heap is 0.9971737408184552 full.  You may need to
> reduce memtable and/or cache sizes.  Cassandra will now flush up to the
> two largest memtables to free up memory.  Adjust
> flush_largest_memtables_at threshold in cassandra.yaml if you don't want
> Cassandra to do this automatically
>
>  WARN 19:18:19,784 Flushing CFS(Keyspace='QpidKeySpace',
> ColumnFamily='DestinationSubscriptionsCountRow') to relieve memory pressure
>
> ERROR 19:20:50,316 Exception in thread Thread[ReadStage:63,5,main]
>
> java.lang.OutOfMemoryError: Java heap space
>
>         at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
>
>         at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
>
>         at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:391)
>
>         at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
>
>         at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
>
>         at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:84)
>
>         at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73)
>
>         at
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:370)
>
>         at
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:325)
>
>         at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:151)
>
>         at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:48)
>
>         at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>
>         at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>
>         at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90)
>
>         at
> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171)
>
>         at
> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154)
>
>         at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143)
>
>         at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)
>
>         at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)
>
>         at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>
>         at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>
>         at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:157)
>
>         at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
>
>         at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
>
>         at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:293)
>
>         at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
>
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
>
>         at org.apache.cassandra.db.Table.getRow(Table.java:347)
>
>         at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
>
>         at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1052)
>
>  INFO 19:20:51,397 Stop listening to thrift clients
>
> --
> Best Regards,
>
> *  Shammi Jayasinghe*
> Associate Tech Lead
> WSO2, Inc.; http://wso2.com,
> mobile: +94 71 4493085
>
>

Mime
View raw message