cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Perrone (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10615) Large partition plus row cache causes JVM OutOfMemory Error
Date Wed, 28 Oct 2015 19:54:27 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Perrone updated CASSANDRA-10615:
----------------------------------------
    Description: 
Found on multiple nodes of cluster, restarting C* resulted in hung startup process, nodes
never came up to Up Joining, but would hang until OOM. The work-around is to disable row cache.


{quote}
ERROR [SharedPool-Worker-22] 2015-10-27 17:22:52,753  JVMStabilityInspector.java:94 - JVM
state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:374)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.1.jar:na]
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.1.jar:na]
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.1.jar:na]
        at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:264)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1967)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:456)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:451)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
{quote}

  was:
Found on multiple nodes of cluster, restarting C* resulted in hung startup process, nodes
never came up to Up Joining, but would hang until OOM. The work-around is to disable row cache.


```
ERROR [SharedPool-Worker-22] 2015-10-27 17:22:52,753  JVMStabilityInspector.java:94 - JVM
state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:374)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.1.jar:na]
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.1.jar:na]
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.1.jar:na]
        at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:264)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1967)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:456)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:451)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
```


> Large partition plus row cache causes JVM OutOfMemory Error
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-10615
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10615
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.1.8.621
>            Reporter: Michael Perrone
>
> Found on multiple nodes of cluster, restarting C* resulted in hung startup process, nodes
never came up to Up Joining, but would hang until OOM. The work-around is to disable row cache.

> {quote}
> ERROR [SharedPool-Worker-22] 2015-10-27 17:22:52,753  JVMStabilityInspector.java:94 -
JVM state determined to be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:374)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
>         at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
>         at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
>         at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:264)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1967)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:456)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:451)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
>         at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[cassandra-all-2.1.8.621.jar:2.1.8.621]
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message