cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "J. Ryan Earl" <...@jryanearl.us>
Subject Re: OOM and high SSTables count
Date Thu, 05 Mar 2015 06:11:06 GMT
We think it is this bug:
https://issues.apache.org/jira/browse/CASSANDRA-8860

We're rolling a patch to beta before rolling it into production.

On Wed, Mar 4, 2015 at 4:12 PM, graham sanderson <graham@vast.com> wrote:

> We can confirm a problem on 2.1.3 (sadly our beta sstable state obviously
> did not match our production ones in some critical way)
>
> We have about 20k sstables on each of 6 nodes right now; actually a quick
> glance shows 15k of those are from OpsCenter, which may have something to
> do with beta/production mismatch
>
> I will look into the open OOM JIRA issue against 2.1.3 - we may being
> penalized for heavy use of JBOD (x7 per node)
>
> It also looks like 2.1.3 is leaking memory, though it eventually recovers
> via GCInspector causing a complete memtable flush.
>
> On Mar 4, 2015, at 12:31 PM, daemeon reiydelle <daemeonr@gmail.com> wrote:
>
> Are you finding a correlation between the shards on the OOM DC1 nodes and
> the OOM DC2 nodes? Does your monitoring tool indicate that the DC1 nodes
> are using significantly more CPU (and memory) than the nodes that are NOT
> failing? I am leading you down the path to suspect that your sharding is
> giving you hot spots. Also are you using vnodes?
>
> Patrick
>
>>
>> On Wed, Mar 4, 2015 at 9:26 AM, Jan <cnet62@yahoo.com> wrote:
>>
>>> HI Roni;
>>>
>>> You mentioned:
>>> DC1 servers have 32GB of RAM and 10GB of HEAP. DC2 machines have 16GB of
>>> RAM and 5GB HEAP.
>>>
>>> Best practices would be be to:
>>> a)  have a consistent type of node across both DC's.  (CPUs, Memory,
>>> Heap & Disk)
>>> b)  increase heap on DC2 servers to be  8GB for C* Heap
>>>
>>> The leveled compaction issue is not addressed by this.
>>> hope this helps
>>>
>>> Jan/
>>>
>>>
>>>
>>>
>>>   On Wednesday, March 4, 2015 8:41 AM, Roni Balthazar <
>>> ronibalthazar@gmail.com> wrote:
>>>
>>>
>>> Hi there,
>>>
>>> We are running C* 2.1.3 cluster with 2 DataCenters: DC1: 30 Servers /
>>> DC2 - 10 Servers.
>>> DC1 servers have 32GB of RAM and 10GB of HEAP. DC2 machines have 16GB
>>> of RAM and 5GB HEAP.
>>> DC1 nodes have about 1.4TB of data and DC2 nodes 2.3TB.
>>> DC2 is used only for backup purposes. There are no reads on DC2.
>>> Every writes and reads are on DC1 using LOCAL_ONE and the RF DC1: 2 and
>>> DC2: 1.
>>> All keyspaces have STCS (Average 20~30 SSTables count each table on
>>> both DCs) except one that is using LCS (DC1: Avg 4K~7K SSTables / DC2:
>>> Avg 3K~14K SSTables).
>>>
>>> Basically we are running into 2 problems:
>>>
>>> 1) High SSTables count on keyspace using LCS (This KS has 500GB~600GB
>>> of data on each DC1 node).
>>> 2) There are 2 servers on DC1 and 4 servers in DC2 that went down with
>>> the OOM error message below:
>>>
>>> ERROR [SharedPool-Worker-111] 2015-03-04 05:03:26,394
>>> JVMStabilityInspector.java:94 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> java.lang.OutOfMemoryError: Java heap space
>>>         at
>>> org.apache.cassandra.db.composites.CompoundSparseCellNameType.copyAndMakeWith(CompoundSparseCellNameType.java:186)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.composites.AbstractCompoundCellNameType$CompositeDeserializer.readNext(AbstractCompoundCellNameType.java:286)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.AtomDeserializer.readNext(AtomDeserializer.java:104)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:426)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:350)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:142)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:44)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>>> ~[guava-16.0.jar:na]
>>>         at
>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>>> ~[guava-16.0.jar:na]
>>>         at
>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:172)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:155)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>>> ~[guava-16.0.jar:na]
>>>         at
>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>>> ~[guava-16.0.jar:na]
>>>         at
>>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:203)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:320)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1915)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1748)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:342)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:57)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1486)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2171)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> ~[na:1.8.0_31]
>>>         at
>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>         at
>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>>
>>> So I am asking how to debug this issue and what are the best practices
>>> in this situation?
>>>
>>> Regards,
>>>
>>> Roni
>>>
>>>
>>>
>>
>
>

Mime
View raw message