cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Zhuang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-13008) Add vm.max_map_count startup StartupCheck
Date Tue, 06 Dec 2016 19:53:59 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jay Zhuang updated CASSANDRA-13008:
-----------------------------------
    Description: 
It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
When the max_map_count is low, it throws OOM exception, which is hard to link to the real
issue of vm.max_map_count.
The problem happened when we tried to remove one node, all the other nodes in cluster crashed.
As each node was trying to load more local SSTable files for streaming.

I would suggest to add a StartupCheck for max_map_count, at least it could give a warning
message to help the debug.

{code}
ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to be unstable. 
Exiting forcefully due to:
java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_112]
    at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
{code}

  was:
It's recommended to set `vm.max_map_count` to 1048575 (CASSANDRA-3563)
When the max_map_count is low, it throws OOM exception, which is hard to link to the real
issue of vm.max_map_count.
The problem happened when we tried to remove one node, all the other nodes in cluster crashed.
As each node was trying to load more local SSTable files for streaming.

I would suggest to add a StartupCheck for max_map_count, at least it could give a warning
message to help the debug.

```
ERROR [MemtableFlushWriter:109] 2016-11-14 00:22:40,598 JVMStabilityInspector.java:117 - JVM
state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_112]
    at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:377)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:188)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:179)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:344)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$800(BigTableWriter.java:56)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:385)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:169)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:205)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:412)
~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1139) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_112]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_112]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
```  


> Add vm.max_map_count startup StartupCheck
> -----------------------------------------
>
>                 Key: CASSANDRA-13008
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Configuration
>            Reporter: Jay Zhuang
>            Priority: Minor
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link to the
real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in cluster
crashed. As each node was trying to load more local SSTable files for streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could give a warning
message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to be unstable.
 Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
>     at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
>     at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_112]
>     at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) ~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message