Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DA95E200BD3 for ; Tue, 6 Dec 2016 20:42:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D95EF160B17; Tue, 6 Dec 2016 19:42:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2CB64160B2D for ; Tue, 6 Dec 2016 20:42:00 +0100 (CET) Received: (qmail 42240 invoked by uid 500); 6 Dec 2016 19:41:59 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 41899 invoked by uid 99); 6 Dec 2016 19:41:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Dec 2016 19:41:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 90C3F2C03DF for ; Tue, 6 Dec 2016 19:41:58 +0000 (UTC) Date: Tue, 6 Dec 2016 19:41:58 +0000 (UTC) From: "Jay Zhuang (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (CASSANDRA-13008) Add vm.max_map_count startup StartupCheck MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 06 Dec 2016 19:42:01 -0000 Jay Zhuang created CASSANDRA-13008: -------------------------------------- Summary: Add vm.max_map_count startup StartupCheck Key: CASSANDRA-13008 URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 Project: Cassandra Issue Type: Improvement Components: Configuration Reporter: Jay Zhuang Priority: Minor It's recommended to set `vm.max_map_count` to 1048575 (CASSANDRA-3563) When the max_map_count is low, it throws OOM exception, which is hard to link to the real issue of vm.max_map_count. The problem happened when we tried to remove one node, all the other nodes in cluster crashed. As each node was trying to load more local SSTable files for streaming. I would suggest to add a StartupCheck for max_map_count, at least it could give a warning message to help the debug. ``` ERROR [MemtableFlushWriter:109] 2016-11-14 00:22:40,598 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_112] at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:377) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:188) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:179) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:344) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$800(BigTableWriter.java:56) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:385) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:169) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:205) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:412) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.5.jar:2.2.5] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1139) ~[apache-cassandra-2.2.5.jar:2.2.5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_112] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112] ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)