cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Kumar Maheshwari <Abhishek.Maheshw...@timesinternet.in>
Subject RE: [Cassandra 3.0.9] Cannot allocate memory
Date Fri, 24 Mar 2017 04:52:08 GMT
Thanks Jayesh,

I found the fix for the same.

I make below changes :

In /etc/sysctl.conf I make below change:
vm.max_map_count = 1048575

in the /etc/security/limits.d file:

root - memlock unlimited
root - nofile 100000
root - nproc 32768
root - as unlimited


Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.

From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: Thursday, March 23, 2017 8:36 PM
To: Abhishek Kumar Maheshwari <Abhishek.Maheshwari@timesinternet.in>; user@cassandra.apache.org
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory

Dmesg will often print a message saying that it had to kill a process if the server was short
of memory, so you will have to dump the output to a file and check.
If a process is killed to reclaim memory for the system, then it will dump a list of all processes
and the actual process that was killed.
So maybe, you can check for a kill like this - "dmesg | grep -i kill"
If you do find a line (or two), then you need to examine the output carefully.

In production, I tend to dump a lot of GC output also which helps troubleshooting.
E.g. Below is what I have.
If you look, I also have a flag that says that if the heap runs out of memory (which is rare),
then dump files.
If dmesg does not show your processes being killed, then you may have to dump gc logging info
to get some insight.

-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms16G
-Xmx16G
-Xmn4800M
-XX:+HeapDumpOnOutOfMemoryError
-Xss256k
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=80
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseParNewGC
-XX:MaxTenuringThreshold=2
-XX:SurvivorRatio=8
-XX:+UnlockDiagnosticVMOptions
-XX:ParGCCardsPerStrideChunk=32768
-XX:NewSize=750m
-XX:MaxNewSize=750m
-XX:+UseCondCardMark
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
-Xloggc:<some-path-for-gc-logs>
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=1M
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=<true|false>
-Dcom.sun.management.jmxremote.authenticate=<true|false>




From: Abhishek Kumar Maheshwari <Abhishek.Maheshwari@timesinternet.in<mailto:Abhishek.Maheshwari@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 5:18 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory

JVM config is as below:

-Xms16G
-Xmx16G
-Xmn3000M

What I need to check in dmesg?

From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 23 March 2017 03:39
To: Abhishek Kumar Maheshwari <Abhishek.Maheshwari@timesinternet.in<mailto:Abhishek.Maheshwari@timesinternet.in>>;
user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory


And what is the configured max heap?
Sometimes you may also be able to see some useful messages in "dmesg" output.

Jayesh

________________________________
From: Abhishek Kumar Maheshwari <Abhishek.Maheshwari@timesinternet.in<mailto:Abhishek.Maheshwari@timesinternet.in>>
Sent: Wednesday, March 22, 2017 5:05:14 PM
To: Thakrar, Jayesh; user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory

No only Cassandra is running on these servers.

From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari <Abhishek.Maheshwari@timesinternet.in<mailto:Abhishek.Maheshwari@timesinternet.in>>;
user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory

Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?

From: Abhishek Kumar Maheshwari <Abhishek.Maheshwari@timesinternet.in<mailto:Abhishek.Maheshwari@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: [Cassandra 3.0.9] Cannot allocate memory

Hi all,

I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below
exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40


Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288,
0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288,
0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288,
0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory'
(errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO:
os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)

Please let me know what I miss?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.

YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The
3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they
bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit
www.et-gbs.com<http://www.et-gbs.com>
Mime
View raw message