cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Babar Tareen (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-8152) Cassandra crashes with Native memory allocation failure
Date Mon, 20 Oct 2014 20:53:35 GMT
Babar Tareen created CASSANDRA-8152:

             Summary: Cassandra crashes with Native memory allocation failure
                 Key: CASSANDRA-8152
             Project: Cassandra
          Issue Type: Bug
         Environment: EC2 (i2.xlarge)
            Reporter: Babar Tareen
            Priority: Critical

On a 6 node Cassandra (datastax-community-2.1) cluster running on EC2 (i2.xlarge) instances,
Jvm hosting the cassandra service randomly crashes with following error.

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#  Out of Memory Error (os_linux.cpp:2747), pid=26159, tid=140305605682944
# JRE version: Java(TM) SE Runtime Environment (7.0_60-b19) (build 1.7.0_60-b19)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode linux-amd64 compressed
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit
-c unlimited" before starting Java again

---------------  T H R E A D  ---------------

Current thread (0x0000000008341000):  JavaThread "MemtableFlushWriter:2055" daemon [_thread_new,
id=23336, stack(0x00007f9b71c56000,0x00007f9b71c97000)]

Stack: [0x00007f9b71c56000,0x00007f9b71c97000],  sp=0x00007f9b71c95820,  free space=254k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  []  VMError::report_and_die()+0x2ea
V  []  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V  []  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
V  []  os::pd_commit_memory(char*, unsigned long, bool)+0xc
V  []  os::commit_memory(char*, unsigned long, bool)+0x2a
V  []  os::pd_create_stack_guard_pages(char*, unsigned long)+0x6d
V  []  JavaThread::create_stack_guard_pages()+0x5e
V  []  JavaThread::run()+0x34
V  []  java_start(Thread*)+0x108

Changes in settings

JVM_OPTS="$JVM_OPTS -XX:TargetSurvivorRatio=50"
JVM_OPTS="$JVM_OPTS -XX:+AggressiveOpts"
JVM_OPTS="$JVM_OPTS -XX:+UseLargePages"

Writes are about 10K-15K/sec and there are very few reads. Cassandra 2.0.9 with same settings
never crashed. JVM crash logs are attached from two machines.

This message was sent by Atlassian JIRA

View raw message