cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Powell (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-12005) Out of memory error in MessagingService
Date Tue, 14 Jun 2016 15:18:01 GMT
Chris Powell created CASSANDRA-12005:
----------------------------------------

             Summary: Out of memory error in MessagingService
                 Key: CASSANDRA-12005
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12005
             Project: Cassandra
          Issue Type: Bug
          Components: Streaming and Messaging
         Environment: Ubuntu 14.04.4 LTS 3.13.0-79-generic #123-Ubuntu SMP x86_64

Cassandra ReleaseVersion: 2.2.5

java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
            Reporter: Chris Powell


I am periodically loosing nodes due to the below OOM error. The nodes restart perfectly fine.
It appears intermittent and randomly affects nodes. There are no other warnings or errors
in the log files.

I am using the {{GCG1}} with the following options:
{quote}
JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
JVM_OPTS="$JVM_OPTS -XX:+ResizeTLAB"
JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=10"
JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"
{quote}

ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,237 CassandraDaemon.java:185
- Exception in thread Thread[MessagingService-Incoming-/10.184.11.109,5,main]
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
~[apache-cassandra-2.2.5.jar:2.2.5]
ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,239 JVMStabilityInspector.java:117
- JVM state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
~[apache-cassandra-2.2.5.jar:2.2.5]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message