cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pranay akula <pranay.akula2...@gmail.com>
Subject JVM state determined to be unstable. Exiting forcefully. what is Java Stability Inspector ?? why it is stopping DSE?
Date Thu, 19 Jan 2017 13:00:49 GMT
>From last few days i am seeing on some of the nodes in cassandra cluster
DSE is getting shutdown due to the error below and i need to kill Java
process and restart DSE service.

I have cross checked reads and writes and compactions nothing looks
suspicious, but i am seeing full Gc pause on these server just before the
issue happening. what can be the reason for this how to debug that ?? how
to fine grain why on those particular nodes this is happening when these
nodes are serving same requests like rest of the cluster ??

Is this happening because of Full Gc is not getting performed properly, we
using G1GC and DSE 4.8.3


ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100
 JVMStabilityInspector.java:117 - JVM state determined to be unstable.
Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space

    at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74]

at
org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at org.apache.cassandra.db.Memtable.put(Memtable.java:210)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_74]

            at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]


    ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100
 SEPWorker.java:141 - Failed to execute task, unexpected exception killed
worker: {}

    java.lang.IllegalStateException: Shutdown in progress

            at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
~[na:1.8.0_74]

            at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
~[na:1.8.0_74]

            at
org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

            at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

at

org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

at

java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]


    INFO  [Thread-6] 2016-12-27 10:14:56,150  DseDaemon.java:420 - DSE
shutting down...

Mime
View raw message