Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CA62BCB52 for ; Tue, 23 Dec 2014 17:15:13 +0000 (UTC) Received: (qmail 54986 invoked by uid 500); 23 Dec 2014 17:15:13 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 54947 invoked by uid 500); 23 Dec 2014 17:15:13 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 54936 invoked by uid 99); 23 Dec 2014 17:15:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Dec 2014 17:15:13 +0000 Date: Tue, 23 Dec 2014 17:15:13 +0000 (UTC) From: "Philip Thompson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-8533) Nodetool Repair hangs on trunk MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8533: --------------------------------------- Summary: Nodetool Repair hangs on trunk (was: Repair hangs on trunk) > Nodetool Repair hangs on trunk > ------------------------------ > > Key: CASSANDRA-8533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8533 > Project: Cassandra > Issue Type: Bug > Reporter: Philip Thompson > Fix For: 3.0 > > > The dtests {{sstableloader_compression_none_to_snappy_test}} and {{sstableloader_compression_none_to_deflate_test}} in {{sstable_generation_loading_test.py}} are hanging on trunk during repair. Test output can be seen here: http://cassci.datastax.com/job/trunk_dtest/752/console > Here is the jstack output for nodetool > {code} > 2014-12-23 12:11:46 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode): > "Attach Listener" daemon prio=5 tid=0x00007fe51301e800 nid=0x370b waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "ClientNotifForwarder-1" daemon prio=5 tid=0x00007fe514969800 nid=0x5d03 runnable [0x0000000113395000] > java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:152) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) > - locked <0x00000007fd504398> (a java.io.BufferedInputStream) > at java.io.DataInputStream.readByte(DataInputStream.java:265) > at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:214) > at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) > at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl_Stub.fetchNotifications(Unknown Source) > at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1342) > at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587) > at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470) > at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451) > at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107) > "JMX client heartbeat 2" daemon prio=5 tid=0x00007fe5148f8000 nid=0x5b03 waiting on condition [0x0000000113292000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:174) > at java.lang.Thread.run(Thread.java:745) > "GC Daemon" daemon prio=5 tid=0x00007fe5148db800 nid=0x5903 in Object.wait() [0x000000011318f000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007f8009570> (a sun.misc.GC$LatencyLock) > at sun.misc.GC$Daemon.run(GC.java:117) > - locked <0x00000007f8009570> (a sun.misc.GC$LatencyLock) > "RMI RenewClean-[10.150.0.64:7100]" daemon prio=5 tid=0x00007fe5148d3000 nid=0x5703 in Object.wait() [0x000000011308c000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007f8010ae0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007f8010ae0> (a java.lang.ref.ReferenceQueue$Lock) > at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535) > at java.lang.Thread.run(Thread.java:745) > "RMI Scheduler(0)" daemon prio=5 tid=0x00007fe51490c800 nid=0x5607 waiting on condition [0x0000000112f89000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007f8018ad0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=5 tid=0x00007fe51480f000 nid=0x4d03 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=5 tid=0x00007fe514811000 nid=0x4b03 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=5 tid=0x00007fe516006000 nid=0x4903 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=5 tid=0x00007fe514075000 nid=0x3917 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=5 tid=0x00007fe51480b000 nid=0x3503 in Object.wait() [0x0000000110972000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007f8009658> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007f8009658> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=5 tid=0x00007fe51480a000 nid=0x3303 in Object.wait() [0x000000011086f000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007f8020b68> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007f8020b68> (a java.lang.ref.Reference$Lock) > "main" prio=5 tid=0x00007fe514000800 nid=0xf07 waiting on condition [0x0000000108bc4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd873e20> (a com.google.common.util.concurrent.AbstractFuture$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285) > at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > at org.apache.cassandra.tools.NodeProbe.repairAsync(NodeProbe.java:262) > at org.apache.cassandra.tools.NodeTool$Repair.execute(NodeTool.java:1810) > at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:250) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:164) > "VM Thread" prio=5 tid=0x00007fe514807800 nid=0x3103 runnable > "GC task thread#0 (ParallelGC)" prio=5 tid=0x00007fe514804800 nid=0x10f runnable > "GC task thread#1 (ParallelGC)" prio=5 tid=0x00007fe514805000 nid=0x1907 runnable > "GC task thread#2 (ParallelGC)" prio=5 tid=0x00007fe51400a000 nid=0x2503 runnable > "GC task thread#3 (ParallelGC)" prio=5 tid=0x00007fe514805800 nid=0x2703 runnable > "GC task thread#4 (ParallelGC)" prio=5 tid=0x00007fe51400a800 nid=0x2903 runnable > "GC task thread#5 (ParallelGC)" prio=5 tid=0x00007fe514806000 nid=0x2b03 runnable > "GC task thread#6 (ParallelGC)" prio=5 tid=0x00007fe51400b000 nid=0x2d03 runnable > "GC task thread#7 (ParallelGC)" prio=5 tid=0x00007fe514807000 nid=0x2f03 runnable > "VM Periodic Task Thread" prio=5 tid=0x00007fe514800000 nid=0x4f03 waiting on condition > JNI global references: 225 > {code} > Here is the jstack output for Cassandra node1 > {code} > 2014-12-23 12:12:36 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode): > "Attach Listener" daemon prio=9 tid=0x00007fd9cf8ef000 nid=0x7d0b waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "MemtablePostFlush:3" daemon prio=9 tid=0x00007fd9d0008800 nid=0x710b waiting on condition [0x000000010f100000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007decdcfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:8" daemon prio=9 tid=0x00007fd9cb42e800 nid=0x14d03 waiting on condition [0x0000000117574000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:7" daemon prio=9 tid=0x00007fd9cf097800 nid=0x14303 waiting on condition [0x0000000117425000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:6" daemon prio=9 tid=0x00007fd9cc0eb800 nid=0x13903 waiting on condition [0x0000000117209000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamingTransferTaskTimeouts:1" daemon prio=9 tid=0x00007fd9cb9c3800 nid=0x13503 waiting on condition [0x0000000117183000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df5c42a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:5" daemon prio=9 tid=0x00007fd9ce83f000 nid=0x12d03 waiting on condition [0x0000000117077000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:4" daemon prio=9 tid=0x00007fd9ce83e000 nid=0x11f03 waiting on condition [0x0000000116ea2000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:3" daemon prio=9 tid=0x00007fd9cb9c0800 nid=0x10d03 waiting on condition [0x0000000116c47000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:2" daemon prio=9 tid=0x00007fd9ce016000 nid=0x10b03 waiting on condition [0x0000000116c04000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamConnectionEstablisher:1" daemon prio=9 tid=0x00007fd9cf019800 nid=0x10903 waiting on condition [0x0000000116bc1000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b9838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Repair#1:1" daemon prio=9 tid=0x00007fd9cba4b000 nid=0xf903 waiting on condition [0x00000001169a9000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df5c29f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RMI TCP Connection(idle)" daemon prio=9 tid=0x00007fd9cd881800 nid=0xf503 waiting on condition [0x0000000116923000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded0dab0> (a java.util.concurrent.SynchronousQueue$TransferStack) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "JMX server connection timeout 124" daemon prio=9 tid=0x00007fd9cca84000 nid=0x5a0f in Object.wait() [0x000000011685a000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007df52c808> (a [I) > at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168) > - locked <0x00000007df52c808> (a [I) > at java.lang.Thread.run(Thread.java:745) > "RMI TCP Connection(6)-10.150.0.64" daemon prio=9 tid=0x00007fd9cd948000 nid=0x5807 in Object.wait() [0x0000000111d7d000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007e2d01b98> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer) > at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:449) > - locked <0x00000007e2d01b98> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer) > at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:227) > at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:275) > at javax.management.remote.rmi.RMIConnectionImpl$3.run(RMIConnectionImpl.java:1289) > at javax.management.remote.rmi.RMIConnectionImpl$3.run(RMIConnectionImpl.java:1287) > at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1293) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) > at sun.rmi.transport.Transport$1.run(Transport.java:177) > at sun.rmi.transport.Transport$1.run(Transport.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:173) > at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RMI Scheduler(0)" daemon prio=9 tid=0x00007fd9cb29e000 nid=0xf307 waiting on condition [0x0000000111d3b000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded0da20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thread-9" prio=9 tid=0x00007fd9cdacc000 nid=0xf003 runnable [0x00000001168e0000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df2a5210> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df2a5240> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007e00f8218> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "Thread-8" prio=9 tid=0x00007fd9cb9c5800 nid=0xee03 runnable [0x000000011689d000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df2b6bc8> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df2b6bf8> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007e00677f8> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "WRITE-/127.0.0.1" daemon prio=9 tid=0x00007fd9cb9c4800 nid=0xec07 waiting on condition [0x0000000116817000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b3d78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "WRITE-/127.0.0.1" daemon prio=9 tid=0x00007fd9cb81d000 nid=0xe60f waiting on condition [0x0000000116791000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df2b6370> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "Thrift:2" daemon prio=9 tid=0x00007fd9cb0b4800 nid=0xe20b waiting on condition [0x000000011670b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df092dd8> (a java.util.concurrent.SynchronousQueue$TransferStack) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "pool-1-thread-1" prio=9 tid=0x00007fd9cb81e800 nid=0xe803 waiting on condition [0x00000001167d4000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0951f98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thrift:1" daemon prio=9 tid=0x00007fd9cd9da000 nid=0xe003 waiting on condition [0x00000001166c8000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df092dd8> (a java.util.concurrent.SynchronousQueue$TransferStack) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thread-5" prio=9 tid=0x00007fd9cb802000 nid=0xde07 runnable [0x0000000116685000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df28bbc0> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df28bbf0> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007dfe00d30> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "nioEventLoopGroup-2-3" prio=9 tid=0x00007fd9cb82f000 nid=0xdc03 runnable [0x0000000115642000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) > at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007decefe48> (a io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000007df09fa38> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007dfdf9e50> (a sun.nio.ch.KQueueSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-10" daemon prio=9 tid=0x00007fd9cb29f000 nid=0xda03 waiting on condition [0x00000001155ff000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-9" daemon prio=9 tid=0x00007fd9cb28a000 nid=0xd803 waiting on condition [0x00000001155bc000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-8" daemon prio=9 tid=0x00007fd9cd9c8000 nid=0xd603 waiting on condition [0x0000000115579000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-7" daemon prio=9 tid=0x00007fd9cb289800 nid=0xd403 waiting on condition [0x0000000115536000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-6" daemon prio=9 tid=0x00007fd9cb288800 nid=0xd203 waiting on condition [0x00000001154f3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-5" daemon prio=9 tid=0x00007fd9cb288000 nid=0xd003 waiting on condition [0x00000001154b0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-4" daemon prio=9 tid=0x00007fd9cb2be000 nid=0xce03 waiting on condition [0x000000010d39f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-3" daemon prio=9 tid=0x00007fd9cb2bd000 nid=0xcc03 waiting on condition [0x000000010d35c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-2" daemon prio=9 tid=0x00007fd9cb95e000 nid=0x3e0b waiting on condition [0x000000010d30d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "threadDeathWatcher-3-1" daemon prio=9 tid=0x00007fd9cb017800 nid=0x4007 waiting on condition [0x000000010d2ca000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "nioEventLoopGroup-2-2" prio=9 tid=0x00007fd9cb01a800 nid=0x3d07 runnable [0x000000010d287000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) > at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007dfbb85e0> (a io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000007def763e0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007dfba7a80> (a sun.nio.ch.KQueueSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "Thread-4" prio=9 tid=0x00007fd9cb9b4800 nid=0xc107 runnable [0x0000000114361000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df0c4220> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df0c4250> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007e00e7690> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "HintedHandoff:2" daemon prio=9 tid=0x00007fd9cd9bb800 nid=0xc903 waiting on condition [0x000000011446d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b075e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-1" daemon prio=9 tid=0x00007fd9cc9f8000 nid=0xc70b waiting on condition [0x000000011442a000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "WRITE-/127.0.0.2" daemon prio=9 tid=0x00007fd9cc85a800 nid=0xc503 waiting on condition [0x00000001143e7000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df0cabc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "WRITE-/127.0.0.2" daemon prio=9 tid=0x00007fd9cc85a000 nid=0xc303 waiting on condition [0x00000001143a4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df0c47f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "DestroyJavaVM" prio=9 tid=0x00007fd9cc860000 nid=0xe07 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Thread-2" prio=9 tid=0x00007fd9cc85f800 nid=0xbf03 runnable [0x000000011431e000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) > at java.net.ServerSocket.implAccept(ServerSocket.java:530) > at java.net.ServerSocket.accept(ServerSocket.java:498) > at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:102) > at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:36) > at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) > at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:110) > at org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:137) > "nioEventLoopGroup-2-1" prio=9 tid=0x00007fd9cb995000 nid=0xbd07 runnable [0x00000001142db000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) > at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007dfbb8600> (a io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000007def76428> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007dfba7ad0> (a sun.nio.ch.KQueueSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "NonPeriodicTasks:1" daemon prio=9 tid=0x00007fd9cf055800 nid=0xb903 waiting on condition [0x0000000114298000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dedba190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "PendingRangeCalculator:1" daemon prio=9 tid=0x00007fd9cb979800 nid=0xb703 waiting on condition [0x0000000114255000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df08f5f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "MiscStage:1" daemon prio=9 tid=0x00007fd9cda83800 nid=0xa503 waiting on condition [0x0000000113ffa000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b1e9f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "MigrationStage:1" daemon prio=9 tid=0x00007fd9cda96800 nid=0xa303 waiting on condition [0x0000000113fb7000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b03148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "AntiEntropyStage:1" daemon prio=9 tid=0x00007fd9cb1d9800 nid=0xa103 waiting on condition [0x0000000113f74000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b13dc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "GossipStage:1" daemon prio=9 tid=0x00007fd9cb1d8800 nid=0x9f03 waiting on condition [0x0000000113f31000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded6cc30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "BatchlogTasks:1" daemon prio=9 tid=0x00007fd9cb8c2000 nid=0x8d03 waiting on condition [0x0000000111bec000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b03190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "HintedHandoff:1" daemon prio=9 tid=0x00007fd9cb968800 nid=0x8b03 waiting on condition [0x0000000111ba9000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b075e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "ACCEPT-/127.0.0.1" prio=9 tid=0x00007fd9cc8f4800 nid=0x8903 runnable [0x0000000111b66000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) > - locked <0x00000007e0b1eac8> (a java.lang.Object) > at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) > - locked <0x00000007e0b1eae0> (a java.lang.Object) > at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:915) > "GossipTasks:1" daemon prio=9 tid=0x00007fd9cc9a0000 nid=0x8703 waiting on condition [0x0000000111b23000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df66ae08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "IndexSummaryManager:1" daemon prio=9 tid=0x00007fd9cf024800 nid=0x8503 waiting on condition [0x0000000111a2b000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dec6e070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "PERIODIC-COMMIT-LOG-SYNCER" prio=9 tid=0x00007fd9cb8c1000 nid=0x7b03 waiting on condition [0x000000010f914000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dfa2a1b8> (a java.util.concurrent.Semaphore$NonfairSync) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414) > at org.apache.cassandra.db.commitlog.AbstractCommitLogService$1.run(AbstractCommitLogService.java:116) > at java.lang.Thread.run(Thread.java:745) > "COMMIT-LOG-ALLOCATOR" prio=9 tid=0x00007fd9cb18f800 nid=0x7903 waiting on condition [0x000000010f42f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dfa22b80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:144) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.lang.Thread.run(Thread.java:745) > "CommitLogArchiver:1" daemon prio=9 tid=0x00007fd9cb18c800 nid=0x7703 waiting on condition [0x000000010f3ec000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dfa0dc58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "SlabPoolCleaner" daemon prio=9 tid=0x00007fd9cb94f800 nid=0x7503 waiting on condition [0x000000010f891000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279) > at org.apache.cassandra.utils.memory.MemtableCleanerThread.run(MemtableCleanerThread.java:66) > "Background_Reporter:1" daemon prio=9 tid=0x00007fd9ce954000 nid=0x6b03 waiting on condition [0x000000010f742000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deead1e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "EXPIRING-MAP-REAPER:1" daemon prio=9 tid=0x00007fd9ce0bf000 nid=0x6903 waiting on condition [0x000000010f6ff000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deeed490> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "OptionalTasks:1" daemon prio=9 tid=0x00007fd9cb8c9000 nid=0x6703 waiting on condition [0x000000010f6bc000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dee4dbc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "metrics-meter-tick-thread-2" daemon prio=9 tid=0x00007fd9cb13f000 nid=0x6503 waiting on condition [0x000000010f53f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deeed418> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "metrics-meter-tick-thread-1" daemon prio=9 tid=0x00007fd9cb122800 nid=0x6303 waiting on condition [0x000000010f4fc000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deeed418> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "ScheduledTasks:1" daemon prio=9 tid=0x00007fd9cb917000 nid=0x6103 waiting on condition [0x000000010f588000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dee35858> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RMI TCP Accept-0" daemon prio=9 tid=0x00007fd9cb108800 nid=0x5d03 runnable [0x000000010ef39000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) > at java.net.ServerSocket.implAccept(ServerSocket.java:530) > at java.net.ServerSocket.accept(ServerSocket.java:498) > at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) > at java.lang.Thread.run(Thread.java:745) > "RMI TCP Accept-7100" daemon prio=9 tid=0x00007fd9cb864000 nid=0x5c07 runnable [0x000000010eef6000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) > at java.net.ServerSocket.implAccept(ServerSocket.java:530) > at java.net.ServerSocket.accept(ServerSocket.java:498) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=9 tid=0x00007fd9cc861000 nid=0x5503 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=9 tid=0x00007fd9cb800000 nid=0x5303 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=9 tid=0x00007fd9cb01b800 nid=0x5103 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=9 tid=0x00007fd9cd877000 nid=0x4f03 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Surrogate Locker Thread (Concurrent GC)" daemon prio=9 tid=0x00007fd9cc83e000 nid=0x430b waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=9 tid=0x00007fd9cd870000 nid=0x3b03 in Object.wait() [0x000000010d238000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007dec21940> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007dec21940> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=9 tid=0x00007fd9cd86f800 nid=0x3903 in Object.wait() [0x000000010d1f5000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007dec26058> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007dec26058> (a java.lang.ref.Reference$Lock) > "VM Thread" prio=9 tid=0x00007fd9cc80b800 nid=0x3703 runnable > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=0x00007fd9cc803000 nid=0x180b runnable > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=0x00007fd9cc808000 nid=0x2313 runnable > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=0x00007fd9ce000000 nid=0x2503 runnable > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=0x00007fd9cd805800 nid=0x2703 runnable > "Gang worker#4 (Parallel GC Threads)" prio=9 tid=0x00007fd9ce001000 nid=0x2903 runnable > "Gang worker#5 (Parallel GC Threads)" prio=9 tid=0x00007fd9cc808800 nid=0x2b03 runnable > "Gang worker#6 (Parallel GC Threads)" prio=9 tid=0x00007fd9cc809000 nid=0x2d03 runnable > "Gang worker#7 (Parallel GC Threads)" prio=9 tid=0x00007fd9cc809800 nid=0x2f03 runnable > "Concurrent Mark-Sweep GC Thread" prio=9 tid=0x00007fd9cd807000 nid=0x3503 runnable > "Gang worker#0 (Parallel CMS Threads)" prio=9 tid=0x00007fd9cc0c4000 nid=0x3103 runnable > "Gang worker#1 (Parallel CMS Threads)" prio=9 tid=0x00007fd9cd806000 nid=0x3303 runnable > "VM Periodic Task Thread" prio=9 tid=0x00007fd9cb84a000 nid=0x5f03 waiting on condition > JNI global references: 975 > {code} > and node2: > {code} > 2014-12-23 12:12:50 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode): > "Attach Listener" daemon prio=9 tid=0x00007f97e600a000 nid=0x800b waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "MemtablePostFlush:7" daemon prio=9 tid=0x00007f97e58af800 nid=0x711b waiting on condition [0x000000011a310000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df4ad4c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "HintedHandoff:2" daemon prio=9 tid=0x00007f97e61c2800 nid=0x7f0b waiting on condition [0x000000011a53b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b06808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "StreamingTransferTaskTimeouts:1" daemon prio=9 tid=0x00007f97e2286000 nid=0xf103 waiting on condition [0x0000000120bfd000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df356d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thread-9" prio=9 tid=0x00007f97e0a84000 nid=0xe003 runnable [0x0000000120830000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df155f10> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df155f40> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007dfd75e80> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "Thread-8" prio=9 tid=0x00007f97e227f000 nid=0xde03 runnable [0x00000001207aa000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df190c90> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df190cc0> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007dfd750c0> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "WRITE-/127.0.0.2" daemon prio=9 tid=0x00007f97e0b80000 nid=0xdc07 waiting on condition [0x000000011cdd6000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df195670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "WRITE-/127.0.0.2" daemon prio=9 tid=0x00007f97e0ad8000 nid=0xd413 waiting on condition [0x000000011cd93000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df198370> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "pool-1-thread-1" prio=9 tid=0x00007f97e236b000 nid=0xda03 waiting on condition [0x00000001207ed000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dfa159e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-6" daemon prio=9 tid=0x00007f97e0ad6000 nid=0xd203 waiting on condition [0x000000011f419000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-5" daemon prio=9 tid=0x00007f97e0ad5800 nid=0xd003 waiting on condition [0x000000011d02c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-4" daemon prio=9 tid=0x00007f97e300d800 nid=0xce03 waiting on condition [0x000000011ce79000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-3" daemon prio=9 tid=0x00007f97e406c800 nid=0xcc07 waiting on condition [0x000000011ce36000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "Thread-5" prio=9 tid=0x00007f97e3a91000 nid=0xca03 runnable [0x00000001205f2000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df0655b8> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df0655e8> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007e0026cd8> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "SharedPool-Worker-2" daemon prio=9 tid=0x00007f97e2175800 nid=0xc803 waiting on condition [0x00000001205af000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "threadDeathWatcher-3-1" daemon prio=9 tid=0x00007f97e0815000 nid=0xc603 waiting on condition [0x00000001185fd000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "nioEventLoopGroup-2-2" prio=9 tid=0x00007f97e3003000 nid=0xc403 runnable [0x00000001185a6000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) > at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007dfa39ff8> (a io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000007df04d4b0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007dfa276a0> (a sun.nio.ch.KQueueSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "DestroyJavaVM" prio=9 tid=0x00007f97e080e000 nid=0xf07 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Thread-4" prio=9 tid=0x00007f97e0a6e800 nid=0x3e0b runnable [0x0000000118563000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) > at java.net.ServerSocket.implAccept(ServerSocket.java:530) > at java.net.ServerSocket.accept(ServerSocket.java:498) > at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:102) > at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:36) > at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) > at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:110) > at org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:137) > "nioEventLoopGroup-2-1" prio=9 tid=0x00007f97e0813000 nid=0x3d0b runnable [0x0000000118520000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) > at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007dfba7480> (a io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000007df04c0b8> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007def6dfe0> (a sun.nio.ch.KQueueSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "SharedPool-Worker-1" daemon prio=9 tid=0x00007f97e122f000 nid=0xc107 waiting on condition [0x000000011f56c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745) > "Thread-3" prio=9 tid=0x00007f97e0a6e000 nid=0xbf0b runnable [0x000000011f529000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > - locked <0x00000007df03c128> (a java.lang.Object) > at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) > - locked <0x00000007df03c158> (a java.lang.Object) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > - locked <0x00000007dfb9c388> (a sun.nio.ch.SocketAdaptor$SocketInputStream) > at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:215) > at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:149) > at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:101) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:149) > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > "WRITE-/127.0.0.1" daemon prio=9 tid=0x00007f97e0a71800 nid=0xbd03 waiting on condition [0x000000011f4e6000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df044f00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "WRITE-/127.0.0.1" daemon prio=9 tid=0x00007f97e0a6d000 nid=0xbb03 waiting on condition [0x000000011f4a3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df04f898> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:139) > "NonPeriodicTasks:1" daemon prio=9 tid=0x00007f97e4019800 nid=0xb903 waiting on condition [0x000000011f460000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dee359d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "PendingRangeCalculator:1" daemon prio=9 tid=0x00007f97e0a46000 nid=0xb703 waiting on condition [0x000000011d074000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df048d28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "MiscStage:1" daemon prio=9 tid=0x00007f97e1137800 nid=0xa503 waiting on condition [0x000000011f1be000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b06790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "MigrationStage:1" daemon prio=9 tid=0x00007f97e0a53000 nid=0xa303 waiting on condition [0x000000011f17b000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b08cc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "AntiEntropyStage:1" daemon prio=9 tid=0x00007f97e1147000 nid=0xa103 waiting on condition [0x000000011f138000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded955d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "GossipStage:1" daemon prio=9 tid=0x00007f97e0a52000 nid=0x9f03 waiting on condition [0x000000011f0f5000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b08d10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "BatchlogTasks:1" daemon prio=9 tid=0x00007f97e0a47000 nid=0x8d03 waiting on condition [0x000000011a798000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dedd4340> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "HintedHandoff:1" daemon prio=9 tid=0x00007f97e1148000 nid=0x8b03 waiting on condition [0x000000011a755000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007e0b06808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "ACCEPT-/127.0.0.2" prio=9 tid=0x00007f97e1145000 nid=0x8903 runnable [0x000000011a712000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) > - locked <0x00000007ded95630> (a java.lang.Object) > at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) > - locked <0x00000007ded95648> (a java.lang.Object) > at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:915) > "GossipTasks:1" daemon prio=9 tid=0x00007f97e4025800 nid=0x8703 waiting on condition [0x000000011a6cf000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df82fda8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "IndexSummaryManager:1" daemon prio=9 tid=0x00007f97e3a53800 nid=0x8503 waiting on condition [0x000000011a609000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deed4200> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "PERIODIC-COMMIT-LOG-SYNCER" prio=9 tid=0x00007f97e4026800 nid=0x7b03 waiting on condition [0x000000011ac27000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dec4cc80> (a java.util.concurrent.Semaphore$NonfairSync) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) > at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414) > at org.apache.cassandra.db.commitlog.AbstractCommitLogService$1.run(AbstractCommitLogService.java:116) > at java.lang.Thread.run(Thread.java:745) > "COMMIT-LOG-ALLOCATOR" prio=9 tid=0x00007f97e1146800 nid=0x7903 waiting on condition [0x000000011abe4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007df00ed58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:144) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.lang.Thread.run(Thread.java:745) > "CommitLogArchiver:1" daemon prio=9 tid=0x00007f97e3ae2000 nid=0x7703 waiting on condition [0x000000011aba1000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dfa1ea40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "SlabPoolCleaner" daemon prio=9 tid=0x00007f97e0a30800 nid=0x7503 waiting on condition [0x000000011ab5e000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) > at org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279) > at org.apache.cassandra.utils.memory.MemtableCleanerThread.run(MemtableCleanerThread.java:66) > "Background_Reporter:1" daemon prio=9 tid=0x00007f97e0a07000 nid=0x6b03 waiting on condition [0x000000011aa0f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deefb0f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "EXPIRING-MAP-REAPER:1" daemon prio=9 tid=0x00007f97e10cf000 nid=0x6903 waiting on condition [0x000000011a9cc000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deee0d78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "OptionalTasks:1" daemon prio=9 tid=0x00007f97e09a7000 nid=0x6703 waiting on condition [0x000000011a989000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deefb128> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "metrics-meter-tick-thread-2" daemon prio=9 tid=0x00007f97e09a6800 nid=0x6503 waiting on condition [0x000000011a946000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dee1b668> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "metrics-meter-tick-thread-1" daemon prio=9 tid=0x00007f97e0986000 nid=0x6303 waiting on condition [0x000000011a903000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dee1b668> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "ScheduledTasks:1" daemon prio=9 tid=0x00007f97e2169000 nid=0x6103 waiting on condition [0x000000011a39a000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007deee0d48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "RMI TCP Accept-0" daemon prio=9 tid=0x00007f97e1083000 nid=0x5d03 runnable [0x000000011a1a9000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) > at java.net.ServerSocket.implAccept(ServerSocket.java:530) > at java.net.ServerSocket.accept(ServerSocket.java:498) > at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) > at java.lang.Thread.run(Thread.java:745) > "RMI TCP Accept-7200" daemon prio=9 tid=0x00007f97e38ba000 nid=0x5c07 runnable [0x000000011a166000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) > at java.net.ServerSocket.implAccept(ServerSocket.java:530) > at java.net.ServerSocket.accept(ServerSocket.java:498) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=9 tid=0x00007f97e2146000 nid=0x5503 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=9 tid=0x00007f97e2145800 nid=0x5303 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=9 tid=0x00007f97e38b9000 nid=0x5103 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=9 tid=0x00007f97e3879000 nid=0x4f03 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Surrogate Locker Thread (Concurrent GC)" daemon prio=9 tid=0x00007f97e20eb000 nid=0x400b waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=9 tid=0x00007f97e1004800 nid=0x3b03 in Object.wait() [0x0000000118484000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007dec23c08> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007dec23c08> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=9 tid=0x00007f97e1003800 nid=0x3903 in Object.wait() [0x0000000118441000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007dec23c38> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007dec23c38> (a java.lang.ref.Reference$Lock) > "VM Thread" prio=9 tid=0x00007f97e080b000 nid=0x3703 runnable > "Gang worker#0 (Parallel GC Threads)" prio=9 tid=0x00007f97e2003000 nid=0x10f runnable > "Gang worker#1 (Parallel GC Threads)" prio=9 tid=0x00007f97e200e000 nid=0x1407 runnable > "Gang worker#2 (Parallel GC Threads)" prio=9 tid=0x00007f97e1802000 nid=0x2503 runnable > "Gang worker#3 (Parallel GC Threads)" prio=9 tid=0x00007f97e3000000 nid=0x2703 runnable > "Gang worker#4 (Parallel GC Threads)" prio=9 tid=0x00007f97e200e800 nid=0x2903 runnable > "Gang worker#5 (Parallel GC Threads)" prio=9 tid=0x00007f97e200f800 nid=0x2b03 runnable > "Gang worker#6 (Parallel GC Threads)" prio=9 tid=0x00007f97e2010000 nid=0x2d03 runnable > "Gang worker#7 (Parallel GC Threads)" prio=9 tid=0x00007f97e2010800 nid=0x2f03 runnable > "Concurrent Mark-Sweep GC Thread" prio=9 tid=0x00007f97e3801000 nid=0x3503 runnable > "Gang worker#0 (Parallel CMS Threads)" prio=9 tid=0x00007f97e20d2000 nid=0x3103 runnable > "Gang worker#1 (Parallel CMS Threads)" prio=9 tid=0x00007f97e3800000 nid=0x3303 runnable > "VM Periodic Task Thread" prio=9 tid=0x00007f97e38ba800 nid=0x5f03 waiting on condition > JNI global references: 362 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)