cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Desimpel, Ignace" <>
Subject deadlock cleanup request due to compaction
Date Tue, 02 Jun 2015 10:22:25 GMT
After adding a 5-th node I started running NodeProbe:: forceKeyspaceCleanup. That function
is not returning. Below I added the program stack trace information
that shows that in this case and probably others cases it is possible that there is a deadlock

In my case, the compactionManager is calling a synchronized method getNextBackgroundTask that
is calling getMaximalTask. And in my case that function is running an endless loop so the
synchronized object is never released. And the request to cleanup is now trying to call the
synchronized pause() function, so that function and thus the cleanup cannot proceed.

Configuration : 5 node, v-nodes, replication 3, LCS, version 2.0.7

Daemon Thread [CompactionExecutor:72] (Suspended)
                owns: LeveledManifest  (id=8722)
                owns: LeveledCompactionStrategy  (id=8552)
                                waited by: Daemon System Thread [RMI TCP Connection(14840)-]
                LeveledManifest.getCompactionCandidates() line: 247
                LeveledCompactionStrategy.getMaximalTask(int) line: 121
                LeveledCompactionStrategy.getNextBackgroundTask(int) line: 113
                CompactionManager$ line: 191
                Executors$RunnableAdapter<T>.call() line: 471
                FutureTask<V>.run() line: 262
line: 1145
                ThreadPoolExecutor$ line: 615
       line: 745

Daemon System Thread [RMI TCP Connection(14840)-] (Suspended)
                owns: ColumnFamilyStore  (id=8551)
                waiting for: LeveledCompactionStrategy  (id=8552)
                                owned by: Daemon Thread [CompactionExecutor:72] (Suspended)
                LeveledCompactionStrategy(AbstractCompactionStrategy).pause() line: 112
                ColumnFamilyStore.runWithCompactionsDisabled(Callable<V>, boolean) line:
                ColumnFamilyStore.markAllCompacting() line: 2125
                CompactionManager.performAllSSTableOperation(ColumnFamilyStore, CompactionManager$AllSSTablesOperation)
line: 214
                CompactionManager.performCleanup(ColumnFamilyStore, CounterId$OneShotRenewer)
line: 265
                ColumnFamilyStore.forceCleanup(CounterId$OneShotRenewer) line: 1105
                StorageService.forceKeyspaceCleanup(String, String...) line: 2215
                GeneratedMethodAccessor42.invoke(Object, Object[]) line: not available
                DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
                Method.invoke(Object, Object...) line: 606
                Trampoline.invoke(Method, Object, Object[]) line: 75
                GeneratedMethodAccessor19.invoke(Object, Object[]) line: not available
                DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
                Method.invoke(Object, Object...) line: 606
                MethodUtil.invoke(Method, Object, Object[]) line: 279
                StandardMBeanIntrospector.invokeM2(Method, Object, Object[], Object) line:
                StandardMBeanIntrospector.invokeM2(Object, Object, Object[], Object) line:
                StandardMBeanIntrospector(MBeanIntrospector<M>).invokeM(M, Object, Object[],
Object) line: 237
                PerInterface<M>.invoke(Object, String, Object[], String[], Object) line:
                StandardMBeanSupport(MBeanSupport<M>).invoke(String, Object[], String[])
line: 252
                DefaultMBeanServerInterceptor.invoke(ObjectName, String, Object[], String[])
line: 819
                JmxMBeanServer.invoke(ObjectName, String, Object[], String[]) line: 801
                RMIConnectionImpl.doOperation(int, Object[]) line: 1487
                RMIConnectionImpl.access$300(RMIConnectionImpl, int, Object[]) line: 97
                RMIConnectionImpl$ line: 1328
                RMIConnectionImpl.doPrivilegedOperation(int, Object[], Subject) line: 1420
                RMIConnectionImpl.invoke(ObjectName, String, MarshalledObject, String[], Subject)
line: 848
                GeneratedMethodAccessor41.invoke(Object, Object[]) line: not available
                DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
                Method.invoke(Object, Object...) line: 606
                UnicastServerRef.dispatch(Remote, RemoteCall) line: 322
                Transport$ line: 177
                Transport$ line: 174
                AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext)
line: not available [native method]
                TCPTransport(Transport).serviceCall(RemoteCall) line: 173
                TCPTransport.handleMessages(Connection, boolean) line: 556
                TCPTransport$ConnectionHandler.run0() line: 811
                TCPTransport$ line: 670
                ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
                ThreadPoolExecutor$ line: 615
       line: 745

View raw message