cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: exception when cleaning up...
Date Tue, 22 May 2012 10:00:39 GMT
CASSANDRA-3712 has not been applied to 0.8.X.  

If I understand the problem correctly the issue is 0.8.10. 

You may be able to avoid the race condition by:

1) Isolating the node from the cluster to stop write activity. You can either start the node
with the -Dcassandra.join_ring=false  JVM option or use nodetool disablethrift and disablegossip
to stop writes. Note that this will not stop existing Hinted Handoff sessions which target
the node. 

2) Flushing all memtables with nodetool flush 

3) Running nodetool cleanup

Hope that helps. 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 20/05/2012, at 10:20 PM, Boris Yen wrote:

> Hi,
> 
> We are currently running 0.8.10 with 4 nodes. We tried to re-balance the range each node
owns by using "nodetool move". After moving the node to the assigned token, we run the cleanup
command, then we saw the exceptions:
> 
>  Error occured during cleanup
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> 
> at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
> 
> at java.util.concurrent.FutureTask.get(Unknown Source)
> 
> at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:226)
> 
> at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1073)
> 
> at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1423)
> 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> 
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 
> at java.lang.reflect.Method.invoke(Unknown Source)
> 
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
> 
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
> 
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
> 
> at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
> 
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
> 
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
> 
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
> 
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
> 
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
> 
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
> 
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
> 
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
> 
> at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source)
> 
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 
> at java.lang.reflect.Method.invoke(Unknown Source)
> 
> at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
> 
> at sun.rmi.transport.Transport$1.run(Unknown Source)
> 
> at java.security.AccessController.doPrivileged(Native Method)
> 
> at sun.rmi.transport.Transport.serviceCall(Unknown Source)
> 
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
> 
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
> 
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
> 
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 
> at java.lang.Thread.run(Unknown Source)
> 
> Caused by: java.lang.AssertionError
> 
> at org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:762)
> 
> at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:844)
> 
> at org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilyStore.java:849)
> 
> at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:928)
> 
> at org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionManager.java:66)
> 
> at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:205)
> 
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> 
> at java.util.concurrent.FutureTask.run(Unknown Source)
> 
> ... 3 more
> 
> 
> 
> It seems to be relative to CASSANDRA-3712. However, it only got fixed on 1.0. I wonder
if this will also be fixed on 0.8? One other question, if we do not run the cleanup, is there
any other way that the unnecessary data can get cleaned?
> 
> 
> 
> Regards.
> 
> Boris
> 


Mime
View raw message