Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6751AC70E for ; Sun, 20 May 2012 10:20:53 +0000 (UTC) Received: (qmail 60422 invoked by uid 500); 20 May 2012 10:20:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 60403 invoked by uid 500); 20 May 2012 10:20:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 60394 invoked by uid 99); 20 May 2012 10:20:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 May 2012 10:20:49 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yulinyen@gmail.com designates 209.85.212.178 as permitted sender) Received: from [209.85.212.178] (HELO mail-wi0-f178.google.com) (209.85.212.178) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 May 2012 10:20:41 +0000 Received: by wibhn6 with SMTP id hn6so1128719wib.7 for ; Sun, 20 May 2012 03:20:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=P+wkXZ7an0Roa2qMOgUzTlYmOFQTMdvmhXSQaVHaThg=; b=IGtruoBYMPOIuYRVgdjZYLqzN1/ogvx3ielvESE0TNzbRoV9g9MJXwCt1T4u6pR8VK 0xrGZ5X3Vg2ESpLHdqTi3MhZxMqY4pSTOZIe/qyE6p016nKxr1HUYI5qsK6UyDi/XXz8 N0d1XPF9MPzRm022op+KWVn9H7WwIyuLekbjIqKIjpiqZqffQKLAosHs49IYdNQ9zsid wBMtk4hGKqRimw/MVOPZpQna65ulmGDcj2seUD7mOqg/ybxHKnnTNTUqGGOjy2rB5ReJ lS1/35PrAdLXnkAzl6OGvhoOoEYd0o7hxvRV4Ddal9h9i8oYhvuCUoTxKBwOqxtelnbO nRbw== MIME-Version: 1.0 Received: by 10.216.141.15 with SMTP id f15mr10953412wej.82.1337509221004; Sun, 20 May 2012 03:20:21 -0700 (PDT) Received: by 10.223.88.20 with HTTP; Sun, 20 May 2012 03:20:20 -0700 (PDT) Date: Sun, 20 May 2012 18:20:20 +0800 Message-ID: Subject: exception when cleaning up... From: Boris Yen To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636c5bf6d00760704c0752220 --001636c5bf6d00760704c0752220 Content-Type: text/plain; charset=ISO-8859-1 Hi, We are currently running 0.8.10 with 4 nodes. We tried to re-balance the range each node owns by using "nodetool move". After moving the node to the assigned token, we run the cleanup command, then we saw the exceptions: Error occured during cleanup java.util.concurrent.ExecutionException: java.lang.AssertionError at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:226) at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1073) at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1423) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source) at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.AssertionError at org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:762) at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:844) at org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilyStore.java:849) at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:928) at org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionManager.java:66) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:205) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) ... 3 more It seems to be relative to CASSANDRA-3712. However, it only got fixed on 1.0. I wonder if this will also be fixed on 0.8? One other question, if we do not run the cleanup, is there any other way that the unnecessary data can get cleaned? Regards. Boris --001636c5bf6d00760704c0752220 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi,

We are currently running 0.8.10 with 4 nodes. We tri= ed to re-balance the range each node owns by using "nodetool move"= ;. After moving the node to the assigned token, we run the cleanup command,= then we saw the exceptions:

=A0Error=20 occured during cleanup

java.util.concurrent.ExecutionE= xception:=20 java.lang.AssertionError

=20 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)

=20 at java.util.concurrent.FutureTask.get(Unknown Source)

=20 at=20 org.apache.cassandra.db.compaction.CompactionManager.performCleanup(Compact= ionManager.java:226)

=20 at=20 org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.ja= va:1073)

=20 at=20 org.apache.cassandra.service.StorageService.forceTableCleanup(StorageServic= e.java:1423)

=20 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

=20 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

=20 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

=20 at java.lang.reflect.Method.invoke(Unknown Source)

=20 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown=20 Source)

=20 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown=20 Source)

=20 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)=

=20 at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)

=20 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)

=20 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown=20 Source)

=20 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)

=20 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown=20 Source)

=20 at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown=20 Source)

=20 at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Un= known=20 Source)

=20 at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unkn= own=20 Source)

=20 at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown=20 Source)

=20 at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source)

=20 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

=20 at java.lang.reflect.Method.invoke(Unknown Source)

=20 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)

=20 at sun.rmi.transport.Transport$1.run(Unknown Source)

=20 at java.security.AccessController.doPrivileged(Native Method)

=20 at sun.rmi.transport.Transport.serviceCall(Unknown Source)

=20 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)=

=20 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown=20 Source)

=20 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown=20 Source)

=20 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown=20 Source)

=20 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

=20 at java.lang.Thread.run(Unknown Source)

Caused=20 by: java.lang.AssertionError

=20 at=20 org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyS= tore.java:762)

=20 at=20 org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java= :844)

=20 at=20 org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilySt= ore.java:849)

=20 at=20 org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(Co= mpactionManager.java:928)

=20 at=20 org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionM= anager.java:66)

=20 at=20 org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManag= er.java:205)

=20 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

=20 at java.util.concurrent.FutureTask.run(Unknown Source)

=20 ... 3 more


It seems to be relative to=A0CASSANDRA-3712. However, i= t only got fixed on 1.0. I wonder if this will also be fixed on 0.8? One ot= her question, if we do not run the cleanup, is there any other way that the= unnecessary data can get cleaned?


Regards.

Boris

--001636c5bf6d00760704c0752220--