Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of dking@ketralnis.com
 designates 68.183.67.83 as permitted sender)
From: David King <dking@ketralnis.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Keys must be written in ascending order
Date: Tue, 18 Jan 2011 23:10:19 -0800
Message-Id: <4D23FBC4-3BCD-4439-9D6B-68D8C7213128@ketralnis.com>
To: user@cassandra.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)

I'm upgrading an 0.6 cluster to 0.7 in a testing environment. In =
cleaning up one of the nodes I get the exception below. Googling around =
seems to reveal people having trouble with it caused by too-small heap =
sizes but that doesn't look to be what's going on here. Am I missing =
something obvious?

$ time ./cassandra-0.7/bin/nodetool -h cassa7test01 cleanup
Error occured while cleaning up keyspace keyspace
java.util.concurrent.ExecutionException: java.io.IOException: Keys must =
be written in ascending order.
        at =
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at =
org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager=
.java:180)
        at =
org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.j=
ava:909)
        at =
org.apache.cassandra.service.StorageService.forceTableCleanup(StorageServi=
ce.java:1127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:=
57)
        at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at =
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIn=
trospector.java:111)
        at =
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIn=
trospector.java:45)
        at =
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:2=
26)
        at =
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at =
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
        at =
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanS=
erverInterceptor.java:857)
        at =
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
        at =
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImp=
l.java:1449)
        at =
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl=
.java:90)
        at =
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIC=
onnectionImpl.java:1284)
        at =
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMICon=
nectionImpl.java:1382)
        at =
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.jav=
a:807)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:=
57)
        at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at =
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at =
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
        at =
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.jav=
a:808)
        at =
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java=
:667)
        at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1110)
        at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Keys must be written in ascending order.
        at =
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.j=
ava:107)
        at =
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:12=
4)
        at =
org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionMa=
nager.java:411)
        at =
org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.jav=
a:54)
        at =
org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:17=
1)
        at =
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        ... 3 more

real    14m27.895s
user    0m0.670s
sys     0m0.200s