cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject java.io.IOException: Keys must be written in ascending order.
Date Tue, 31 Aug 2010 20:38:39 GMT
I'm running the nightly trunk build from 2010-08-26_13-52-57 and got a bunch of java.io.IOException:
Keys must be written in ascending order errors. 

Am running a 4 node cluster, this error occured on a single nodes. Eventually the node went
OOM, not sure if it has anything to do with the error. There was a constant work load of around
400 to 500 reads and write per second, all columns under 10k. 


 Below if the first occurrence, there were a lot more over the next 4 hours until it ran
out of heap space around 4am.  I restarted this morning and got some more, they are below
this one. It appears to be functioning now and is doing some handed hint off. Once that is
finished I will do some repairing.


Quick question, do threads in the HINTED-HANDOFF-POOL just do sending or also handle receiving?
 
Did a search on jira and could not find any mention of this. 

thanks
Aaron


(sorry forgot to turn up the logging, its only INFO mode)

INFO [CompactionExecutor:1] 2010-09-01 00:06:24,966 SSTableWriter.java (line 72) Last written
key : DecoratedKey(86578664653752589598792334348786543293, c0a8221d)
 INFO [CompactionExecutor:1] 2010-09-01 00:06:24,966 SSTableWriter.java (line 73) Current
key : DecoratedKey(27007427795986317455504170376786785255, c0a8221c)
 INFO [CompactionExecutor:1] 2010-09-01 00:06:24,966 SSTableWriter.java (line 74) Writing
into file /local1/junkbox/cassandra/data/system/HintsColumnFamily-tmp-e-29-Data.db
ERROR [HINTED-HANDOFF-POOL:1] 2010-09-01 00:06:24,993 CassandraDaemon.java (line 85) Uncaught
exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending
order.
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
      g  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.IOException: Keys must be written in ascending order.
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException:
Keys must be written in ascending order.
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:243)
        at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:78)
        at org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:296)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        .. 6 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written
in ascending order.
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:239)
        ... 9 more
Caused by: java.io.IOException: Keys must be written in ascending order.
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:75)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:92)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:352)
        at org.apache.cassandra.db.CompactionManager$5.call(CompactionManager.java:229)
        ... 5 more

ERROR [CompactionExecutor:1] 2010-09-01 00:06:24,993 CassandraDaemon.java (line 85) Uncaught
exception in thread Thread[CompactionExecutor:1,1,main]
java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending
order.
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
        at org.apachecassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:651)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Keys must be written in ascending order.
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:75)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:92)
        at orgapache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:352)
        at org.apache.cassandra.db.CompactionManager$5.call(CompactionManager.java:229)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more
 INFO [Timer-0] 2010-09-01 00:06:41,764 Gossiper.java (line 188) InetAddress /192.168.34.28
is now dead.


================

Died 4 hours later, after the restart got more 

 INFO [CompactionExecutor:1] 2010-09-01 07:58:31,707 SSTableWriter.java (line 72) Last written
key : DecoratedKey(86578664653752589598792334348786543293, c0a8221d)
 INFO [CompactionExecutor:1] 2010-09-01 07:58:31,708 SSTableWriter.java (line 73) Current
key : DecoratedKey(27007427795986317455504170376786785255, c0a8221c)
 INFO [CompactionExecutor:1] 2010-09-01 07:58:31,708 SSTableWriter.java (line 74) Writing
into file /local1/junkbox/cassandra/data/system/HintsColumnFamily-tmp-e-63-Data.db
ERROR [CompactionExecutor:1] 2010-09-01 07:58:31,709 CassandraDaemon.java (line 85) Uncaught
exception in thread Thread[CompactionExecutor:1,1,main]
java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending
order.
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTaskjava:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
        at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:651)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.langThread.run(Thread.java:619)
Caused by: java.io.IOException: Keys must be written in ascending order.
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:75)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:92)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:352)
        at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:154)
        at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:135)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
    • Unnamed multipart/related (inline, None, 0 bytes)
View raw message