cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
Date Tue, 19 Apr 2011 14:23:05 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021602#comment-13021602
] 

Jonathan Ellis edited comment on CASSANDRA-2408 at 4/19/11 2:21 PM:
--------------------------------------------------------------------

Did you run any repairs _before_ seeing this?

SSTableWriter checks order (of data written out of a memtable) at write time, so the only
way to get data out-of-order should be to stream it from a corrupt node.

Repair would explain why it affects adjacent nodes too.

      was (Author: jbellis):
    Did you run any repairs _before_ seeing this?

SSTableWriter checks order (of data written out of a memtable) at write time, so the only
way to get data out-of-order should be to stream it from a corrupt node.

Repair would explain why it affects adjacent noces too.
  
> Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException:
Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2408
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.6
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive,
we also observerd a "keys must be written in ascending order" exception on all adjacent quorum
nodes, causing their compactation to fail.
> We had another node with faulty ram a week earlier, and it was causing the same errors
on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
>  INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396)
Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last
written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current
key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
>  INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing
into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line
112) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
>       at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>       at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>       at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
>       at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
>       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>       at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message