incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: CompactionExecutor EOF During Bootstrap
Date Mon, 07 Mar 2011 19:27:42 GMT
It sounds like it doesn't realize the data it's streaming over is
older-version data.  Can you create a ticket?

In the meantime nodetool scrub (on the existing nodes) will rewrite
the data in the new format which should workaround the problem.

On Mon, Mar 7, 2011 at 1:23 PM, Erik Onnen <eonnen@gmail.com> wrote:
> During a recent upgrade of our cassandra ring from 0.6.8 to 0.7.3 and
> prior to a drain on the 0.6.8 nodes, we lost a node for reasons
> unrelated to cassandra. We decided to push forward with the drain on
> the remaining healthy nodes. The upgrade completed successfully for
> the remaining nodes and the ring was healthy. However, we're unable to
> boostrap in a new node. The bootstrap process starts and we can see
> streaming activity in the logs for the node giving up tokens, but the
> bootstrapping node encounters the following:
>
>
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505)
> Joining: sleeping 30000 ms for pending range setup
>  INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505)
> Bootstrapping
>  INFO [HintedHandoff:1] 2011-03-07 10:38:02,899
> HintedHandOffManager.java (line 304) Started hinted handoff for
> endpoint /10.211.14.200
>  INFO [HintedHandoff:1] 2011-03-07 10:38:02,900
> HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
> to endpoint /10.211.14.200
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
>        at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>        at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
>        at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>        at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
>        at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>        at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
>  INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855
> SSTableReader.java (line 154) Opening
> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
>
>
> The same behavior has happened on both attempts. Logs from the node
> giving up tokens show activity by the StreamStage thread but after the
> failure on the bootstrapping node not much else relative to the
> stream.
>
> Lastly, the behavior in both cases seems to have issue with the third
> data file. Files f-1,f-2 and f-4 are present but f-3 is not.
>
> Any help would be appreciated.
>
> -erik
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message