incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Onnen <eon...@gmail.com>
Subject Re: CompactionExecutor EOF During Bootstrap
Date Mon, 07 Mar 2011 21:01:44 GMT
Thanks Jonathan.

Filed: https://issues.apache.org/jira/browse/CASSANDRA-2283

We'll start the scrub during our normal compaction cycle and update
this thread and the bug with the results.

-erik

On Mon, Mar 7, 2011 at 11:27 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
> It sounds like it doesn't realize the data it's streaming over is
> older-version data.  Can you create a ticket?
>
> In the meantime nodetool scrub (on the existing nodes) will rewrite
> the data in the new format which should workaround the problem.
>
> On Mon, Mar 7, 2011 at 1:23 PM, Erik Onnen <eonnen@gmail.com> wrote:
>> During a recent upgrade of our cassandra ring from 0.6.8 to 0.7.3 and
>> prior to a drain on the 0.6.8 nodes, we lost a node for reasons
>> unrelated to cassandra. We decided to push forward with the drain on
>> the remaining healthy nodes. The upgrade completed successfully for
>> the remaining nodes and the ring was healthy. However, we're unable to
>> boostrap in a new node. The bootstrap process starts and we can see
>> streaming activity in the logs for the node giving up tokens, but the
>> bootstrapping node encounters the following:
>>
>>
>> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505)
>> Joining: sleeping 30000 ms for pending range setup
>>  INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505)
>> Bootstrapping
>>  INFO [HintedHandoff:1] 2011-03-07 10:38:02,899
>> HintedHandOffManager.java (line 304) Started hinted handoff for
>> endpoint /10.211.14.200
>>  INFO [HintedHandoff:1] 2011-03-07 10:38:02,900
>> HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows
>> to endpoint /10.211.14.200
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
>> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056
>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>> Thread[CompactionExecutor:1,1,main]
>> java.io.EOFException
>>        at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>>        at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
>>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
>>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
>> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635
>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>> Thread[CompactionExecutor:1,1,main]
>> java.io.EOFException
>>        at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>>        at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
>>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
>>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666
>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>> Thread[CompactionExecutor:1,1,main]
>> java.io.EOFException
>>        at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
>>        at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
>>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
>>        at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>>  INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855
>> SSTableReader.java (line 154) Opening
>> /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
>>
>>
>> The same behavior has happened on both attempts. Logs from the node
>> giving up tokens show activity by the StreamStage thread but after the
>> failure on the bootstrapping node not much else relative to the
>> stream.
>>
>> Lastly, the behavior in both cases seems to have issue with the third
>> data file. Files f-1,f-2 and f-4 are present but f-3 is not.
>>
>> Any help would be appreciated.
>>
>> -erik
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Mime
View raw message