cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David King (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-1248) ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
Date Mon, 05 Jul 2010 22:14:52 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885354#action_12885354
] 

David King commented on CASSANDRA-1248:
---------------------------------------

I tried forcing the node to compact (by bringing it up listening only on localhost to keep
it from joining the other nodes, and telling it to compact), and it throws this exception:

{quote}

 INFO 15:00:12,012 Deleted /cassandra/data/reddit/CommentVote-1015-Data.db
 INFO 15:00:12,314 Deleted /cassandra/data/system/HintsColumnFamily-1239-Data.db
 INFO 15:00:12,923 Deleted /cassandra/data/reddit/VotesByLink-1059-Data.db
 INFO 15:00:19,785 Deleted /cassandra/data/reddit/LinkVote-991-Data.db

ERROR 15:10:11,307 Skipping row DecoratedKey(15495971576493891752174312399692978238, ac5b905cb26978367a0a777c6fb7a9d1f2f162b4)
in /cassandra/data/permacache/permacache-45341-Data.db
java.io.EOFException
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:29)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:117)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserializeFromSSTable(ColumnFamilySerializer.java:160)
        at org.apache.cassandra.io.IteratingRow.getColumnFamily(IteratingRow.java:79)
        at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:112)
        at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
        at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
        at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
        at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:295)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:177)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
{quote}

Telling it to compact again throws the same exception.

On the up-side, assuming there's only one corrupt file at least I know which one it is now.

> ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
> ----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-1248
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 0.6.3
> java version "1.6.0_0"
> IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
> OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
> Ubuntu 8.10
> 6 node cluster
> RF==3
>            Reporter: David King
>             Fix For: 0.8
>
>
> A node died by running out of disk space. After growing the volume, trying to execute
reads dies with:
> {quote}
> ERROR 11:02:51,957 Uncaught exception in thread Thread[ROW-READ-STAGE:10,5,main]
> java.lang.ArrayIndexOutOfBoundsException
>         at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
>         at java.io.DataInputStream.readUTF(DataInputStream.java:609)
>         at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
>         at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
>         at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>         at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
>         at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
>         at org.apache.cassandra.db.Table.getRow(Table.java:381)
>         at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
>         at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> {quote}
> The clients then die with TimeoutExceptions.
> It might be better to return an exception to the client immediately instead of letting
them die, and it would be helpful if this exception were accompanied with a message indicating
*which* SStable is malformed. 
> This command does *not* turn up the failed table (nor does the same with sstablekeys):
> {quote}
> ri@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x; /usr/local/cassandra/bin/sstable2json
$x > /dev/null; done
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message