incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Maizel <j...@soundcloud.com>
Subject Help with Error on reading sstable
Date Tue, 22 Feb 2011 12:32:19 GMT
I'm getting this error after a space problem caused issues during a
repair operation on one of six nodes in our cluster:

2011-02-22_11:54:50.26788 'ERROR [ROW-READ-STAGE:305] 11:54:50,267
CassandraDaemon.java:87 Uncaught exception in thread
Thread[ROW-READ-STAGE:305,5,main]
2011-02-22_11:54:50.26789 'java.lang.ArrayIndexOutOfBoundsException
2011-02-22_11:54:50.26789       at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
2011-02-22_11:54:50.26790       at
java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
2011-02-22_11:54:50.26790       at
java.io.DataInputStream.readUTF(DataInputStream.java:592)
2011-02-22_11:54:50.26790       at
java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
2011-02-22_11:54:50.26791       at
org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
2011-02-22_11:54:50.26791       at
org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
2011-02-22_11:54:50.26792       at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
2011-02-22_11:54:50.26792       at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:990)
2011-02-22_11:54:50.26793       at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:901)
2011-02-22_11:54:50.26793       at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:870)
2011-02-22_11:54:50.26794       at
org.apache.cassandra.db.Table.getRow(Table.java:382)
2011-02-22_11:54:50.26794       at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
2011-02-22_11:54:50.26794       at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
2011-02-22_11:54:50.26795       at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:49)
2011-02-22_11:54:50.26795       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
2011-02-22_11:54:50.26796       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
2011-02-22_11:54:50.26796       at java.lang.Thread.run(Thread.java:619)
2011-02-22_11:54:54.71933 'ERROR [ROW-READ-STAGE:302] 11:54:54,718
DebuggableThreadPoolExecutor.java:102 Error in ThreadPoolExecutor
2011-02-22_11:54:54.71935 'java.lang.ArrayIndexOutOfBoundsException
2011-02-22_11:54:54.71935       at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
2011-02-22_11:54:54.71936       at
java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
2011-02-22_11:54:54.71936       at
java.io.DataInputStream.readUTF(DataInputStream.java:592)
2011-02-22_11:54:54.71937       at
java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
2011-02-22_11:54:54.71937       at
org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
2011-02-22_11:54:54.71937       at
org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
2011-02-22_11:54:54.71938       at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
2011-02-22_11:54:54.71938       at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:990)
2011-02-22_11:54:54.71939       at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:901)
2011-02-22_11:54:54.71939       at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:870)
2011-02-22_11:54:54.71941       at
org.apache.cassandra.db.Table.getRow(Table.java:382)
2011-02-22_11:54:54.71942       at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
2011-02-22_11:54:54.71942       at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
2011-02-22_11:54:54.71942       at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:49)
2011-02-22_11:54:54.71943       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
2011-02-22_11:54:54.71943       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
2011-02-22_11:54:54.71944       at java.lang.Thread.run(Thread.java:619)

I am thinking that there was a failure with writing out an SSTable
because of space and now its corrupt.   Also, the repair caused a huge
amount of disk to be used and therefore ran out.  Currently, is there
a way to clear space in this situation?  Would running a clean up
help?

Running ver 0.6.6.

Thanks,

-- 
Jake Maizel

Mime
View raw message