incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janne Jalkanen <janne.jalka...@ecyrd.com>
Subject HintedHandoff IOError?
Date Mon, 11 Mar 2013 14:05:39 GMT
I keep seeing these in my log.  Three-node cluster, one node is working fine, but two other
nodes have increased latencies and these in the error logs (might of course be unrelated).
No obvious GC pressure, no disk errors that I can see.  Ubuntu 12.04 on EC2, Java 7. Repair
is run regularly.

My two questions: 1) should I worry, and 2) what might be going on, and 3) is there any way
to get rid of these? Can I just blow my HintedHandoff table to smithereens?

The only relevant issue I might see is CASSANDRA-5158, but it's not about HH.

Any more info I could dig?

Node A:

ERROR [OptionalTasks:1] 2013-03-11 13:34:19,153 AbstractCassandraDaemon.java (line 135) Exception
in thread Thread[OptionalTasks:1,5,main]
java.io.IOError: java.io.EOFException: bloom filter claims to be 0 bytes, longer than entire
row size 2147483647
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:101)
        at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:66)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:86)
        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator$1.create(SSTableScanner.java:198)
        at org.apache.cassandra.db.columniterator.LazyColumnIterator.getSubIterator(LazyColumnIterator.java:54)
        at org.apache.cassandra.db.columniterator.LazyColumnIterator.getColumnFamily(LazyColumnIterator.java:66)
        at org.apache.cassandra.db.RowIteratorFactory$2.reduce(RowIteratorFactory.java:95)
        at org.apache.cassandra.db.RowIteratorFactory$2.reduce(RowIteratorFactory.java:79)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1403)
        at org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1399)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1476)
        at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1455)
        at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1450)
        at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:406)
        at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:85)
        at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:120)
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException: bloom filter claims to be 0 bytes, longer than entire row
size 2147483647
        at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:129)
        at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:110)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:113)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:96)
        ... 30 more

Node B:

ERROR [OptionalTasks:1] 2013-03-11 13:51:02,177 AbstractCassandraDaemon.java (line 135) Exception
in thread Thread[OptionalTasks:1,5,main]
java.io.IOError: java.io.EOFException
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:101)
        at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:66)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:86)
        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator$1.create(SSTableScanner.java:198)
        at org.apache.cassandra.db.columniterator.LazyColumnIterator.getSubIterator(LazyColumnIterator.java:54)
        at org.apache.cassandra.db.columniterator.LazyColumnIterator.getColumnFamily(LazyColumnIterator.java:66)
        at org.apache.cassandra.db.RowIteratorFactory$2.reduce(RowIteratorFactory.java:95)
        at org.apache.cassandra.db.RowIteratorFactory$2.reduce(RowIteratorFactory.java:79)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1403)
        at org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1399)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
        at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1476)
        at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1455)
        at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1450)
        at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:406)
        at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:85)
        at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:120)
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
        at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
        at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:130)
        at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:110)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:113)
        at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:96)
        ... 30 more

/Janne
Mime
View raw message