cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: mmap segment underflow
Date Sat, 27 Aug 2011 16:41:38 GMT
My guess would be CASSANDRA-2675.  Upgrade to the latest version in
your release series, if you haven't.

On Sat, Aug 27, 2011 at 9:53 AM, Chad Johnson <chad.johnson.w@gmail.com> wrote:
> Hi,
>
> We periodically get the following exception in our Cassandra cluster. What I mean by
periodically is once or twice a week. It always affects the same column family ( containing
super columns).  The frequency of the problems seems to center on the same couple nodes,
but not always. I've been getting past the problem by performing a scrub on the column family,
which corrects the problem by eliminating one unreadable row.
>
>  WARN [CompactionExecutor:1] 2011-08-27 04:08:42,916 CompactionManager.java (line 654)
Unable to recover 1 rows that were skipped.  You can attempt manual recovery from the pre-scrub
snapshot.  You can also run nodetool repair to transfer the data from a healthy replica,
if any
>
> Next time I have the opportunity, I'll throw the node into debug mode before scrubbing,
perhaps more information will be available.
>
> I have a copy of the SSTables if anybody has an interest in dissecting it. I'm not a
Cassandra developer, but currently I've been able to open the SSTable using the SSTableReader
class, but not sure where to go from there. Maybe I should be using the CompactionManager.scrubOne
method or maybe use the SSTableScanner. Any tips from developers on the cassandra list in
this regard would also be much appreciated. Specifically, I'd like to dissect the sstable
and to figure out what the key is to the bad row and what is wrong with the columns/supercolumns
in that row.
>
> The only issue I've found WRT to mmap segment underflow is CASSANDRA-2315, which doesn't
seem to apply in this case.
>
> Thanks for your help.
>
> Chad
>
> ERROR [ReadStage:2772] 2011-08-27 03:42:08,606 AbstractCassandraDaemon.java (line 113)
Fatal exception in thread Thread[ReadStage:2772,5,main]
> java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 2612010 but
692929875 requested
>        at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:252)
>        at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:268)
>        at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:227)
>        at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1521)
>        at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1471)
>        at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:379)
>        at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:362)
>        at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:322)
>        at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
>        at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
>        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>        at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>        at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>        at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>        at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:69)
>        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>        at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1390)
>        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1267)
>        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
>        at org.apache.cassandra.db.Table.getRow(Table.java:324)
>        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:451)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.IOException: mmap segment underflow; remaining is 2612010 but 692929875
requested
>        at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:118)
>        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:356)
>        at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:318)
>        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:81)
>        at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:248)
>        ... 30 more
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message