incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjeev Kulkarni <sanj...@locomatix.com>
Subject Re: 0.7.4 Bad sstables?
Date Tue, 26 Apr 2011 02:47:14 GMT
The only other interesting information is that the columns of these rows all
had some ttl attached to them. Not sure if that matters.
Thanks!

On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen <tmarthinussen@gmail.com
> wrote:

> First column in the row has offset in the file of 190226525, last valid
> column is at 380293592, about 181MB from first column to last.
>
> in_memory_compaction_limit was 128MB, so almost certainly above the limit.
>
> Terje
>
>
> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen <
> tmarthinussen@gmail.com> wrote:
>
>> In my case, probably yes. From thw rows I have looked at, I think I have
>> only seen this on rows with 1 million plus columns/supercolumns.
>>
>> May very well been larger than in memory limit. I think the compacted row
>> I looked closer at was about 200MB and the in memory limit may have been
>> 256MB.
>>
>> I will see if we still got files around to verify.
>>
>> Regards,
>> Terje
>>
>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> > Was it on a "large" row?  (> in_memory_compaction_limit?)
>> >
>> > I'm starting to suspect that LazilyCompactedRow is computing row size
>> > incorrectly in some cases.
>> >
>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
>> > <tmarthinussen@gmail.com> wrote:
>> >> I have been hunting similar looking corruptions, especially in the
>> hints
>> >> column family, but I believe it occurs somewhere while  compacting.
>> >> I looked in greater detail on one sstable and the row length was longer
>> than
>> >> the actual data in the row, and as far as I could see, either the
>> length was
>> >> wrong or the row was missing data as there was was no extra data in the
>> row
>> >> after the last column.
>> >> This was however on a somewhat aging dataset, so suspected it could be
>> >> related to 2376.
>> >>
>> >> Playing around with 0.8 at the moment and not seen it there yet....
>> (bet it
>> >> will show up tomorrow once I wrote that.. :))
>> >> Terje
>> >>
>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni <
>> sanjeev@locomatix.com>
>> >> wrote:
>> >>>
>> >>> Hi Sylvain,
>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
>> >>> Thanks!
>> >>>
>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <
>> sylvain@datastax.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Sanjeev,
>> >>>>
>> >>>> What's the story of the cluster ? Did you started with 0.7.4, or
is
>> it
>> >>>> upgraded from
>> >>>> some earlier version ?
>> >>>>
>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <
>> sanjeev@locomatix.com>
>> >>>> wrote:
>> >>>>> Hey guys,
>> >>>>> Running a one node cassandra server with version 0.7.4 patched
>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
>> >>>>> The system was running fine for a couple of days when we started
>> >>>>> noticing
>> >>>>> something strange with cassandra. I stopped all applications
and
>> >>>>> restarted
>> >>>>> cassandra. And then did a scrub. During scrub, I noticed these
in
>> the
>> >>>>> logs
>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>> >>>>> CompactionManager.java
>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
>> >>>>> 1516029079813320210
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>>>>        at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>>>>         at
>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>>>    at
>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>>        at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>>        at java.lang.Thread.run(Thread.java:662)
>> >>>>> Caused by: java.io.IOException: Impossible row size
>> 1516029079813320210
>> >>>>>    ... 8 more
>> >>>>>  INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>> >>>>> CompactionManager.java
>> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
>> starting
>> >>>>> at
>> >>>>> 2626524914
>> >>>>>  WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>> >>>>> CompactionManager.java
>> >>>>> (line 633) Retry failed too.  Skipping to next row (retry's
>> stacktrace
>> >>>>> follows)
>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to
be
>> >>>>> 1868982636
>> >>>>> bytes, longer than entire row size -1768177699        at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>> >>>>>        at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>>>>        at
>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>>>  at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>>        at java.lang.Thread.run(Thread.java:662)
>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
>> 1868982636
>> >>>>> bytes,
>> >>>>> longer than entire row size -1768177699        at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>> >>>>>        ... 8 more
>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>> >>>>> CompactionManager.java
>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >>>>> java.io.IOError: java.io.EOFException
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>>         at
>> >>>>>
>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>> >>>>>         at
>> >>>>>
>> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>>>>         at
>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>>         at java.lang.Thread.run(Thread.java:662)
>> >>>>> Caused by: java.io.EOFException
>> >>>>>         at
>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>> >>>>>         at
>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>> >>>>>         at
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>> >>>>>         ... 22 more
>> >>>>>  INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>> >>>>> CompactionManager.java
>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes starting
>> at
>> >>>>> 2229643127
>> >>>>>
>> >>>>> And then when i restarted the readers, i get the following crash
>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>> >>>>> AbstractCassandraDaemon.java
>> >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining
is
>> >>>>> 791462117 but
>> >>>>> 1970433058 requested
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>> >>>>>         at
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>> >>>>>         at org.apache.cassandra.db.Table.getRow(Table.java:333)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>> >>>>>         at
>> >>>>>
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>>         at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>>         at java.lang.Thread.run(Thread.java:662)
>> >>>>>
>> >>>>> Any ideas?
>> >>>>> Thanks!
>> >>>
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>>
>
>

Mime
View raw message