incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjeev Kulkarni <sanj...@locomatix.com>
Subject Re: 0.7.4 Bad sstables?
Date Tue, 26 Apr 2011 04:19:54 GMT
Hi,
Thanks for pointing out the fix. My followup question is if I install 0.7.5
will the problem go away with the current data?
Thanks!

On Mon, Apr 25, 2011 at 8:25 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> Ah...  could be https://issues.apache.org/jira/browse/CASSANDRA-2349
> (fixed for 0.7.5)
>
> On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni <sanjeev@locomatix.com>
> wrote:
> > The only other interesting information is that the columns of these rows
> all
> > had some ttl attached to them. Not sure if that matters.
> > Thanks!
> >
> > On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
> > <tmarthinussen@gmail.com> wrote:
> >>
> >> First column in the row has offset in the file of 190226525, last valid
> >> column is at 380293592, about 181MB from first column to last.
> >> in_memory_compaction_limit was 128MB, so almost certainly above the
> limit.
> >> Terje
> >>
> >> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
> >> <tmarthinussen@gmail.com> wrote:
> >>>
> >>> In my case, probably yes. From thw rows I have looked at, I think I
> have
> >>> only seen this on rows with 1 million plus columns/supercolumns.
> >>>
> >>> May very well been larger than in memory limit. I think the compacted
> row
> >>> I looked closer at was about 200MB and the in memory limit may have
> been
> >>> 256MB.
> >>>
> >>> I will see if we still got files around to verify.
> >>>
> >>> Regards,
> >>> Terje
> >>>
> >>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jbellis@gmail.com> wrote:
> >>>
> >>> > Was it on a "large" row?  (> in_memory_compaction_limit?)
> >>> >
> >>> > I'm starting to suspect that LazilyCompactedRow is computing row size
> >>> > incorrectly in some cases.
> >>> >
> >>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> >>> > <tmarthinussen@gmail.com> wrote:
> >>> >> I have been hunting similar looking corruptions, especially in
the
> >>> >> hints
> >>> >> column family, but I believe it occurs somewhere while  compacting.
> >>> >> I looked in greater detail on one sstable and the row length was
> >>> >> longer than
> >>> >> the actual data in the row, and as far as I could see, either the
> >>> >> length was
> >>> >> wrong or the row was missing data as there was was no extra data
in
> >>> >> the row
> >>> >> after the last column.
> >>> >> This was however on a somewhat aging dataset, so suspected it could
> be
> >>> >> related to 2376.
> >>> >>
> >>> >> Playing around with 0.8 at the moment and not seen it there yet....
> >>> >> (bet it
> >>> >> will show up tomorrow once I wrote that.. :))
> >>> >> Terje
> >>> >>
> >>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
> >>> >> <sanjeev@locomatix.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> Hi Sylvain,
> >>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
> >>> >>> Thanks!
> >>> >>>
> >>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
> >>> >>> <sylvain@datastax.com>
> >>> >>> wrote:
> >>> >>>>
> >>> >>>> Hi Sanjeev,
> >>> >>>>
> >>> >>>> What's the story of the cluster ? Did you started with
0.7.4, or
> is
> >>> >>>> it
> >>> >>>> upgraded from
> >>> >>>> some earlier version ?
> >>> >>>>
> >>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
> >>> >>>> <sanjeev@locomatix.com>
> >>> >>>> wrote:
> >>> >>>>> Hey guys,
> >>> >>>>> Running a one node cassandra server with version 0.7.4
patched
> >>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
> >>> >>>>> The system was running fine for a couple of days when
we started
> >>> >>>>> noticing
> >>> >>>>> something strange with cassandra. I stopped all applications
and
> >>> >>>>> restarted
> >>> >>>>> cassandra. And then did a scrub. During scrub, I noticed
these in
> >>> >>>>> the
> >>> >>>>> logs
> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace
follows)
> >>> >>>>> java.io.IOError: java.io.IOException: Impossible row
size
> >>> >>>>> 1516029079813320210
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>> >>>>>        at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>> >>>>>         at
> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>> >>>>>    at
> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>>        at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>>        at java.lang.Thread.run(Thread.java:662)
> >>> >>>>> Caused by: java.io.IOException: Impossible row size
> >>> >>>>> 1516029079813320210
> >>> >>>>>    ... 8 more
> >>> >>>>>  INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 613) Retrying from row index; data is -1768177699
bytes
> >>> >>>>> starting
> >>> >>>>> at
> >>> >>>>> 2626524914
> >>> >>>>>  WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 633) Retry failed too.  Skipping to next row
(retry's
> >>> >>>>> stacktrace
> >>> >>>>> follows)
> >>> >>>>> java.io.IOError: java.io.EOFException: bloom filter
claims to be
> >>> >>>>> 1868982636
> >>> >>>>> bytes, longer than entire row size -1768177699    
   at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> >>> >>>>>        at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>> >>>>>        at
> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>> >>>>>         at
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> >>>>>  at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>>        at java.lang.Thread.run(Thread.java:662)
> >>> >>>>> Caused by: java.io.EOFException: bloom filter claims
to be
> >>> >>>>> 1868982636
> >>> >>>>> bytes,
> >>> >>>>> longer than entire row size -1768177699        at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> >>> >>>>>        ... 8 more
> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace
follows)
> >>> >>>>> java.io.IOError: java.io.EOFException
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>> >>>>>         at
> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>> >>>>>         at
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>>         at java.lang.Thread.run(Thread.java:662)
> >>> >>>>> Caused by: java.io.EOFException
> >>> >>>>>         at
> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> >>> >>>>>         at
> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> >>> >>>>>         ... 22 more
> >>> >>>>>  INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 613) Retrying from row index; data is 78540539
bytes
> starting
> >>> >>>>> at
> >>> >>>>> 2229643127
> >>> >>>>>
> >>> >>>>> And then when i restarted the readers, i get the following
crash
> >>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
> >>> >>>>> AbstractCassandraDaemon.java
> >>> >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
> >>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining
is
> >>> >>>>> 791462117 but
> >>> >>>>> 1970433058 requested
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> >>> >>>>>         at org.apache.cassandra.db.Table.getRow(Table.java:333)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>>         at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>>         at java.lang.Thread.run(Thread.java:662)
> >>> >>>>>
> >>> >>>>> Any ideas?
> >>> >>>>> Thanks!
> >>> >>>
> >>> >>
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Jonathan Ellis
> >>> > Project Chair, Apache Cassandra
> >>> > co-founder of DataStax, the source for professional Cassandra support
> >>> > http://www.datastax.com
> >>
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Mime
View raw message