cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjeev Kulkarni <sanj...@locomatix.com>
Subject Re: 0.7.4 Bad sstables?
Date Tue, 26 Apr 2011 06:28:00 GMT
Hi Jonathan,
I installed 0.7.5. The first scrub reported some errors. See logs below. But
I scrubed again and this time things went smoothly. I am now able to read
all my data.
Thanks for the help!

java.io.IOError: java.io.IOException: Error validating row
DecoratedKey(45412894640737485043586798643875005293,
706172746974696f6e696e666f2d6f626a6563742d3234313835)
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:169)
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:41)
        at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
        at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
        at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
        at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:69)
        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
        at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
        at
org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
        at
org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
        at
org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
        at
org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:783)
        at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590)
        at
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
        at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error validating row
DecoratedKey(45412894640737485043586798643875005293,
706172746974696f6e696e666f2d6f626a6563742d3234313835)
        ... 23 more
Caused by: org.apache.cassandra.db.marshal.MarshalException: String didn't
validate.
        at
org.apache.cassandra.db.marshal.UTF8Type.validate(UTF8Type.java:70)
        at org.apache.cassandra.db.Column.validateName(Column.java:241)
        at
org.apache.cassandra.db.DeletedColumn.validateFields(DeletedColumn.java:71)
        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:160)
        ... 22 more
 INFO [CompactionExecutor:1] 2011-04-26 02:13:58,917 CompactionManager.java
(line 612) Retrying from row index; data is 87082049 bytes starting at
259440243
 WARN [CompactionExecutor:1] 2011-04-26 02:13:59,218 CompactionManager.java
(line 632) Retry failed too.  Skipping to next row (retry's stacktrace
follows)


On Mon, Apr 25, 2011 at 10:57 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> You shouldn't be using beta1 unless your goal is to help shake the bugs
> out. :)
>
> 0.7.5 release is in progress; the artifacts are at
> http://people.apache.org/~slebresne/
>
> On Tue, Apr 26, 2011 at 12:19 AM, Sanjeev Kulkarni
> <sanjeev@locomatix.com> wrote:
> > BTW where do i download 0.7.5? I went
> > to
> http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.7.5/apache-cassandra-0.7.5-bin.tar.gz
> > but all the links there are broken.
> > I was thinking if I just skip 0.7.5 and go with 0.8-beta1, would that be
> > more advisable?
> > Thanks!
> >
> > On Mon, Apr 25, 2011 at 9:30 PM, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >>
> >> No. You'll need to run scrub.
> >>
> >> On Mon, Apr 25, 2011 at 11:19 PM, Sanjeev Kulkarni
> >> <sanjeev@locomatix.com> wrote:
> >> > Hi,
> >> > Thanks for pointing out the fix. My followup question is if I install
> >> > 0.7.5
> >> > will the problem go away with the current data?
> >> > Thanks!
> >> > On Mon, Apr 25, 2011 at 8:25 PM, Jonathan Ellis <jbellis@gmail.com>
> >> > wrote:
> >> >>
> >> >> Ah...  could be https://issues.apache.org/jira/browse/CASSANDRA-2349
> >> >> (fixed for 0.7.5)
> >> >>
> >> >> On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni
> >> >> <sanjeev@locomatix.com>
> >> >> wrote:
> >> >> > The only other interesting information is that the columns of these
> >> >> > rows
> >> >> > all
> >> >> > had some ttl attached to them. Not sure if that matters.
> >> >> > Thanks!
> >> >> >
> >> >> > On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
> >> >> > <tmarthinussen@gmail.com> wrote:
> >> >> >>
> >> >> >> First column in the row has offset in the file of 190226525, last
> >> >> >> valid
> >> >> >> column is at 380293592, about 181MB from first column to last.
> >> >> >> in_memory_compaction_limit was 128MB, so almost certainly above
> the
> >> >> >> limit.
> >> >> >> Terje
> >> >> >>
> >> >> >> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
> >> >> >> <tmarthinussen@gmail.com> wrote:
> >> >> >>>
> >> >> >>> In my case, probably yes. From thw rows I have looked at, I think
> I
> >> >> >>> have
> >> >> >>> only seen this on rows with 1 million plus columns/supercolumns.
> >> >> >>>
> >> >> >>> May very well been larger than in memory limit. I think the
> >> >> >>> compacted
> >> >> >>> row
> >> >> >>> I looked closer at was about 200MB and the in memory limit may
> have
> >> >> >>> been
> >> >> >>> 256MB.
> >> >> >>>
> >> >> >>> I will see if we still got files around to verify.
> >> >> >>>
> >> >> >>> Regards,
> >> >> >>> Terje
> >> >> >>>
> >> >> >>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >> >> >>>
> >> >> >>> > Was it on a "large" row?  (> in_memory_compaction_limit?)
> >> >> >>> >
> >> >> >>> > I'm starting to suspect that LazilyCompactedRow is computing
> row
> >> >> >>> > size
> >> >> >>> > incorrectly in some cases.
> >> >> >>> >
> >> >> >>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> >> >> >>> > <tmarthinussen@gmail.com> wrote:
> >> >> >>> >> I have been hunting similar looking corruptions, especially in
> >> >> >>> >> the
> >> >> >>> >> hints
> >> >> >>> >> column family, but I believe it occurs somewhere while
> >> >> >>> >>  compacting.
> >> >> >>> >> I looked in greater detail on one sstable and the row length
> was
> >> >> >>> >> longer than
> >> >> >>> >> the actual data in the row, and as far as I could see, either
> >> >> >>> >> the
> >> >> >>> >> length was
> >> >> >>> >> wrong or the row was missing data as there was was no extra
> data
> >> >> >>> >> in
> >> >> >>> >> the row
> >> >> >>> >> after the last column.
> >> >> >>> >> This was however on a somewhat aging dataset, so suspected it
> >> >> >>> >> could
> >> >> >>> >> be
> >> >> >>> >> related to 2376.
> >> >> >>> >>
> >> >> >>> >> Playing around with 0.8 at the moment and not seen it there
> >> >> >>> >> yet....
> >> >> >>> >> (bet it
> >> >> >>> >> will show up tomorrow once I wrote that.. :))
> >> >> >>> >> Terje
> >> >> >>> >>
> >> >> >>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
> >> >> >>> >> <sanjeev@locomatix.com>
> >> >> >>> >> wrote:
> >> >> >>> >>>
> >> >> >>> >>> Hi Sylvain,
> >> >> >>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
> >> >> >>> >>> Thanks!
> >> >> >>> >>>
> >> >> >>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
> >> >> >>> >>> <sylvain@datastax.com>
> >> >> >>> >>> wrote:
> >> >> >>> >>>>
> >> >> >>> >>>> Hi Sanjeev,
> >> >> >>> >>>>
> >> >> >>> >>>> What's the story of the cluster ? Did you started with
> 0.7.4,
> >> >> >>> >>>> or
> >> >> >>> >>>> is
> >> >> >>> >>>> it
> >> >> >>> >>>> upgraded from
> >> >> >>> >>>> some earlier version ?
> >> >> >>> >>>>
> >> >> >>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
> >> >> >>> >>>> <sanjeev@locomatix.com>
> >> >> >>> >>>> wrote:
> >> >> >>> >>>>> Hey guys,
> >> >> >>> >>>>> Running a one node cassandra server with version 0.7.4
> >> >> >>> >>>>> patched
> >> >> >>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
> >> >> >>> >>>>> The system was running fine for a couple of days when we
> >> >> >>> >>>>> started
> >> >> >>> >>>>> noticing
> >> >> >>> >>>>> something strange with cassandra. I stopped all
> applications
> >> >> >>> >>>>> and
> >> >> >>> >>>>> restarted
> >> >> >>> >>>>> cassandra. And then did a scrub. During scrub, I noticed
> >> >> >>> >>>>> these
> >> >> >>> >>>>> in
> >> >> >>> >>>>> the
> >> >> >>> >>>>> logs
> >> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >> >> >>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
> >> >> >>> >>>>> 1516029079813320210
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >> >>> >>>>>        at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >> >>> >>>>>    at
> >> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>>        at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>>        at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>> Caused by: java.io.IOException: Impossible row size
> >> >> >>> >>>>> 1516029079813320210
> >> >> >>> >>>>>    ... 8 more
> >> >> >>> >>>>>  INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 613) Retrying from row index; data is -1768177699
> bytes
> >> >> >>> >>>>> starting
> >> >> >>> >>>>> at
> >> >> >>> >>>>> 2626524914
> >> >> >>> >>>>>  WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 633) Retry failed too.  Skipping to next row (retry's
> >> >> >>> >>>>> stacktrace
> >> >> >>> >>>>> follows)
> >> >> >>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims
> to
> >> >> >>> >>>>> be
> >> >> >>> >>>>> 1868982636
> >> >> >>> >>>>> bytes, longer than entire row size -1768177699        at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> >> >> >>> >>>>>        at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >> >>> >>>>>        at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >> >>> >>>>>  at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>>        at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
> >> >> >>> >>>>> 1868982636
> >> >> >>> >>>>> bytes,
> >> >> >>> >>>>> longer than entire row size -1768177699        at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> >> >> >>> >>>>>        ... 8 more
> >> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >> >> >>> >>>>> java.io.IOError: java.io.EOFException
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>>         at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>> Caused by: java.io.EOFException
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> >> >> >>> >>>>>         ... 22 more
> >> >> >>> >>>>>  INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes
> >> >> >>> >>>>> starting
> >> >> >>> >>>>> at
> >> >> >>> >>>>> 2229643127
> >> >> >>> >>>>>
> >> >> >>> >>>>> And then when i restarted the readers, i get the following
> >> >> >>> >>>>> crash
> >> >> >>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
> >> >> >>> >>>>> AbstractCassandraDaemon.java
> >> >> >>> >>>>> (line 112) Fatal exception in thread
> >> >> >>> >>>>> Thread[ReadStage:24,5,main]
> >> >> >>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining
> >> >> >>> >>>>> is
> >> >> >>> >>>>> 791462117 but
> >> >> >>> >>>>> 1970433058 requested
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>> org.apache.cassandra.db.Table.getRow(Table.java:333)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>>         at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>>         at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>>
> >> >> >>> >>>>> Any ideas?
> >> >> >>> >>>>> Thanks!
> >> >> >>> >>>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > --
> >> >> >>> > Jonathan Ellis
> >> >> >>> > Project Chair, Apache Cassandra
> >> >> >>> > co-founder of DataStax, the source for professional Cassandra
> >> >> >>> > support
> >> >> >>> > http://www.datastax.com
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Jonathan Ellis
> >> >> Project Chair, Apache Cassandra
> >> >> co-founder of DataStax, the source for professional Cassandra support
> >> >> http://www.datastax.com
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Mime
View raw message