cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sylvain Lebresne <sylv...@datastax.com>
Subject Re: 1.0.2 Assertion Error
Date Thu, 10 Nov 2011 20:22:44 GMT
That would be a bug (as any assertion error would be), likely some
race condition.
Could you open a ticket?
The fact that this block the MemtablePostFlusher is unfortunately
related. Restarting the
node would fix but we need to make that more solid too.

--
Sylvain

On Thu, Nov 10, 2011 at 9:04 PM, Dan Hendry <dan.hendry.junk@gmail.com> wrote:
> Just happened again, seems to be with the same column family (at least on a
> flusher thread for which the last activity was flushing a memtable for that
> CF).
>
>
>
> It also looks like MemtablePostFlusher tasks blocked (and not getting
> cleared) as evidenced by tpstats:
>
>
>
> Pool Name                    Active   Pending      Completed  
Blocked  All
> time blocked
>
> MemtablePostFlusher               1        18          
  16
> 0                 0
>
>
>
> Dan
>
>
>
> From: Dan Hendry [mailto:dan.hendry.junk@gmail.com]
> Sent: November-10-11 14:49
> To: 'user@cassandra.apache.org'
> Subject: 1.0.2 Assertion Error
>
>
>
> Just saw this weird assertion after upgrading one of my nodes from 0.8.6 to
> 1.0.2 (its been running fine for a few hours now):
>
>
>
> INFO [FlushWriter:9] 2011-11-10 13:08:58,882 Memtable.java (line 237)
> Writing Memtable-Data@1388955390(25676955/430716097 serialized/live bytes,
> 478913 ops)
>
> ERROR [FlushWriter:9] 2011-11-10 13:08:59,513 AbstractCassandraDaemon.java
> (line 133) Fatal exception in thread Thread[FlushWriter:9,5,main]
>
> java.lang.AssertionError: CF size changed during serialization: was 4
> initially but 3 written
>
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:94)
>
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:112)
>
>         at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:177)
>
>         at
> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:264)
>
>         at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
>
>         at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
>
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
>
> After taking a quick peek at the code, it looks like the numbers (“4
> initially but 3 written”) refer to the number of columns (in the memtable?).
> Given the byte size of the memtable being flushed, there are *certainly*
> more than 4 columns. Besides this error, there are no other unexpected or
> unusual log entries and the node seems to be behaving normally.
>
>
>
> Thoughts?
>
>
>
> Dan Hendry

Mime
View raw message