cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hendry" <dan.hendry.j...@gmail.com>
Subject RE: 1.0.2 Assertion Error
Date Thu, 10 Nov 2011 20:04:34 GMT
Just happened again, seems to be with the same column family (at least on a
flusher thread for which the last activity was flushing a memtable for that
CF).

 

It also looks like MemtablePostFlusher tasks blocked (and not getting
cleared) as evidenced by tpstats:

 

Pool Name                    Active   Pending      Completed   Blocked  All
time blocked

MemtablePostFlusher               1        18             16         0
0

 

Dan

 

From: Dan Hendry [mailto:dan.hendry.junk@gmail.com] 
Sent: November-10-11 14:49
To: 'user@cassandra.apache.org'
Subject: 1.0.2 Assertion Error

 

Just saw this weird assertion after upgrading one of my nodes from 0.8.6 to
1.0.2 (its been running fine for a few hours now):

                                                                     

INFO [FlushWriter:9] 2011-11-10 13:08:58,882 Memtable.java (line 237)
Writing Memtable-Data@1388955390(25676955/430716097 serialized/live bytes,
478913 ops)

ERROR [FlushWriter:9] 2011-11-10 13:08:59,513 AbstractCassandraDaemon.java
(line 133) Fatal exception in thread Thread[FlushWriter:9,5,main]

java.lang.AssertionError: CF size changed during serialization: was 4
initially but 3 written

        at
org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFam
ilySerializer.java:94)

        at
org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFa
milySerializer.java:112)

        at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:177)

        at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:264)

        at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)

        at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)

       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:662)

 

 

After taking a quick peek at the code, it looks like the numbers ("4
initially but 3 written") refer to the number of columns (in the memtable?).
Given the byte size of the memtable being flushed, there are *certainly*
more than 4 columns. Besides this error, there are no other unexpected or
unusual log entries and the node seems to be behaving normally.

 

Thoughts?

 

Dan Hendry


Mime
View raw message