cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hendry" <dan.hendry.j...@gmail.com>
Subject RE: 1.0.2 Assertion Error
Date Thu, 10 Nov 2011 21:18:29 GMT
https://issues.apache.org/jira/browse/CASSANDRA-3482 

I restarted the node and the problem has cropped up again. 

Is it possible to downgrade back to 0.8? Is there any way to convert 'h'
version SSTables to the old 'g' version? Any other data file changes to be
aware of?

Dan

-----Original Message-----
From: Sylvain Lebresne [mailto:sylvain@datastax.com] 
Sent: November-10-11 15:23
To: user@cassandra.apache.org
Subject: Re: 1.0.2 Assertion Error

That would be a bug (as any assertion error would be), likely some
race condition.
Could you open a ticket?
The fact that this block the MemtablePostFlusher is unfortunately
related. Restarting the
node would fix but we need to make that more solid too.

--
Sylvain

On Thu, Nov 10, 2011 at 9:04 PM, Dan Hendry <dan.hendry.junk@gmail.com>
wrote:
> Just happened again, seems to be with the same column family (at least on
a
> flusher thread for which the last activity was flushing a memtable for
that
> CF).
>
>
>
> It also looks like MemtablePostFlusher tasks blocked (and not getting
> cleared) as evidenced by tpstats:
>
>
>
> Pool Name                    Active   Pending      Completed  
Blocked 
All
> time blocked
>
> MemtablePostFlusher               1        18          
  16
> 0                 0
>
>
>
> Dan
>
>
>
> From: Dan Hendry [mailto:dan.hendry.junk@gmail.com]
> Sent: November-10-11 14:49
> To: 'user@cassandra.apache.org'
> Subject: 1.0.2 Assertion Error
>
>
>
> Just saw this weird assertion after upgrading one of my nodes from 0.8.6
to
> 1.0.2 (its been running fine for a few hours now):
>
>
>
> INFO [FlushWriter:9] 2011-11-10 13:08:58,882 Memtable.java (line 237)
> Writing Memtable-Data@1388955390(25676955/430716097 serialized/live bytes,
> 478913 ops)
>
> ERROR [FlushWriter:9] 2011-11-10 13:08:59,513 AbstractCassandraDaemon.java
> (line 133) Fatal exception in thread Thread[FlushWriter:9,5,main]
>
> java.lang.AssertionError: CF size changed during serialization: was 4
> initially but 3 written
>
>         at
>
org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFam
ilySerializer.java:94)
>
>         at
>
org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFa
milySerializer.java:112)
>
>         at
>
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:177)
>
>         at
> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:264)
>
>         at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
>
>         at
org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
>
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>        at
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
>
>         at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
>
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
>
> After taking a quick peek at the code, it looks like the numbers (“4
> initially but 3 written”) refer to the number of columns (in the
memtable?).
> Given the byte size of the memtable being flushed, there are *certainly*
> more than 4 columns. Besides this error, there are no other unexpected or
> unusual log entries and the node seems to be behaving normally.
>
>
>
> Thoughts?
>
>
>
> Dan Hendry
No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.920 / Virus Database: 271.1.1/4007 - Release Date: 11/10/11
02:34:00


Mime
View raw message