incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kais Ahmed <k...@neteck-fr.com>
Subject MemtablePostFlusher pending
Date Mon, 21 Oct 2013 09:17:31 GMT
Hi all,

We have recently run in production a new cluster C* 2.0.0 with 3 nodes RF 3.

After 7 days, we seen on log this error on one node :

ERROR [FlushWriter:216] 2013-10-07 07:11:46,538 CassandraDaemon.java (line
186) Exception in thread Thread[FlushWriter:216,5,main]
java.lang.AssertionError
        at
org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:198)
        at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:186)
        at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417)
        at
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:376)
        at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)


$ nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked  All
time blocked
ReadStage                         0         0       20102712
0                 0
RequestResponseStage              0         0       12186090
0                 0
MutationStage                     0         0       17999146
0                 0
ReadRepairStage                   0         0        1013914
0                 0
ReplicateOnWriteStage             0         0          77415
0                 0
GossipStage                       0         0         234894
0                 0
AntiEntropyStage                  0         0              0
0                 0
MigrationStage                    0         0              4
0                 0
MemoryMeter                       0         0            475
0                 0
MemtablePostFlusher               1        59            210
0                 0
FlushWriter                       0         0            248
0               155
MiscStage                         0         0              0
0                 0
commitlog_archiver                0         0              0
0                 0
InternalResponseStage             0         0              3
0                 0
HintedHandoff                     0         0              5
0                 0

$ nodetool compactionstats
pending tasks: 72
Active compaction remaining time :        n/a

A few days later, the other two nodes had this error, since it seems that
the active thread on MemtablePostFlusher blocks any compaction, I can not
make snapshot, or drain, I tried rebooted servers one by one, it works
again for
a moment, but after a few days in the log I have this error again. I also
had loss of data that are occurring after the reboot. Now I do not know how
to fix the problem, I think the solution would be to install another
cluster and
copy the data, except if one of you had this problem and could guide me.

Thanks a lot,

Mime
View raw message