Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D22FF11977 for ; Wed, 16 Jul 2014 14:23:28 +0000 (UTC) Received: (qmail 68273 invoked by uid 500); 16 Jul 2014 14:23:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 68238 invoked by uid 500); 16 Jul 2014 14:23:25 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 68228 invoked by uid 99); 16 Jul 2014 14:23:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 14:23:25 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of horschi@gmail.com designates 209.85.220.52 as permitted sender) Received: from [209.85.220.52] (HELO mail-pa0-f52.google.com) (209.85.220.52) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 14:23:20 +0000 Received: by mail-pa0-f52.google.com with SMTP id bj1so1400170pad.39 for ; Wed, 16 Jul 2014 07:23:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=rnLlVg5w0Gezz8GqzBwI0u7x/VT++akubISBh2tDQFM=; b=fBI5T5TBF+axFJU43JP66oaWjlBU7Ezk9D89i2/uxmGNUKN9N7CA5YXJDqixVqjRue mPdFERFnLv5jOw8LOfxVrzihNFTu/1aqwpwG99fNcTxnXqISB0bgFwOqoVhDhdfBbIlf OXGuQFCQMCUUTtNegTsUNBfh1dBhus1Zyi1Glc4G3TjlIC040lxH335fzILpP33phaiT cPiLhOg5swExcdLL/MxIeq7yoQG/KBzDHBHiGL+3qTAGqVqtn0g6WflLwtwLLCBbcNge JSf9r+lhQpYIOPRW4ru3aR/n+6Stv0ojLRCoE759aHbnqk3JpZBwVTsPee5nXCyn7Mwz r7fA== MIME-Version: 1.0 X-Received: by 10.66.235.232 with SMTP id up8mr29768767pac.55.1405520580265; Wed, 16 Jul 2014 07:23:00 -0700 (PDT) Received: by 10.70.92.199 with HTTP; Wed, 16 Jul 2014 07:23:00 -0700 (PDT) In-Reply-To: References: <17302FFC-EECB-45AE-BF48-569F36C1D9CD@blackbirdit.com> Date: Wed, 16 Jul 2014 16:23:00 +0200 Message-ID: Subject: Re: MemtablePostFlusher and FlushWriter From: horschi To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a1136c94ee9495004fe5042c9 X-Virus-Checked: Checked by ClamAV on apache.org --001a1136c94ee9495004fe5042c9 Content-Type: text/plain; charset=UTF-8 Hi Ahmed, this exception is caused by you creating rows with a key-length of more than 64kb. Your key is 394920 bytes long it seems. Keys and column-names are limited to 64kb. Only values may be larger. I cannot say for sure if this is the cause of your high MemtablePostFlusher pending count, but I would say it is possible. kind regards, Christian PS: I still use good old thrift lingo. On Wed, Jul 16, 2014 at 3:14 PM, Kais Ahmed wrote: > Hi chris, christan, > > Thanks for reply, i'm not using DSE. > > I have in the log files, this error that appear two times. > > ERROR [FlushWriter:3456] 2014-07-01 18:25:33,607 CassandraDaemon.java > (line 196) Exception in thread Thread[FlushWriter:3456,5,main] > java.lang.AssertionError: 394920 > at > org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:342) > at > org.apache.cassandra.db.ColumnIndex$Builder.maybeWriteRowHeader(ColumnIndex.java:201) > at > org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:188) > at > org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:133) > at > org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:202) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:187) > at > org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:365) > at > org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:318) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > It's the same error than this link > http://mail-archives.apache.org/mod_mbox/cassandra-user/201305.mbox/%3CBAY169-W52699DD7A1C0007783F8D8A8920@phx.gbl%3E > , > with the same configuration 2 nodes RF 2 with SimpleStrategy. > > Hope this help. > > Thanks, > > > > 2014-07-16 1:49 GMT+02:00 Chris Lohfink : > > The MemtablePostFlusher is also used for flushing non-cf backed (solr) >> indexes. Are you using DSE and solr by chance? >> >> Chris >> >> On Jul 15, 2014, at 5:01 PM, horschi wrote: >> >> I have seen this behavour when Commitlog files got deleted (or >> permissions were set to read only). >> >> MemtablePostFlusher is the stage that marks the Commitlog as flushed. >> When they fail it usually means there is something wrong with the commitlog >> files. >> >> Check your logfiles for any commitlog related errors. >> >> regards, >> Christian >> >> >> On Tue, Jul 15, 2014 at 7:03 PM, Kais Ahmed wrote: >> >>> Hi all, >>> >>> I have a small cluster (2 nodes RF 2) running with C* 2.0.6 on I2 Extra >>> Large (AWS) with SSD disk, >>> the nodetool tpstats shows many MemtablePostFlusher pending and >>> FlushWriter All time blocked. >>> >>> The two nodes have the default configuration. All CF use size-tiered >>> compaction strategy. >>> >>> There are 10 times more reads than writes (1300 reads/s and 150 >>> writes/s). >>> >>> >>> ubuntu@node1 :~$ nodetool tpstats >>> Pool Name Active Pending Completed Blocked >>> All time blocked >>> MemtablePostFlusher 1 1158 159590 >>> 0 0 >>> FlushWriter 0 0 11568 >>> 0 1031 >>> >>> ubuntu@node1:~$ nodetool compactionstats >>> pending tasks: 90 >>> Active compaction remaining time : n/a >>> >>> >>> ubuntu@node2:~$ nodetool tpstats >>> Pool Name Active Pending Completed Blocked >>> All time blocked >>> MemtablePostFlusher 1 1020 50987 >>> 0 0 >>> FlushWriter 0 0 6672 >>> 0 948 >>> >>> >>> ubuntu@node2:~$ nodetool compactionstats >>> pending tasks: 89 >>> Active compaction remaining time : n/a >>> >>> I think there is something wrong, thank you for your help. >>> >>> >> >> > --001a1136c94ee9495004fe5042c9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Ahmed,

this exception = is caused by you creating rows with a key-length of more than 64kb. Your ke= y is 394920 bytes long it seems.

Keys and column-names are limited t= o 64kb. Only values may be larger.

I cannot say for sure if this is the cause of your high MemtableP= ostFlusher pending count, but I would say it is possible.

kind= regards,
Christian

PS: I still use good old thrift lingo. =






On Wed, Jul 16, 2014 at 3:14 PM, = Kais Ahmed <kais@neteck-fr.com> wrote:
Hi chris, ch= ristan,

Thanks for reply, i'm not using DSE.

I have in the log files, this error that appear two times.

ERROR [FlushWriter:3456] 2014-07-01 18:25:33,607 CassandraDaemon.java (= line 196) Exception in thread Thread[FlushWriter:3456,5,main]
java.lang.AssertionError: 394920
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteB= ufferUtil.java:342)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.ap= ache.cassandra.db.ColumnIndex$Builder.maybeWriteRowHeader(ColumnIndex.java:= 201)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.Colum= nIndex$Builder.add(ColumnIndex.java:188)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnInd= ex.java:133)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.ca= ssandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:202)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.io.sstab= le.SSTableWriter.append(SSTableWriter.java:187)
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.Memtable$FlushRunnable.writeS= ortedContents(Memtable.java:365)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java= :318)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.io.util.= DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.utils.WrappedRunnable.r= un(WrappedRunnable.java:28)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a= t java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1145)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPo= olExecutor$Worker.run(ThreadPoolExecutor.java:615)
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:744)


<= /div>
with the same configuration 2 nodes RF 2 with SimpleStrategy.
Hope this help.

Thanks,



2014-07-16 = 1:49 GMT+02:00 Chris Lohfink <clohfink@blackbirdit.com>:

The Memt= ablePostFlusher is also used for flushing non-cf backed (solr) indexes. =C2= =A0Are you using DSE and solr by chance?

Chris
<= div>
On Jul 15, 2014, at 5:01 PM, horschi <horschi@gmail.com> wrote:
I have seen this behavo= ur when Commitlog files got deleted (or permissions were set to read only).=

MemtablePostFlusher is the stage that marks the Commitlo= g as flushed. When they fail it usually means there is something wrong with= the commitlog files.

Check your logfiles for any commitlog related errors.
regards,
Christian


On Tue, Jul 15, 2014 at 7:03 PM, Kais A= hmed <kais@neteck-fr.com> wrote:
Hi all,

I= have a small cluster (2 nodes RF 2)=C2=A0 running with C* 2.0.6 on I2 Extr= a Large (AWS) with SSD disk,
the nodetool tpstats shows many MemtablePostFlusher pending and FlushWriter= All time blocked.

The two nodes have the default configuration. All CF use size-tiered co= mpaction strategy.

There are 10 times more reads than writes (1300 r= eads/s and 150 writes/s).


ubuntu@node1 :~$ nodetool = tpstats
Pool Name=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Active=C2=A0=C2=A0 Pending= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Completed=C2=A0=C2=A0 Blocked=C2=A0 All time= blocked
MemtablePostFlusher=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = 1158=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 159590=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0
FlushWriter=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 11568=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 1031

ubuntu@node1:= ~$ nodetool compactionstats
pending tasks: 90
Active compaction remaining time :=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 n/a


ubuntu@node2:~$ nodetool tpstatsPool Name=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Active=C2=A0=C2=A0 Pend= ing=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Completed=C2=A0=C2=A0 Blocked=C2=A0 All t= ime blocked
MemtablePostFlusher=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 1020=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 50987=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0
FlushWriter=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 6672=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 948
<= br>
ubuntu@node2:~$ nodetool compactionstats
pending tasks: 89
Act= ive compaction remaining time :=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 n= /a

I think there is something wrong, thank you for your help.





--001a1136c94ee9495004fe5042c9--