Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52951963E for ; Wed, 5 Oct 2011 08:33:52 +0000 (UTC) Received: (qmail 64833 invoked by uid 500); 5 Oct 2011 08:33:50 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 64803 invoked by uid 500); 5 Oct 2011 08:33:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 64794 invoked by uid 99); 5 Oct 2011 08:33:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Oct 2011 08:33:50 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of watcherfr@gmail.com designates 74.125.82.44 as permitted sender) Received: from [74.125.82.44] (HELO mail-ww0-f44.google.com) (74.125.82.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Oct 2011 08:33:41 +0000 Received: by wwf22 with SMTP id 22so1701504wwf.25 for ; Wed, 05 Oct 2011 01:33:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=NuBisgwG8+5Ut7FzyCTK/Ns7Ayc/4qmINqti52lSpbw=; b=Gv6SPmoR/r0AWi8w7HD+OR256lI7fow2yfVEPnmnqZPdaeFlmXCRPxhihgll8iq/x1 wGYM9yHX7PRWjBMxLLC+qidxHlLXCxvb7uSOm01jWUgyiy4QrNKzrX9mvf9eOuydCvlA 7fdmhXFpNhADkdAlzm60UigsjsdQMRjNwEUdQ= MIME-Version: 1.0 Received: by 10.216.143.153 with SMTP id l25mr5983798wej.24.1317803601491; Wed, 05 Oct 2011 01:33:21 -0700 (PDT) Received: by 10.180.106.67 with HTTP; Wed, 5 Oct 2011 01:33:21 -0700 (PDT) Date: Wed, 5 Oct 2011 10:33:21 +0200 Message-ID: Subject: Why is mutation stage increasing ?? From: Philippe To: user Content-Type: multipart/alternative; boundary=0016e6dd86208cf88104ae890fbb X-Virus-Checked: Checked by ClamAV on apache.org --0016e6dd86208cf88104ae890fbb Content-Type: text/plain; charset=ISO-8859-1 Hello, I have my 3-node, RF=3 cluster acting strangely. Can someone shed a light as to what is going on ? It was stuck for a couple of hours (all clients TimedOut). nodetool tpstats showed huge increasing MutationStages (in the hundreds of thousands). I restarted one node and it took a while to reply GB of commitlog. I've shutdown all clients that write to the cluster and it's just weird All nodes are still showing huge MutationStages including the new one and it's either increasing or stable. The pending count is stuck at 32. Compactionstats shows no compaction on 2 nodes and dozens of Scrub compactions (all at 100%) on the 3rd one. This is a scrub I did last week when I encountered assertion errors. Netstats shows no streams being exchanged at any node but each on is expecting a few Responses. Any ideas ? Thanks For example (increased to 567062 while I was writing this email) Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 18372664517 0 0 RequestResponseStage 0 0 10731370183 0 0 MutationStage 32 565879 295492216 0 0 ReadRepairStage 0 0 23654 0 0 ReplicateOnWriteStage 0 0 7733659 0 0 GossipStage 0 0 3502922 0 0 AntiEntropyStage 0 0 1631 0 0 MigrationStage 0 0 0 0 0 MemtablePostFlusher 0 0 5716 0 0 StreamStage 0 0 10 0 0 FlushWriter 0 0 5714 0 499 FILEUTILS-DELETE-POOL 0 0 773 0 0 MiscStage 0 0 1266 0 0 FlushSorter 0 0 0 0 0 AntiEntropySessions 0 0 18 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 1798 0 0 Mode: Normal Not sending any streams. Not receiving any streams. Pool Name Active Pending Completed Commands n/a 0 1223769753 Responses n/a 4 1627481305 --0016e6dd86208cf88104ae890fbb Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: base64 SGVsbG8sPGJyPkkgaGF2ZSBteSAzLW5vZGUsIFJGPTMgY2x1c3RlciBhY3Rpbmcgc3RyYW5nZWx5 LiBDYW4gc29tZW9uZSBzaGVkIGEgbGlnaHQgYXMgdG8gd2hhdCBpcyBnb2luZyBvbiA/PGJyPkl0 IHdhcyBzdHVjayBmb3IgYSBjb3VwbGUgb2YgaG91cnMgKGFsbCBjbGllbnRzIFRpbWVkT3V0KS4g bm9kZXRvb2wgdHBzdGF0cyBzaG93ZWQgaHVnZSBpbmNyZWFzaW5nIE11dGF0aW9uU3RhZ2VzIChp biB0aGUgaHVuZHJlZHMgb2YgdGhvdXNhbmRzKS48YnI+CkkgcmVzdGFydGVkIG9uZSBub2RlIGFu ZCBpdCB0b29rIGEgd2hpbGUgdG8gcmVwbHkgR0Igb2YgY29tbWl0bG9nLiBJJiMzOTt2ZSBzaHV0 ZG93biBhbGwgY2xpZW50cyB0aGF0IHdyaXRlIHRvIHRoZSBjbHVzdGVyIGFuZCBpdCYjMzk7cyBq dXN0IHdlaXJkPGJyPjxicj5BbGwgbm9kZXMgYXJlIHN0aWxsIHNob3dpbmcgaHVnZSBNdXRhdGlv blN0YWdlcyBpbmNsdWRpbmcgdGhlIG5ldyBvbmUgYW5kIGl0JiMzOTtzIGVpdGhlciBpbmNyZWFz aW5nIG9yIHN0YWJsZS4gVGhlIHBlbmRpbmcgY291bnQgaXMgc3R1Y2sgYXQgMzIuPGJyPgpDb21w YWN0aW9uc3RhdHMgc2hvd3Mgbm8gY29tcGFjdGlvbiBvbiAyIG5vZGVzIGFuZCBkb3plbnMgb2Yg U2NydWIgY29tcGFjdGlvbnMgKGFsbCBhdCAxMDAlKSBvbiB0aGUgM3JkIG9uZS4gVGhpcyBpcyBh IHNjcnViIEkgZGlkIGxhc3Qgd2VlayB3aGVuIEkgZW5jb3VudGVyZWQgYXNzZXJ0aW9uIGVycm9y cy48YnI+TmV0c3RhdHMgc2hvd3Mgbm8gc3RyZWFtcyBiZWluZyBleGNoYW5nZWQgYXQgYW55IG5v ZGUgYnV0IGVhY2ggb24gaXMgZXhwZWN0aW5nIGEgZmV3IFJlc3BvbnNlcy48YnI+Cjxicj5Bbnkg aWRlYXMgPzxicj5UaGFua3M8YnI+PGJyPkZvciBleGFtcGxlIChpbmNyZWFzZWQgdG8gNTY3MDYy IHdoaWxlIEkgd2FzIHdyaXRpbmcgdGhpcyBlbWFpbCk8YnI+PHNwYW4gc3R5bGU9ImZvbnQtZmFt aWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5Qb29sIE5hbWWgoKCgoKCgoKCgoKCgoKCgoKCg IEFjdGl2ZaCgIFBlbmRpbmegoKCgoCBDb21wbGV0ZWSgoCBCbG9ja2VkoCBBbGwgdGltZSBibG9j a2VkPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsi Pgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPlJlYWRT dGFnZaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCAwoKCgoKCgoKAgMKCgoCAxODM3MjY2NDUxN6Cg oKCgoKCgIDCgoKCgoKCgoKCgoKCgoKCgIDA8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTog Y291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVy IG5ldyxtb25vc3BhY2U7Ij5SZXF1ZXN0UmVzcG9uc2VTdGFnZaCgoKCgoKCgoKCgoKAgMKCgoKCg oKCgIDCgoKAgMTA3MzEzNzAxODOgoKCgoKCgoCAwoKCgoKCgoKCgoKCgoKCgoCAwPC9zcGFuPjxi ciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHls ZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPk11dGF0aW9uU3RhZ2WgoKCg oKCgoKCgoKCgoKCgoKCgIDMyoKCgIDU2NTg3OaCgoKCgIDI5NTQ5MjIxNqCgoKCgoKCgIDCgoKCg oKCgoKCgoKCgoKCgIDA8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcs bW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3Bh Y2U7Ij5SZWFkUmVwYWlyU3RhZ2WgoKCgoKCgoKCgoKCgoKCgoKAgMKCgoKCgoKCgIDCgoKCgoKCg oKAgMjM2NTSgoKCgoKCgoCAwoKCgoKCgoKCgoKCgoKCgoCAwPC9zcGFuPjxiciBzdHlsZT0iZm9u dC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1p bHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPlJlcGxpY2F0ZU9uV3JpdGVTdGFnZaCgoKCgoKCg oKCgoCAwoKCgoKCgoKAgMKCgoKCgoKAgNzczMzY1OaCgoKCgoKCgIDCgoKCgoKCgoKCgoKCgoKCg IDA8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+ PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5Hb3NzaXBT dGFnZaCgoKCgoKCgoKCgoKCgoKCgoKCgoKAgMKCgoKCgoKCgIDCgoKCgoKCgIDM1MDI5MjKgoKCg oKCgoCAwoKCgoKCgoKCgoKCgoKCgoCAwPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNv dXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIg bmV3LG1vbm9zcGFjZTsiPkFudGlFbnRyb3B5U3RhZ2WgoKCgoKCgoKCgoKCgoKCgoCAwoKCgoKCg oKAgMKCgoKCgoKCgoKAgMTYzMaCgoKCgoKCgIDCgoKCgoKCgoKCgoKCgoKCgIDA8L3NwYW4+PGJy IHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9 ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5NaWdyYXRpb25TdGFnZaCgoKCg oKCgoKCgoKCgoKCgoKAgMKCgoKCgoKCgIDCgoKCgoKCgoKCgoKCgIDCgoKCgoKCgoCAwoKCgoKCg oKCgoKCgoKCgoCAwPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1v bm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFj ZTsiPk1lbXRhYmxlUG9zdEZsdXNoZXKgoKCgoKCgoKCgoKCgoCAwoKCgoKCgoKAgMKCgoKCgoKCg oKAgNTcxNqCgoKCgoKCgIDCgoKCgoKCgoKCgoKCgoKCgIDA8L3NwYW4+PGJyIHN0eWxlPSJmb250 LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5 OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5TdHJlYW1TdGFnZaCgoKCgoKCgoKCgoKCgoKCgoKCg oKAgMKCgoKCgoKCgIDCgoKCgoKCgoKCgoKAgMTCgoKCgoKCgoCAwoKCgoKCgoKCgoKCgoKCgoCAw PC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8 c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPkZsdXNoV3Jp dGVyoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCAwoKCgoKCgoKAgMKCgoKCgoKCgoKAgNTcxNKCgoKCg oKCgIDCgoKCgoKCgoKCgoKCgoCA0OTk8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291 cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5l dyxtb25vc3BhY2U7Ij5GSUxFVVRJTFMtREVMRVRFLVBPT0ygoKCgoKCgoKCgoKAgMKCgoKCgoKCg IDCgoKCgoKCgoKCgoCA3NzOgoKCgoKCgoCAwoKCgoKCgoKCgoKCgoKCgoCAwPC9zcGFuPjxiciBz dHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0i Zm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPk1pc2NTdGFnZaCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoCAwoKCgoKCgoKAgMKCgoKCgoKCgoKAgMTI2NqCgoKCgoKCgIDCgoKCgoKCg oKCgoKCgoKCgIDA8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9u b3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7 Ij5GbHVzaFNvcnRlcqCgoKCgoKCgoKCgoKCgoKCgoKCgoKAgMKCgoKCgoKCgIDCgoKCgoKCgoKCg oKCgIDCgoKCgoKCgoCAwoKCgoKCgoKCgoKCgoKCgoCAwPC9zcGFuPjxiciBzdHlsZT0iZm9udC1m YW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6 IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPkFudGlFbnRyb3B5U2Vzc2lvbnOgoKCgoKCgoKCgoKCg oCAwoKCgoKCgoKAgMKCgoKCgoKCgoKCgoCAxOKCgoKCgoKCgIDCgoKCgoKCgoKCgoKCgoKCgIDA8 L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNw YW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5JbnRlcm5hbFJl c3BvbnNlU3RhZ2WgoKCgoKCgoKCgoKAgMKCgoKCgoKCgIDCgoKCgoKCgoKCgoKCgIDCgoKCgoKCg oCAwoKCgoKCgoKCgoKCgoKCgoCAwPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJp ZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3 LG1vbm9zcGFjZTsiPkhpbnRlZEhhbmRvZmagoKCgoKCgoKCgoKCgoKCgoKCgoCAwoKCgoKCgoKAg MKCgoKCgoKCgoKAgMTc5OKCgoKCgoKCgIDCgoKCgoKCgoKCgoKCgoKCgIDA8L3NwYW4+PGJyIHN0 eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PGJyPjxicj48c3BhbiBz dHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPk1vZGU6IE5vcm1hbDwv c3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KPHNw YW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5Ob3Qgc2VuZGlu ZyBhbnkgc3RyZWFtcy48L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcs bW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3Bh Y2U7Ij5Ob3QgcmVjZWl2aW5nIGFueSBzdHJlYW1zLjwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFt aWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KPHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBj b3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5Qb29sIE5hbWWgoKCgoKCgoKCgoKCgoKCgoKCgIEFjdGl2 ZaCgIFBlbmRpbmegoKCgoCBDb21wbGV0ZWQ8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTog Y291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVy IG5ldyxtb25vc3BhY2U7Ij5Db21tYW5kc6CgoKCgoKCgoKCgoKCgoKCgoKCgoKCgIG4vYaCgoKCg oKCgIDCgoKCgIDEyMjM3Njk3NTM8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmll ciBuZXcsbW9ub3NwYWNlOyI+CjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcs bW9ub3NwYWNlOyI+UmVzcG9uc2VzoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCBuL2GgoKCgoKCgoCA0 oKCgoCAxNjI3NDgxMzA1PC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3 LG1vbm9zcGFjZTsiPjxicj4K --0016e6dd86208cf88104ae890fbb--