Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CDBC817209 for ; Thu, 26 Mar 2015 08:57:30 +0000 (UTC) Received: (qmail 10861 invoked by uid 500); 26 Mar 2015 08:57:28 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 10817 invoked by uid 500); 26 Mar 2015 08:57:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 10807 invoked by uid 99); 26 Mar 2015 08:57:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Mar 2015 08:57:27 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of anishek@gmail.com designates 74.125.82.51 as permitted sender) Received: from [74.125.82.51] (HELO mail-wg0-f51.google.com) (74.125.82.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Mar 2015 08:57:22 +0000 Received: by wgra20 with SMTP id a20so56119542wgr.3 for ; Thu, 26 Mar 2015 01:56:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=cGg1QeRoc2oMz1umduvaf8w6NL9XBQ/acIFxf95g2w8=; b=DBhKmyauaJOcifVreO5di0Nl8rCOh8WL+8GbHRPKvENUQN9qAFRFrsoYZ1n7j0RgdE zHeUpBI4dr7ClptnwrYaTaQp5Ux/QmIYH8dBeRXbzeyYKoLk0htwoiv8lJxOiTGj5DMq TQI/1LgQ5qqhuvpVdQWEfZS3xgPUgMUPQR54iKtJd9cP0asAHjgmlbTiNiVaS3GpEkCx SaJqX780kOnTL/l4cRgABPAk5haVkCmTgNL1uqm9pRbXPVlowt2Y4YhgOubbD9rb5W/z g7kpxa8wgpLQnFGa9BiWeGt0eTx/GuBkb+QZHMiyqLKnmvQIca65XaKuGGodlsDDR23F qDaw== MIME-Version: 1.0 X-Received: by 10.194.192.167 with SMTP id hh7mr26084745wjc.151.1427360176742; Thu, 26 Mar 2015 01:56:16 -0700 (PDT) Received: by 10.27.51.73 with HTTP; Thu, 26 Mar 2015 01:56:16 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Mar 2015 14:26:16 +0530 Message-ID: Subject: Re: Disastrous profusion of SSTables From: Anishek Agarwal To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=047d7b8743f84d253905122d30c7 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b8743f84d253905122d30c7 Content-Type: text/plain; charset=UTF-8 Are you frequently updating same rows ? What is the memtable flush size ? can you post the table create query here in please. On Thu, Mar 26, 2015 at 1:21 PM, Dave Galbraith wrote: > Hey! So I'm running Cassandra 2.1.2 and using the > SizeTieredCompactionStrategy. I'm doing about 3k writes/sec on a single > node. My read performance is terrible, all my queries just time out. So I > do nodetool cfstats: > > Read Count: 42071 > Read Latency: 67.47804242827601 ms. > Write Count: 131964300 > Write Latency: 0.011721604274792501 ms. > Pending Flushes: 0 > Table: metrics16513 > SSTable count: 641 > Space used (live): 6366740812 > Space used (total): 6366740812 > Space used by snapshots (total): 0 > SSTable Compression Ratio: 0.25272488401992765 > Memtable cell count: 0 > Memtable data size: 0 > Memtable switch count: 1016 > Local read count: 42071 > Local read latency: 67.479 ms > Local write count: 131964300 > Local write latency: 0.012 ms > Pending flushes: 0 > Bloom filter false positives: 994 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 37840376 > Compacted partition minimum bytes: 104 > Compacted partition maximum bytes: 24601 > Compacted partition mean bytes: 255 > Average live cells per slice (last five minutes): > 111.67243951154147 > Maximum live cells per slice (last five minutes): 1588.0 > Average tombstones per slice (last five minutes): 0.0 > Maximum tombstones per slice (last five minutes): 0.0 > > and nodetool cfhistograms: > > Percentile SSTables Write Latency Read Latency Partition > Size Cell Count > (micros) (micros) > (bytes) > 50% 46.00 6.99 154844.95 > 149 1 > 75% 430.00 8.53 3518837.53 > 179 1 > 95% 430.00 11.32 7252897.25 > 215 2 > 98% 430.00 15.54 22103886.34 > 215 3 > 99% 430.00 29.86 22290608.19 > 1597 50 > Min 0.00 1.66 26.91 > 104 0 > Max 430.00 269795.38 27311364.89 > 24601 924 > > Gross!! There are 641 SSTables in there, and all my reads are hitting > hundreds of them and timing out. How could this possibly have happened, and > what can I do about it? Nodetool compactionstats says pending tasks: 0, by > the way. Thanks! > --047d7b8743f84d253905122d30c7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+PGRpdj5BcmUgeW91IGZyZXF1ZW50bHkgdXBkYXRpbmcgc2FtZSByb3dz ID8gV2hhdCBpcyB0aGUgbWVtdGFibGUgZmx1c2ggc2l6ZSA/IGNhbiB5b3UgcG9zdCB0aGUgdGFi bGUgY3JlYXRlIHF1ZXJ5IGhlcmUgaW4gcGxlYXNlLjxicj48L2Rpdj48L2Rpdj48ZGl2IGNsYXNz PSJnbWFpbF9leHRyYSI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiBUaHUsIE1hciAy NiwgMjAxNSBhdCAxOjIxIFBNLCBEYXZlIEdhbGJyYWl0aCA8c3BhbiBkaXI9Imx0ciI+Jmx0Ozxh IGhyZWY9Im1haWx0bzpkYXZpZDkyZ2FsYnJhaXRoQGdtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsi PmRhdmlkOTJnYWxicmFpdGhAZ21haWwuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48Ymxv Y2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3Jk ZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48 ZGl2PjxkaXY+SGV5ISBTbyBJJiMzOTttIHJ1bm5pbmcgQ2Fzc2FuZHJhIDIuMS4yIGFuZCB1c2lu ZyB0aGUgU2l6ZVRpZXJlZENvbXBhY3Rpb25TdHJhdGVneS4gSSYjMzk7bSBkb2luZyBhYm91dCAz ayB3cml0ZXMvc2VjIG9uIGEgc2luZ2xlIG5vZGUuIE15IHJlYWQgcGVyZm9ybWFuY2UgaXMgdGVy cmlibGUsIGFsbCBteSBxdWVyaWVzIGp1c3QgdGltZSBvdXQuIFNvIEkgZG8gbm9kZXRvb2wgY2Zz dGF0czo8YnI+PGJyPsKgwqDCoCBSZWFkIENvdW50OiA0MjA3MTxicj7CoMKgwqAgUmVhZCBMYXRl bmN5OiA2Ny40NzgwNDI0MjgyNzYwMSBtcy48YnI+wqDCoMKgIFdyaXRlIENvdW50OiAxMzE5NjQz MDA8YnI+wqDCoMKgIFdyaXRlIExhdGVuY3k6IDAuMDExNzIxNjA0Mjc0NzkyNTAxIG1zLjxicj7C oMKgwqAgUGVuZGluZyBGbHVzaGVzOiAwPGJyPsKgwqDCoCDCoMKgwqAgVGFibGU6IG1ldHJpY3Mx NjUxMzxicj7CoMKgwqAgwqDCoMKgIFNTVGFibGUgY291bnQ6IDY0MTxicj7CoMKgwqAgwqDCoMKg IFNwYWNlIHVzZWQgKGxpdmUpOiA2MzY2NzQwODEyPGJyPsKgwqDCoCDCoMKgwqAgU3BhY2UgdXNl ZCAodG90YWwpOiA2MzY2NzQwODEyPGJyPsKgwqDCoCDCoMKgwqAgU3BhY2UgdXNlZCBieSBzbmFw c2hvdHMgKHRvdGFsKTogMDxicj7CoMKgwqAgwqDCoMKgIFNTVGFibGUgQ29tcHJlc3Npb24gUmF0 aW86IDAuMjUyNzI0ODg0MDE5OTI3NjU8YnI+wqDCoMKgIMKgwqDCoCBNZW10YWJsZSBjZWxsIGNv dW50OiAwPGJyPsKgwqDCoCDCoMKgwqAgTWVtdGFibGUgZGF0YSBzaXplOiAwPGJyPsKgwqDCoCDC oMKgwqAgTWVtdGFibGUgc3dpdGNoIGNvdW50OiAxMDE2PGJyPsKgwqDCoCDCoMKgwqAgTG9jYWwg cmVhZCBjb3VudDogNDIwNzE8YnI+wqDCoMKgIMKgwqDCoCBMb2NhbCByZWFkIGxhdGVuY3k6IDY3 LjQ3OSBtczxicj7CoMKgwqAgwqDCoMKgIExvY2FsIHdyaXRlIGNvdW50OiAxMzE5NjQzMDA8YnI+ wqDCoMKgIMKgwqDCoCBMb2NhbCB3cml0ZSBsYXRlbmN5OiAwLjAxMiBtczxicj7CoMKgwqAgwqDC oMKgIFBlbmRpbmcgZmx1c2hlczogMDxicj7CoMKgwqAgwqDCoMKgIEJsb29tIGZpbHRlciBmYWxz ZSBwb3NpdGl2ZXM6IDk5NDxicj7CoMKgwqAgwqDCoMKgIEJsb29tIGZpbHRlciBmYWxzZSByYXRp bzogMC4wMDAwMDxicj7CoMKgwqAgwqDCoMKgIEJsb29tIGZpbHRlciBzcGFjZSB1c2VkOiAzNzg0 MDM3Njxicj7CoMKgwqAgwqDCoMKgIENvbXBhY3RlZCBwYXJ0aXRpb24gbWluaW11bSBieXRlczog MTA0PGJyPsKgwqDCoCDCoMKgwqAgQ29tcGFjdGVkIHBhcnRpdGlvbiBtYXhpbXVtIGJ5dGVzOiAy NDYwMTxicj7CoMKgwqAgwqDCoMKgIENvbXBhY3RlZCBwYXJ0aXRpb24gbWVhbiBieXRlczogMjU1 PGJyPsKgwqDCoCDCoMKgwqAgQXZlcmFnZSBsaXZlIGNlbGxzIHBlciBzbGljZSAobGFzdCBmaXZl IG1pbnV0ZXMpOiAxMTEuNjcyNDM5NTExNTQxNDc8YnI+wqDCoMKgIMKgwqDCoCBNYXhpbXVtIGxp dmUgY2VsbHMgcGVyIHNsaWNlIChsYXN0IGZpdmUgbWludXRlcyk6IDE1ODguMDxicj7CoMKgwqAg wqDCoMKgIEF2ZXJhZ2UgdG9tYnN0b25lcyBwZXIgc2xpY2UgKGxhc3QgZml2ZSBtaW51dGVzKTog MC4wPGJyPsKgwqDCoCDCoMKgwqAgTWF4aW11bSB0b21ic3RvbmVzIHBlciBzbGljZSAobGFzdCBm aXZlIG1pbnV0ZXMpOiAwLjA8YnI+PGJyPjwvZGl2PmFuZCBub2RldG9vbCBjZmhpc3RvZ3JhbXM6 PGJyPjxicj5QZXJjZW50aWxlwqAgU1NUYWJsZXPCoMKgwqDCoCBXcml0ZSBMYXRlbmN5wqDCoMKg wqDCoCBSZWFkIExhdGVuY3nCoMKgwqAgUGFydGl0aW9uIFNpemXCoMKgwqDCoMKgwqDCoCBDZWxs IENvdW50PGJyPsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqAgKG1pY3JvcynCoMKgwqDCoMKgwqDCoMKgwqAgKG1pY3JvcynCoMKgwqDCoMKg wqDCoMKgwqDCoCAoYnl0ZXMpwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCA8YnI+ NTAlwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCA0Ni4wMMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg IDYuOTnCoMKgwqDCoMKgwqDCoMKgIDE1NDg0NC45NcKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqAgMTQ5wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMTxicj43NSXCoMKgwqDCoMKg wqDCoMKgwqDCoCA0MzAuMDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCA4LjUzwqDCoMKgwqDC oMKgwqAgMzUxODgzNy41M8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMTc5wqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMTxicj45NSXCoMKgwqDCoMKgwqDCoMKgwqDCoCA0MzAu MDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMTEuMzLCoMKgwqDCoMKgwqDCoCA3MjUyODk3LjI1 wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAyMTXCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCAyPGJyPjk4JcKgwqDCoMKgwqDCoMKgwqDCoMKgIDQzMC4wMMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoCAxNS41NMKgwqDCoMKgwqDCoCAyMjEwMzg4Ni4zNMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqAgMjE1wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMzxicj45OSXC oMKgwqDCoMKgwqDCoMKgwqDCoCA0MzAuMDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMjkuODbC oMKgwqDCoMKgwqAgMjIyOTA2MDguMTnCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAxNTk3wqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDUwPGJyPk1pbsKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoCAwLjAwwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMS42NsKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCAyNi45McKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMTA0wqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqAgMDxicj5NYXjCoMKgwqDCoMKgwqDCoMKgwqDCoCA0MzAuMDDC oMKgwqDCoMKgwqDCoMKgIDI2OTc5NS4zOMKgwqDCoMKgwqDCoCAyNzMxMTM2NC44OcKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoCAyNDYwMcKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgOTI0PGJy Pjxicj48L2Rpdj5Hcm9zcyEhIFRoZXJlIGFyZSA2NDEgU1NUYWJsZXMgaW4gdGhlcmUsIGFuZCBh bGwgbXkgcmVhZHMgYXJlIGhpdHRpbmcgaHVuZHJlZHMgb2YgdGhlbSBhbmQgdGltaW5nIG91dC4g SG93IGNvdWxkIHRoaXMgcG9zc2libHkgaGF2ZSBoYXBwZW5lZCwgYW5kIHdoYXQgY2FuIEkgZG8g YWJvdXQgaXQ/IE5vZGV0b29sIGNvbXBhY3Rpb25zdGF0cyBzYXlzIHBlbmRpbmcgdGFza3M6IDAs IGJ5IHRoZSB3YXkuIFRoYW5rcyE8YnI+PC9kaXY+DQo8L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwv ZGl2Pg0K --047d7b8743f84d253905122d30c7--