From user-return-59287-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Fri Jan 5 08:54:36 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 94AB2180647 for ; Fri, 5 Jan 2018 08:54:36 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 82D4D160C27; Fri, 5 Jan 2018 07:54:36 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7A35F160C19 for ; Fri, 5 Jan 2018 08:54:35 +0100 (CET) Received: (qmail 21593 invoked by uid 500); 5 Jan 2018 07:54:33 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 21583 invoked by uid 99); 5 Jan 2018 07:54:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jan 2018 07:54:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1A31F180705 for ; Fri, 5 Jan 2018 07:54:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.175 X-Spam-Level: **** X-Spam-Status: No, score=4.175 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, INVALID_MSGID=1.167, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id QO3GNB0jPKem for ; Fri, 5 Jan 2018 07:54:28 +0000 (UTC) Received: from smtpbgau1.qq.com (smtpbgau1.qq.com [54.206.16.166]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 065555F1BA for ; Fri, 5 Jan 2018 07:54:26 +0000 (UTC) X-QQ-mid: bizesmtp9t1515138859t0i2nc3uk Received: from DESKTOP-TI0RKKB (unknown [115.195.35.55]) by esmtp4.qq.com (ESMTP) with SMTP id 0 for ; Fri, 05 Jan 2018 15:54:19 +0800 (CST) X-QQ-SSF: 01100000002000F0FF30B00A0000000 X-QQ-FEAT: E57Mx3KGKS+jvix/nSNVMruDE4RWJ3mFZeyGDPDcxQS5PM6XC2ZBY+gdXdfsO TrMEnpuI7PArGNL52gx7Hbemj8UL5JlZyjEa8Qb0IPKu+Zy60YasQdJ+qecPdz71r67mtZc kmSQrX5vy/iReIcE07Co8ronxhrs4z/gtNyY7htazVj6go/gbkwKYFu3BZavQ/ZiBtd+pjn xA1d+BoFd56ffntKhAoESjFaXudz2GIoEqcHvM7gHspOiDAjqmg8a+tcDr4i/6jyhUlLKID giVvGvd406BBOL4GZItt4Z8Z25945/7ZxVmJexiABN+n18XqQAe5iAyBGBlwd5BdDfFQ== X-QQ-GoodBg: 0 Date: Fri, 5 Jan 2018 15:54:16 +0800 From: "wxn002@zjqunshuo.com" To: user Subject: Re: Full repair caused disk space increase issue References: <5a4f1e97.03fe620a.e069b.75adSMTPIN_ADDED_BROKEN@mx.google.com>, X-Priority: 3 X-GUID: 26DB94E3-A3C1-43B9-ABD7-A7DCA602D5CB X-Has-Attach: no X-Mailer: Foxmail 7, 2, 8, 379[cn] Mime-Version: 1.0 Message-ID: <2018010515541579374965@zjqunshuo.com>+A523697841289B12 Content-Type: multipart/alternative; boundary="----=_001_NextPart043205002314_=----" X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:zjqunshuo.com:qybgforeign:qybgforeign4 X-QQ-Bgrelay: 1 ------=_001_NextPart043205002314_=---- Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 VGhhbmtzIEFsZXguIFNvbWUgbm9kZXMgaGF2ZSBmaW5pc2hlZCBhbnRpY29tcGFjdGlvbiBhbmQg ZGlzayBzcGFjZSBnb3QgcmVjbGFpbWVkIGFzIHlvdSBtZW50aW9uZWQuIA0KQlRXLCBhZnRlciBy ZWFkaW5nIHlvdXIgcG9zdChodHRwOi8vdGhlbGFzdHBpY2tsZS5jb20vYmxvZy8yMDE2LzEyLzA4 L1RXQ1MtcGFydDEuaHRtbCkgb24gVFdDUywgSSBkZWNpZGVkIHRvIHVzZSBUV0NTLCBhbmQgZG9p bmcgdGhlIGZ1bGwgcmVwYWlyIGlzIG9uZSBvZiB0aGUgcHJlcGFyYXRpb24gb2YgY2hhbmdpbmcg dG8gVFdDUy4NCiANCkZyb206IEFsZXhhbmRlciBEZWphbm92c2tpDQpEYXRlOiAyMDE4LTAxLTA1 IDE1OjE3DQpUbzogdXNlcg0KU3ViamVjdDogUmU6IEZ1bGwgcmVwYWlyIGNhdXNlZCBkaXNrIHNw YWNlIGluY3JlYXNlIGlzc3VlDQpIaSBTaW1vbiwNCg0Kc2luY2UgQ2Fzc2FuZHJhIDIuMiwgYW50 aWNvbXBhY3Rpb24gaXMgcGVyZm9ybWVkIGluIGFsbCB0eXBlcyBvZiByZXBhaXJzLCBleGNlcHQg c3VicmFuZ2UgcmVwYWlyLg0KR2l2ZW4gdGhhdCB5b3UgaGF2ZSBzb21lIHZlcnkgYmlnIFNTVGFi bGVzLCB0aGUgdGVtcG9yYXJ5IHNwYWNlIHVzZWQgYnkgYW50aWNvbXBhY3Rpb24gKHdoaWNoIGRv ZXMgdGhlIG9wcG9zaXRlIG9mIGNvbXBhY3Rpb24gOiByZWFkIG9uZSBzc3RhYmxlLCBvdXRwdXQg dHdvIHNzdGFibGVzKSB3aWxsIGltcGFjdCB5b3VyIGRpc2sgdXNhZ2Ugd2hpbGUgaXQncyBydW5u aW5nLiBJdCB3aWxsIHJlYWNoIGEgcGVhayB3aGVuIHRoZXkgYXJlIGNsb3NlIHRvIGNvbXBsZXRp b24uDQpUaGUgYW50aWNvbXBhY3Rpb24gdGhhdCBpcyByZXBvcnRlZCBieSBjb21wYWN0aW9uc3Rh dHMgaXMgY3VycmVudGx5IHVzaW5nIGFuIGV4dHJhIDE0N0dCKltjb21wcmVzc2lvbiByYXRpb10u IFNvIHdpdGggYSBjb21wcmVzc2lvbiByYXRpbyBvZiAwLjMgZm9yIGV4YW1wbGUsIHRoYXQgd291 bGQgYmUgNDRHQiB0aGF0IHdpbGwgZ2V0IHJlY2xhaW1lZCBzaG9ydGx5IGFmdGVyIHRoZSBhbnRp Y29tcGFjdGlvbiBpcyBvdmVyLg0KDQpZb3UgY2FuIGNoZWNrIHRoZSBjdXJyZW50IG92ZXJoZWFk IG9mIGNvbXBhY3Rpb24gYnkgbGlzdGluZyB0ZW1wb3Jhcnkgc3N0YWJsZXMgOiAqdG1wKkRhdGEu ZGINCg0KSXQncyBhbHNvIHBvc3NpYmxlIHRoYXQgeW91IGhhdmUgc29tZSBvdmVyc3RyZWFtaW5n IHRoYXQgb2NjdXJyZWQgZHVyaW5nIHlvdXIgcmVwYWlyLCB3aGljaCB3aWxsIGluY3JlYXNlIHRo ZSBzaXplIG9uIGRpc2sgdW50aWwgaXQgZ2V0cyBjb21wYWN0ZWQgYXdheSAob3ZlciB0aW1lKS4N CllvdSBzaG91bGQgYWxzbyBjaGVjayBpZiB5b3UgZG9uJ3QgaGF2ZSBzbmFwc2hvdHMgc3RpY2tp bmcgYXJvdW5kIGJ5IHJ1bm5pbmcgIm5vZGV0b29sIGxpc3RzbmFwc2hvdHMiLg0KDQpOb3csIHlv dSdyZSBtZW50aW9uaW5nIHRoYXQgeW91IHJhbiByZXBhaXIgdG8gZXZpY3QgdG9tYnN0b25lcy4g VGhpcyBpcyBub3Qgd2hhdCByZXBhaXIgZG9lcywgYW5kIHRvbWJzdG9uZXMgYXJlIGV2aWN0ZWQg dGhyb3VnaCBjb21wYWN0aW9uIHdoZW4gdGhleSBtZWV0IHRoZSByZXF1aXJlbWVudHMgKGdjX2dy YWNlX3NlY29uZHMgYW5kIGFsbCB0aGUgY2VsbHMgb2YgdGhlIHBhcnRpdGlvbiBpbnZvbHZlZCBp biB0aGUgc2FtZSBjb21wYWN0aW9uKS4NCklmIHlvdSB3YW50IHRvIG9wdGltaXplIHlvdXIgdG9t YnN0b25lIGV2aWN0aW9uLCBlc3BlY2lhbGx5IHdpdGggU1RDUywgSSBhZHZpc2UgdG8gdHVybiBv biB1bmNoZWNrZWRfdG9tYnN0b25lX2NvbXBhY3Rpb24sIHdoaWNoIHdpbGwgYWxsb3cgc2luZ2xl IHNzdGFibGVzIGNvbXBhY3Rpb25zIHRvIGJlIHRyaWdnZXJlZCBieSBDYXNzYW5kcmEgd2hlbiB0 aGVyZSBpcyBtb3JlIHRoYW4gMjAlIG9mIGVzdGltYXRlZCBkcm9wcGFibGUgdG9tYnN0b25lcyBp biBhbiBTU1RhYmxlLg0KWW91IGNhbiBjaGVjayB5b3VyIGN1cnJlbnQgZHJvcHBhYmxlIHRvbWJz dG9uZSByYXRpbyBieSBydW5uaW5nIHNzdGFibGVtZXRhZGF0YSBvbiBhbGwgeW91ciBzc3RhYmxl cy4NCkEgY29tbWFuZCBsaWtlIHRoZSBmb2xsb3dpbmcgc2hvdWxkIGRvIHRoZSB0cmljayAoaXQg d2lsbCBwcmludCBvdXQgbWluL21heCB0aW1lc3RhbXBzIHRvbykgOiANCg0KZm9yIGYgaW4gKkRh dGEuZGI7IGRvIG1ldGE9JChzdWRvIHNzdGFibGVtZXRhZGF0YSAkZik7IGVjaG8gLWUgIk1heDoi ICQoZGF0ZSAtLWRhdGU9QCQoZWNobyAiJG1ldGEiIHwgZ3JlcCBNYXhpbXVtXCB0aW1lIHwgY3V0 IC1kIiAiICAtZjN8IGN1dCAtYyAxLTEwKSAnKyVtLyVkLyVZJykgIk1pbjoiICQoZGF0ZSAtLWRh dGU9QCQoZWNobyAiJG1ldGEiIHwgZ3JlcCBNaW5pbXVtXCB0aW1lIHwgY3V0IC1kIiAiICAtZjN8 IGN1dCAtYyAxLTEwKSAnKyVtLyVkLyVZJykgJChlY2hvICIkbWV0YSIgfCBncmVwIGRyb3BwYWJs ZSkgJyBcdCAnICQobHMgLWxoICRmIHwgYXdrICd7cHJpbnQgJDUiICIkNiIgIiQ3IiAiJDgiICIk OX0nKTsgZG9uZSB8IHNvcnQNCg0KQ2hlY2sgaWYgdGhlIDIwJSB0aHJlc2hvbGQgaXMgaGlnaCBl bm91Z2ggYnkgdmVyaWZ5aW5nIHRoYXQgbmV3bHkgY3JlYXRlZCBTU1RhYmxlcyBkb24ndCBhbHJl YWR5IHJlYWNoIHRoYXQgbGV2ZWwsIGFuZCBhZGp1c3QgYWNjb3JkaW5nbHkgaWYgaXQncyB0aGUg Y2FzZSAoZm9yIGV4YW1wbGUgcmFpc2UgdGhlIHRocmVzaG9sZCB0byA1MCUpLg0KDQpUbyBhY3Rp dmF0ZSB0aGUgdG9tYnN0b25lIGNvbXBhY3Rpb25zLCB3aXRoIGEgNTAlIGRyb3BwYWJsZSB0b21i c3RvbmUgdGhyZXNob2xkLCBwZXJmb3JtIHRoZSBmb2xsb3dpbmcgc3RhdGVtZW50IG9uIHlvdXIg dGFibGUgOiANCg0KQUxURVIgVEFCTEUgY2FyZ3RzLmV2ZW50ZGF0YSBXSVRIIGNvbXBhY3Rpb24g PSB7J2NsYXNzJzonU2l6ZVRpZXJlZENvbXBhY3Rpb25TdHJhdGVneScsICd1bmNoZWNrZWRfdG9t YnN0b25lX2NvbXBhY3Rpb24nOid0cnVlJywgJ3RvbWJzdG9uZV90aHJlc2hvbGQnOicwLjUnfQ0K DQpQaWNraW5nIHRoZSByaWdodCB0aHJlc2hvbGQgaXMgdXAgdG8geW91Lg0KTm90ZSB0aGF0IHRv bWJzdG9uZSBjb21wYWN0aW9ucyBydW5uaW5nIG1vcmUgb2Z0ZW4gd2lsbCB1c2UgdGVtcG9yYXJ5 IHNwYWNlIGFzIHdlbGwsIGJ1dCB0aGV5IHNob3VsZCBoZWxwIGV2aWN0aW5nIHRvbWJzdG9uZXMg ZmFzdGVyIGlmIHRoZSBwYXJ0aXRpb25zIGFyZSBjb250YWluZWQgd2l0aGluIGEgc2luZ2xlIFNT VGFibGUuDQoNCklmIHlvdSBhcmUgZGVhbGluZyB3aXRoIFRUTGVkIGRhdGEgYW5kIHlvdXIgcGFy dGl0aW9ucyBzcHJlYWQgb3ZlciB0aW1lLCBJJ2Qgc3Ryb25nbHkgc3VnZ2VzdCBjb25zaWRlcmlu ZyBUV0NTIGluc3RlYWQgb2YgU1RDUyB3aGljaCBjYW4gcmVtb3ZlIGZ1bGx5IGV4cGlyZWQgU1NU YWJsZXMgbXVjaCBtb3JlIGVmZmljaWVudGx5Lg0KDQpDaGVlcnMsDQoNCg0KT24gRnJpLCBKYW4g NSwgMjAxOCBhdCA3OjQzIEFNIHd4bjAwMkB6anF1bnNodW8uY29tIDx3eG4wMDJAempxdW5zaHVv LmNvbT4gd3JvdGU6DQpIaSBBbGwsDQpJbiBvcmRlciB0byBldmljdCB0b21ic3RvbmVzLCBJIGlz c3VlZCBmdWxsIHJlcGFpciB3aXRoIHRoZSBjb21tYW5kICJub2RldG9vbCAtcHIgLWZ1bGwiLiBU aGVuIHRoZSBkYXRhIGxvYWQgc2l6ZSB3YXMgaW5kZWVkIGRlY3JlYXNlZCBieSAxMDBHIGZvciBl YWNoIG5vZGUgYnkgdXNpbmcgIm5vZGV0b29sIHN0YXR1cyIgdG8gY2hlY2suIEJ1dCB0aGUgYWN0 dWFsIGRpc2sgdXNhZ2UgaW5jcmVhc2VkIGJ5IDUwMEcgZm9yIGVhY2ggbm9kZS4gVGhlIHJlcGFp ciBpcyBzdGlsbCBvbmdvaW5nIGFuZCBsZWF2aW5nIGxlc3MgYW5kIGxlc3MgZGlzayBzcGFjZSBm b3IgbWUuDQoNCkZyb20gY29tcGFjdGlvbnN0YXRzLCBJIHNlZSAiQW50aWNvbXBhY3Rpb24gYWZ0 ZXIgcmVwYWlyIi4gQmFzZWQgb24gbXkgdW5kZXJzdGFuZGluZywgaXQgaXMgZm9yIGluY3JlbWVu dGFsIHJlcGFpciBieSBjaGFuZ2luZyBzc3RhYmxlIG1ldGFkYXRhIHRvIGluZGljYXRlIHdoaWNo IGZpbGUgaXMgcmVwYWlyZWQsIHNvIGluIG5leHQgcmVwYWlyIGl0IGlzIG5vdCBnb2luZyB0byBi ZSByZXBhaXJlZC4gQnV0IEknbSBkb2luZyBmdWxsIHJlcGFpciwgV2h5IEFudGljb21wYWN0aW9u Pw0KOWUwOWM0OTAtZjFiZS0xMWU3LWIyZWEtYjMwODVmODVjY2FlICAgQW50aWNvbXBhY3Rpb24g YWZ0ZXIgcmVwYWlyICAgICBjYXJndHMgICBldmVudGRhdGEgICAgMTQ3LjMgR0IgICAxNTguNTQg R0IgICBieXRlcyAgICAgOTIuOTElDQoNClRoZXJlIGFyZSBwYXJlIHNzdGFibGUgZmlsZXMuIEkg bWVhbiB0aGV5IGhhdmUgdGhlIHNhbWUgdGltZXN0YW1wIGFzIGJlbG93LiBJIGd1ZXNzIG9uZSBv ZiB0aGVtIG9yIGJvdGggb2YgdGhlbSBzaG91bGQgYmUgZGVsZXRlZCBhZnRlciBkdXJpbmcgcmVw YWlyLCBidXQgZm9yIHNvbWUgdW5rbm93biByZWFzb24sIHRoZSByZXBhaXIgcHJvY2VzcyBmYWls ZWQgdG8gZGVsZXRlIHRoZW0uIA0KLXJ3LXItLXItLSAxIHJvb3Qgcm9vdCAyMzdHIERlYyAzMSAx Mjo0OCBsYi0xMjM4MDAtYmlnLURhdGEuZGINCi1ydy1yLS1yLS0gMSByb290IHJvb3QgMjQzRyBE ZWMgMzEgMTI6NDggbGItMTIzODAxLWJpZy1EYXRhLmRiDQoNCkMqIHZlcnNpb24gaXMgMi4yLjgg d2l0aCBTVENTLiBBbnkgaWRlYXM/DQoNCkNoZWVycywNCi1TaW1vbg0KDQoNCi0tIA0KLS0tLS0t LS0tLS0tLS0tLS0NCkFsZXhhbmRlciBEZWphbm92c2tpDQpGcmFuY2UNCkBhbGV4YW5kZXJkZWph DQoNCkNvbnN1bHRhbnQNCkFwYWNoZSBDYXNzYW5kcmEgQ29uc3VsdGluZw0KaHR0cDovL3d3dy50 aGVsYXN0cGlja2xlLmNvbQ0K ------=_001_NextPart043205002314_=---- Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0A
Than= ks Alex. Some nodes have finished anticompaction and disk space got reclai= med as you mentioned. 
BTW, after reading = your post(http://thelastpickle.com/blog/2016/12/08/T= WCS-part1.html) on TWCS, I decided to use TWCS, an= d doing the full repair is one of the preparation of changing to TWCS.
=0A
 
Date: 2018-01-05 15:17
To:&nb= sp;user
Subj= ect: Re: Full repair caused disk space increase issue
=
Hi S= imon,

since Cassandra 2.2, anticompaction is performed = in all types of repairs, except subrange repair.
Given that you = have some very big SSTables, the temporary space used by anticompaction (w= hich does the opposite of compaction : read one sstable, output two sstabl= es) will impact your disk usage while it's running. It will reach a peak w= hen they are close to completion.
The anticompaction that is rep= orted by compactionstats is currently using an extra 147GB*[compression ra= tio]. So with a compression ratio of 0.3 for example, that would be 44GB t= hat will get reclaimed shortly after the anticompaction is over.

You can check the current overhead of compaction by listin= g temporary sstables : *tmp*Data.db

It's also pos= sible that you have some overstreaming that occurred during your repair, w= hich will increase the size on disk until it gets compacted away (over tim= e).
You should also check if you don't have snapshots sticking a= round by running "nodetool listsnapshots".

Now, y= ou're mentioning that you ran repair to evict tombstones. This is not what= repair does, and tombstones are evicted through compaction when they meet= the requirements (gc_grace_seconds and all the cells of the partition inv= olved in the same compaction).
If you want to optimize your tomb= stone eviction, especially with STCS, I advise to turn on unchecked_tombst= one_compaction, which will allow single sstables compactions to be trigger= ed by Cassandra when there is more than 20% of estimated droppable tombsto= nes in an SSTable.
You can check your current droppable tombston= e ratio by running sstablemetadata on all your sstables.
A comma= nd like the following should do the trick (it will print out min/max times= tamps too) : 

for f= in *Data.db; do meta=3D$(sudo sstablemetadata $f); echo -e "Max:" $(date = --date=3D@$(echo "$meta" | grep Maximum\ time | cut -d" "  -f3| cut -= c 1-10) '+%m/%d/%Y') "Min:" $(date --date=3D@$(echo "$meta" | grep Minimum= \ time | cut -d" "  -f3| cut -c 1-10) '+%m/%d/%Y') $(echo "$meta" | g= rep droppable) ' \t ' $(ls -lh $f | awk '{print $5" "$6" "$7" "$8" "$9}');= done | sort

Check if the 20% threshol= d is high enough by verifying that newly created SSTables don't already re= ach that level, and adjust accordingly if it's the case (for example raise= the threshold to 50%).

To activate the tombstone= compactions, with a 50% droppable tombstone threshold, perform the follow= ing statement on your table : 
ALTER TABLE cargts.eventdata W= ITH compaction =3D {'class':'SizeTieredCompactionStrategy', 'unchecked_tom= bstone_compaction':'true', 'tombstone_threshold':'0.5'}
<= br>
Picking the right threshold is up to you.
Note tha= t tombstone compactions running more often will use temporary space as wel= l, but they should help evicting tombstones faster if the partitions are c= ontained within a single SSTable.

If you are deal= ing with TTLed data and your partitions spread over time, I'd strongly sug= gest considering TWCS instead of STCS which can remove fully expired SSTab= les much more efficiently.

Cheers,

=



--
=0A
=0A ------=_001_NextPart043205002314_=------