Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 20ABE110E3 for ; Wed, 27 Aug 2014 22:03:03 +0000 (UTC) Received: (qmail 52008 invoked by uid 500); 27 Aug 2014 22:03:00 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 51970 invoked by uid 500); 27 Aug 2014 22:02:59 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 51955 invoked by uid 99); 27 Aug 2014 22:02:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 22:02:59 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.217.180] (HELO mail-lb0-f180.google.com) (209.85.217.180) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 22:02:34 +0000 Received: by mail-lb0-f180.google.com with SMTP id w7so70084lbi.25 for ; Wed, 27 Aug 2014 15:02:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=b2IC5gtvEW/98AKrsbTdCnVZ5OpJ93vzo7MxBHxEDck=; b=SNg5Q4lpOlCTfL67vex37PuFHkzqugcXLpVlDOyF9S0BQSqGkdcv4B5NDXXD+IG9AI WgjI6VrY6OKEbauAaSEXXyQT9s7MFvbMMHtp+vYZX/JqbsUNDA22n+psZRM+lFEMHYiH IsuHIhQm0QRMZmD3qTxjiOOhrKV1bjwQC9mqBw4fFa8mPDV4VMeHvvbYxt/xmcG37Dyy V7LI/8BeG5C7mmKa1zbaUinfhx39qBDsdeVmx5R4Shc1vvmcLyv3wfoZZQtWDZ1ztzv9 7fUXWQrbNg6GO1bbANn3XUreM+SlD/UkX6CmGpivM7YXJV22lPA0Wb2i1HHk6Xr6hJOm f8dw== X-Gm-Message-State: ALoCoQk5w6W9Q0/Twir4bh1GZSp6NUCTl6sfrPF8XO4UBuYvjRyeSOLsZvJPMpp8qdESnV2O0fPd MIME-Version: 1.0 X-Received: by 10.112.50.230 with SMTP id f6mr35458111lbo.56.1409176953247; Wed, 27 Aug 2014 15:02:33 -0700 (PDT) Received: by 10.112.141.133 with HTTP; Wed, 27 Aug 2014 15:02:33 -0700 (PDT) In-Reply-To: References: Date: Wed, 27 Aug 2014 17:02:33 -0500 Message-ID: Subject: Re: Too many SSTables after rebalancing cluster (LCS) From: Nate McCall To: Cassandra Users Content-Type: multipart/alternative; boundary=001a1133b54eb96f9e0501a3931e X-Virus-Checked: Checked by ClamAV on apache.org --001a1133b54eb96f9e0501a3931e Content-Type: text/plain; charset=UTF-8 Try turning down 'tombstone_threshold' to something like '0.05' from it's default of '0.2.' This will cause the SSTable to be considered for tombstone only compactions more frequently (if %5 of the columns are tombstones instead of 20%). For a bit more info, see: http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/compactSubprop.html On Tue, Aug 26, 2014 at 1:38 PM, Paulo Ricardo Motta Gomes < paulo.motta@chaordicsystems.com> wrote: > Hey folks, > > After adding more nodes and moving tokens of "old" nodes to rebalance the > ring, I noticed that the "old" nodes had significant more data then the > newly bootstrapped nodes, even after cleanup. > > I noticed that the old nodes had a much larger number of SSTables on LCS > CFs, and most of them located on the last level: > > Node N-1 (old node): [1, 10, 102/100, 173, 2403, 0, 0, 0, 0] (total:2695) > > *Node N (new node): [1, 10, 108/100, 214, 0, 0, 0, 0, 0] (total: 339)*Node > N+1 (old node): [1, 10, 87, 113, 1076, 0, 0, 0, 0] (total: 1287) > > Since these sstables have a lot of tombstones, and they're not updated > frequently, they remain in the last level forever, and are never cleaned. > > What is the solution here? The good old "change to STCS and then back to > LCS", or is there something less brute force? > > Environment: Cassandra 1.2.16 - non-vnondes > > Any help would be very much appreciated. > > Cheers, > > -- > *Paulo Motta* > > Chaordic | *Platform* > *www.chaordic.com.br * > +55 48 3232.3200 > -- ----------------- Nate McCall Austin, TX @zznate Co-Founder & Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com --001a1133b54eb96f9e0501a3931e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Try turning down 'tombstone_threshold' to somethin= g like '0.05' from it's default of '0.2.' This will cau= se the SSTable to be considered for tombstone only compactions more frequen= tly (if %5 of the columns are tombstones instead of 20%).=C2=A0

For a bit more info, see:


O= n Tue, Aug 26, 2014 at 1:38 PM, Paulo Ricardo Motta Gomes <p= aulo.motta@chaordicsystems.com> wrote:
Hey folks,

After adding more nodes and moving tokens of "old" nodes to re= balance the ring, I noticed that the "old" nodes had significant = more data then the newly bootstrapped nodes, even after cleanup.

I noticed that the old nodes had a much larger number o= f SSTables on LCS CFs, and most of them located on the last level:

Node N-1 (old node): [1, 10, 102/100, 173, 2403, 0, 0,= 0, 0] (total:2695)
Node N (new node):=C2=A0[1, 10, 108/100, 214, 0, 0, 0, 0, 0]= =C2=A0(total: 339)
Node N+1 (old node):=C2=A0[1, 10, 87, 113, 1076, = 0, 0, 0, 0] (total: 1287)

Since these sstables hav= e a lot of tombstones, and they're not updated frequently, they remain = in the last level forever, and are never cleaned.

What is the solution here? The good old "change to= STCS and then back to LCS", or is there something less brute force?

Environment: Cassandra 1.2.16 - non-vnondes

Any help would be very much appreciated.

Cheers,

--
Paulo Motta

Chaordic | Platform



--
-----------------
Nate McCall
Austin, TX
@zznate

Co-Fo= under & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpi= ckle.com
--001a1133b54eb96f9e0501a3931e--