Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 68290 invoked from network); 28 Mar 2011 15:15:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 28 Mar 2011 15:15:44 -0000 Received: (qmail 22612 invoked by uid 500); 28 Mar 2011 15:15:42 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 22587 invoked by uid 500); 28 Mar 2011 15:15:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22579 invoked by uid 99); 28 Mar 2011 15:15:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 15:15:42 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of roberto.bentivoglio@gmail.com designates 209.85.213.44 as permitted sender) Received: from [209.85.213.44] (HELO mail-yw0-f44.google.com) (209.85.213.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 15:15:37 +0000 Received: by ywi6 with SMTP id 6so1369609ywi.31 for ; Mon, 28 Mar 2011 08:15:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=n2FLywE+l+X4RCNT3JCctiNP4KQ+fTJpxz5i4ZqzCHU=; b=Ko4WQK7CcbJ89IQiz3EGz55eQ4CLMM4/POUevVEjyTZmJMyfMrPr3/EF8FTzn/DPQr Vs5lAtC33EF5PsOUWe1lNG3x9bh8bVq72Lv38EdbmpGfLxKa3rfEUiLfpEA0l7N9B5s5 VX4CDAqSeOu3daDazcOVFrJJpBUOTsYXNCAaI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=UH/ZEhyDv/9EFofPoBJnnDZeSdF+c1JwdHF1LShEXk08Jgnmo5htgmVtpABArByVzW RofeNuM8gaD/4yt3p8nZNhVq3RWqHzhmlwhEkh45xNWXchpspptg/N5okW4CbXEsMbud uCt5eEsLv+5HkI5dtA7oi52q9IqhD2XsIZeeI= MIME-Version: 1.0 Received: by 10.90.38.7 with SMTP id l7mr3924142agl.133.1301325316182; Mon, 28 Mar 2011 08:15:16 -0700 (PDT) Received: by 10.90.80.1 with HTTP; Mon, 28 Mar 2011 08:15:16 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Mar 2011 17:15:16 +0200 Message-ID: Subject: Re: Problem about freeing space after a major compaction From: Roberto Bentivoglio To: user@cassandra.apache.org Cc: Ching-Cheng Chen Content-Type: multipart/alternative; boundary=00163628363a355596049f8c69f2 --00163628363a355596049f8c69f2 Content-Type: text/plain; charset=ISO-8859-1 Thanks you again, we're going to update our enviroment. Regards, Roberto On 28 March 2011 17:08, Ching-Cheng Chen wrote: > > AFAIK, setting gc_grace_period to 0 shouldn't cause this issue. In fact, > that what I'm using now in a single node environment like yours. > > However, I'm using 0.7.2 with some patches. If you are still using 0.7.0, > most likely you got hit with this bug. > You might want to patch it or upgrade to latest release. > > https://issues.apache.org/jira/browse/CASSANDRA-2059 > > Regards, > > > > Chen > > Senior Developer, EvidentSoftware(Leaders in Monitoring of NoSQL & JAVA ) > > http://www.evidentsoftware.com > > On Mon, Mar 28, 2011 at 11:04 AM, Roberto Bentivoglio < > roberto.bentivoglio@gmail.com> wrote: > >> Hi Chen, >> we've set the gc grace period of the column families to 0 as suggest in a >> single node enviroment. >> Can this setting cause the problem? I don't think so... >> >> Thanks, >> Roberto >> >> On 28 March 2011 16:54, Ching-Cheng Chen wrote: >> >>> tombstones removal also depends on your gc grace period setting. >>> >>> If you are pretty sure that you have proper gc grace period set and still >>> on 0.7.0, then probably related to this bug. >>> >>> https://issues.apache.org/jira/browse/CASSANDRA-2059 >>> >>> Regards, >>> >>> >>> >>> Chen >>> >>> Senior Developer, EvidentSoftware(Leaders in Monitoring of NoSQL & JAVA ) >>> >>> http://www.evidentsoftware.com >>> >>> On Mon, Mar 28, 2011 at 10:40 AM, Roberto Bentivoglio < >>> roberto.bentivoglio@gmail.com> wrote: >>> >>>> Hi all, >>>> we're working on a Cassandra 0.7.0 production enviroment with a store of >>>> data near to 500 GB. >>>> We need to periodically remove the tombstones from deleted/expired data >>>> performing a major compaction operation through nodetool. >>>> After invoking the compaction on a single column family we can see from >>>> JConsole that the LiveSSTableCount is going from 15 to 3 while the >>>> LiveDiskSpaceUsed is going from 90GB to 50GB. >>>> The problem now is that the space on the file system is been taken from >>>> Cassandra (I assumed from the old SSTable) and it isn't freed. We have tried >>>> to perform a full GC from the JConsole as described in >>>> http://wiki.apache.org/cassandra/MemtableSSTable without any success. >>>> The space is freed only after a database restart. >>>> >>>> How can we free this disk space without restart the db? >>>> >>>> Thanks you very much, >>>> Roberto Bentivoglio >>>> >>> >>> >> > --00163628363a355596049f8c69f2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks you again, we're going to update our enviroment.

<= div>Regards,
Roberto

On = 28 March 2011 17:08, Ching-Cheng Chen <cchen@evidentsoftware.com> wrot= e:

AFAIK, setting gc_grace_peri= od to 0 shouldn't cause this issue. =A0 In fact, that what I'm usin= g now in a single node environment like yours.

However, I'm using 0.7.2 with some patches. =A0 If you a= re still using 0.7.0, most likely you got hit with this bug.
You might want to patch it or upgrade to latest release.

Regards,

Chen

Senior Developer, EvidentSoftware(Leaders i= n Monitoring of NoSQL & JAVA )

http://www.evidentsoftware.com


O= n Mon, Mar 28, 2011 at 11:04 AM, Roberto Bentivoglio <= roberto.= bentivoglio@gmail.com> wrote:
Hi Chen,
we've set the gc= grace period of the column families to 0 as suggest in a single node envir= oment.
Can this setting cause the problem? I don't think so...
=
Thanks,
Roberto

= On 28 March 2011 16:54, Ching-Cheng Chen <cchen@evidentsoftware.co= m> wrote:
tombstones removal also depends on your gc grace period setting.

If you are pretty sure that you have proper gc grace period set an= d still on 0.7.0, then probably related to this bug.

=

Regards,

Chen

Senior Developer, EvidentSoftware(Leaders i= n Monitoring of NoSQL & JAVA )

http://www.evidentsoftware.com

=

On Mon, Mar 28, 2011 at 10:40 AM,= Roberto Bentivoglio <roberto.bentivoglio@gmail.com> wrote:
Hi all,
we're working on = a Cassandra 0.7.0 production enviroment with a store of data near to 500 GB= .
We need to periodically remove the tombstones from deleted/expired dat= a performing a major compaction operation through nodetool.
After invoking the compaction on a single column family we can see fro= m JConsole that the LiveSSTableCount is going from 15 to 3 while the LiveDi= skSpaceUsed is going from 90GB to 50GB.
The problem now is that t= he space on the file system is been taken from Cassandra (I assumed from th= e old SSTable) and it isn't freed. We have tried to perform a full GC f= rom the JConsole as described in http://wiki.apache.org/cassandra/Memta= bleSSTable without any success. The space is freed only after a databas= e restart.

How can we free this disk space without restart the db?=

Thanks you very much,
Roberto Bentivogl= io




--00163628363a355596049f8c69f2--