Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of kamioshot@gmail.com designates
 209.85.214.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=PTk21t5Yw/KHrTuzOyO2Phr/TXjRbF34jRGMjmquY6H4CsijAdH0N9tL3Q2HTF7/8y
         FkqVxh36RE8rRxFXuzScmmmtrinevk+NWNksW+pzTRAZ+f9LIm8gU4SmOftN64eHLG5+
         lZZFJsdwVOTORUcfhZqTT5+SPuHsRPAD7QEaM=
MIME-Version: 1.0
In-Reply-To: <BANLkTinDAHrF8SQ2XsyCq5HzdG+U=s8aEw@mail.gmail.com>
References: <BANLkTikaaSvc6CDJWXJS9m6EJJLoFSaswA@mail.gmail.com>
	<2121177157.226992.1306450684707.JavaMail.root@mail-1.01.com>
	<BANLkTikGn=jP5_M=56atvYiae1putFnFJg@mail.gmail.com>
	<BANLkTinDAHrF8SQ2XsyCq5HzdG+U=s8aEw@mail.gmail.com>
Date: Wed, 15 Jun 2011 23:50:42 +0900
Message-ID: <BANLkTin02hek_Jw5fSzF9ixVSfEMDDzeHQ@mail.gmail.com>
Subject: Re: Forcing Cassandra to free up some space
From: Shotaro Kamio <kamioshot@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

We've encountered the situation that compacted sstable files aren't
deleted after node repair. Even when gc is triggered via jmx, it
sometimes leaves compacted files. In a case, a lot of files are left.
Some files stay more than 10 hours already. There is no guarantee that
gc will cleanup all compacted sstable files.

We have a great interest on the following ticket.
https://issues.apache.org/jira/browse/CASSANDRA-2521


Regards,
Shotaro


On Fri, May 27, 2011 at 11:27 AM, Jeffrey Kesselman <jeffpk@gmail.com> wrot=
e:
> Im also not sure that will guarantee all space is cleaned up. =A0It
> really depends on what you are doing inside Cassandra. =A0If you have
> your on garbage collect that is just in some way tied to the gc run,
> then it will run when =A0it runs.
>
> If otoh you are associating records in your storage with specific
> objects in memory and using one of the post-mortem hooks (finalize or
> PhantomReference) to tell you to clean up that particular record then
> its quite possible they wont all get cleaned up. =A0In general hotspot
> does not find and clean every candidate object on every GC run. =A0It
> starts with the easiest/fastest to find and then sees what more it
> thinks it needs to do to create enough memory for anticipated near
> future needs.
>
> On Thu, May 26, 2011 at 10:16 PM, Jonathan Ellis <jbellis@gmail.com> wrot=
e:
>> In summary, system.gc works fine unless you've deliberately done
>> something like setting the -XX:-DisableExplicitGC flag.
>>
>> On Thu, May 26, 2011 at 5:58 PM, Konstantin =A0Naryshkin
>> <konstantinn@a-bb.net> wrote:
>>> So, in summary, there is no way to predictably and efficiently tell Cas=
sandra to get rid of all of the extra space it is using on disk?
>>>
>>> ----- Original Message -----
>>> From: "Jeffrey Kesselman" <jeffpk@gmail.com>
>>> To: user@cassandra.apache.org
>>> Sent: Thursday, May 26, 2011 8:57:49 PM
>>> Subject: Re: Forcing Cassandra to free up some space
>>>
>>> Which JVM? =A0Which collector? =A0There have been and continue to be ma=
ny.
>>>
>>> Hotspot itself supports a number of different collectors with
>>> different behaviors. =A0 Many of them do not collect every candidate on
>>> every gc, but merely the easiest ones to find. =A0This is why depending
>>> on finalizers is a *bad* idea in java code. =A0They may well never get
>>> run. =A0(Finalizer is one of a few features the Sun Java team always
>>> regretted putting in Java to start with. =A0It has caused quite a few
>>> application problems over the years)
>>>
>>> The really important thing is that NONE of these behaviors of the
>>> colelctors are guaranteed by specification not to change from version
>>> to version. =A0Basing your code on non-specified behaviors is a good wa=
y
>>> to hit mysterious failures on updates.
>>>
>>> For instance, in the mid 90s, IBM had a mode of their Vm called
>>> "infinite heap." =A0it *never* garbage collected, even if you called
>>> System.gc. =A0Instead it just threw away address space and counted on
>>> the total memory needs for the life of the program being less then the
>>> total addressable space of the processor.
>>>
>>> It was *very* fast for certain kinds of applications.
>>>
>>> Far from being pedantic, not depending on undocumented behavior is
>>> simply good engineering.
>>>
>>>
>>> On Thu, May 26, 2011 at 4:51 PM, Jonathan Ellis <jbellis@gmail.com> wro=
te:
>>>> I've read the relevant source. While you're pedantically correct re
>>>> the spec, you're wrong as to what the JVM actually does.
>>>>
>>>> On Thu, May 26, 2011 at 3:14 PM, Jeffrey Kesselman <jeffpk@gmail.com> =
wrote:
>>>>> Some references...
>>>>>
>>>>> "An object enters an unreachable state when no more strong references
>>>>> to it exist. When an object is unreachable, it is a candidate for
>>>>> collection. Note the wording: Just because an object is a candidate
>>>>> for collection doesn't mean it will be immediately collected. The JVM
>>>>> is free to delay collection until there is an immediate need for the
>>>>> memory being consumed by the object."
>>>>>
>>>>> http://java.sun.com/docs/books/performance/1st_edition/html/JPAppGC.f=
m.html#998394
>>>>>
>>>>> and "Calling the gc method suggests that the Java Virtual Machine
>>>>> expend effort toward recycling unused objects"
>>>>>
>>>>> http://download.oracle.com/javase/6/docs/api/java/lang/System.html#gc=
()
>>>>>
>>>>> It goes on to say that the VM will make a "best effort", but "best
>>>>> effort" is *deliberately* left up to the definition of the gc
>>>>> implementor.
>>>>>
>>>>> I guess you missed the many lectures I have given on this subject ove=
r
>>>>> the years at Java One Conferences....
>>>>>
>>>>> On Thu, May 26, 2011 at 3:53 PM, Jonathan Ellis <jbellis@gmail.com> w=
rote:
>>>>>> It's a common misunderstanding that system.gc is only a suggestion; =
on
>>>>>> any VM you're likely to run Cassandra on, System.gc will actually
>>>>>> invoke a full collection.
>>>>>>
>>>>>> On Thu, May 26, 2011 at 2:18 PM, Jeffrey Kesselman <jeffpk@gmail.com=
> wrote:
>>>>>>> Actually this is no gaurantee.=A0=A0 Its a common misunderstanding =
that
>>>>>>> System.gc "forces" gc.=A0 It does not. It is a suggestion only. The=
 vm always
>>>>>>> has the option as to when and how much it gcs
>>>>>>>
>>>>>>> On May 26, 2011 2:51 PM, "Jonathan Ellis" <jbellis@gmail.com> wrote=
:
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jonathan Ellis
>>>>>> Project Chair, Apache Cassandra
>>>>>> co-founder of DataStax, the source for professional Cassandra suppor=
t
>>>>>> http://www.datastax.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> It's always darkest just before you are eaten by a grue.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jonathan Ellis
>>>> Project Chair, Apache Cassandra
>>>> co-founder of DataStax, the source for professional Cassandra support
>>>> http://www.datastax.com
>>>>
>>>
>>>
>>>
>>> --
>>> It's always darkest just before you are eaten by a grue.
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
>
> --
> It's always darkest just before you are eaten by a grue.
>


--=20
Shotaro Kamio