Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <4D7553DD.1080702@yellowseo.com>
Date: Mon, 07 Mar 2011 16:53:33 -0500
From: Paul Pak <ppak@yellowseo.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US;
 rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9
 ThunderBrowse/3.3.5
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: Nodes frozen in GC
References: <AANLkTi=BcxNdpjALJufGuJN0y9Ykwi-0Kbabotf_=hgr@mail.gmail.com>
 <CB9DCAEA-8DAF-48BE-B098-2A6E8169ACBE@thelastpickle.com>
 <AANLkTinZifrFYK=Fp8WNArAo7WaND5GY1cY6qwauUC-8@mail.gmail.com>
 <AANLkTikWBBTQpRcbQcCSM2GL+ebphnGeqtCFnfpMU4k8@mail.gmail.com>
In-Reply-To: <AANLkTikWBBTQpRcbQcCSM2GL+ebphnGeqtCFnfpMU4k8@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

So, are you saying this is normal and expected from Cassandra?  So,
under load, we can expect java garbage collection to stop the Cassandra
process on that server from time to time, essentially taking out the
node for short periods of time while it does garbage collection?

Also, why is there so much garbage collection to begin with?  Memcache
uses a slab allocator to reuse blocks to prevent allocation/deallocation
of blocks from consuming all the cpu time.  Are there any plans to reuse
blocks so the garbage collector doesn't have to work so hard?

Paul

On 3/7/2011 4:35 PM, Jonathan Ellis wrote:
> It sounds like you're complaining that the JVM sometimes does stop-the-world GC.
>
> You can mitigate this but not (for most workloads) eliminate it with
> GC option tuning.  That's simply the state of the art for Java garbage
> collection right now.
>
> On Sun, Mar 6, 2011 at 2:18 AM, ruslan usifov <ruslan.usifov@gmail.com> wrote:
>>
>> 2011/3/6 aaron morton <aaron@thelastpickle.com>
>>> Your node is under memory pressure, after the GC there is still 5.7GB in
>>> use. In fact it looks like memory usage went up during the GC process.
>>>
>>> Can you reduce the memtable size, caches or the number of CF's or increase
>>> the JVM size? Also is this happening under heavy load ?
>>>
>> Yes I do bulk load to cluster
>>
>>
>> I reduce memtable to 64MB sutuation became better, but it is all the same
>> very rare GC more than 15 Sec happens. Can reduce
>> flush_largest_memtables_at helps? O may be some GC options?
>>
>
>