cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Nodes dropping out of cluster due to GC
Date Thu, 03 Jun 2010 06:25:26 GMT
remember: you get concurrent mode failures, when the old gen fills up
with garbage before it can finish the CMS.  so adding capacity =
reducing load per machine is the easiest way to make this a non-issue.

On Wed, Jun 2, 2010 at 12:45 PM, Eric Halpern <eric@dnagamesinc.com> wrote:
>
>
> Ryan King wrote:
>>
>> Why run with so few nodes?
>>
>> -ryan
>>
>> On Tue, Jun 1, 2010 at 4:20 PM, Eric Halpern <eric@dnagamesinc.com> wrote:
>>>
>>> Hello,
>>>
>>> We're running a 4 node cluster on beefy EC2 virtual instances (8 core, 32
>>> GB) using EBS storage with 8 GB of heap allocated to the JVM.
>>>
>>> Every couple of hours, each of the nodes does a concurrent mark/sweep
>>> that
>>> takes around 30 seconds to complete.  During that GC, the node
>>> temporarily
>>> drops out of the cluster, usually for about 15 seconds.
>>>
>>> The frequency of the concurrent mark sweeps seems reasonable, but the
>>> fact
>>> that the node drops out of the cluster temporarily is a major problem
>>> since
>>> this has significant impact on the performance and stability of our
>>> service.
>>>
>>> Has anyone experienced this sort of problem?  It would be great to hear
>>> from
>>> anyone who has had experience with this sort of issue and/or suggestions
>>> for
>>> how to deal with it.
>>>
>>> Thanks, Eric
>>> --
>>
>>
>
> We wanted to start with a small number of nodes to test things out before
> going big.  Is there some reason that a small cluster would cause more
> problems in this regard.  The actual request load is actually pretty light
> for the cluster.
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Nodes-dropping-out-of-cluster-due-to-GC-tp5128481p5132279.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message