cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Samuelsson <samuelsson.j...@gmail.com>
Subject Re: Extremely long GC
Date Thu, 23 Jan 2014 07:12:34 GMT
Here is one example. 12GB data, no load besides OpsCenter and perhaps 1-2
requests per minute.
INFO [ScheduledTasks:1] 2013-12-29 01:03:25,381 GCInspector.java (line 119)
GC for ParNew: 426400 ms for 1 collections, 2253360864 used; max is
4114612224


2014/1/22 Yogi Nerella <ynerella999@gmail.com>

> Hi,
>
> Can you share the GC logs for the systems you are running problems into?
>
> Yogi
>
>
> On Wed, Jan 22, 2014 at 6:50 AM, Joel Samuelsson <
> samuelsson.joel@gmail.com> wrote:
>
>> Hello,
>>
>> We've been having problems with long GC pauses and can't seem to get rid
>> of them.
>>
>> Our latest test is on a clean machine with Ubuntu 12.04 LTS, Java
>> 1.7.0_45 and JNA installed.
>> It is a single node cluster with most settings being default, the only
>> things changed are ip-addresses, cluster name and partitioner (to
>> RandomPartitioner).
>> We are running Cassandra 2.0.4.
>> We are running on a virtual machine with Xen.
>> We have 16GB of ram and default memory settings for C* (i.e. heap size of
>> 4GB). CPU specified as 8 cores by our provider.
>>
>> Right now, we have no data on the machine and no requests to it at all.
>> Still we get ParNew GCs like the following:
>> INFO [ScheduledTasks:1] 2014-01-18 10:54:42,286 GCInspector.java (line
>> 116) GC for ParNew: 464 ms for 1 collections, 102838776 used; max is
>> 4106223616
>>
>> While this may not be extremely long, on other machines with the same
>> setup but some data (around 12GB) and around 10 read requests/s (i.e.
>> basically no load) we have seen ParNew GC for 20 minutes or more. During
>> this time, the machine goes down completely (I can't even ssh to it). The
>> requests are mostly from OpsCenter and the rows requested are not extremely
>> large (typically less than 1KB).
>>
>> We have tried a lot of different things to solve these issues since we've
>> been having them for a long time including:
>> - Upgrading Cassandra to new versions
>> - Upgrading Java to new versions
>> - Printing promotion failures in GC-log (no failures found!)
>> - Different sizes of heap and heap space for different GC spaces (Eden
>> etc.)
>> - Different versions of Ubuntu
>> - Running on Amazon EC2 instead of the provider we are using now (not
>> with Datastax AMI)
>>
>> Something that may be a clue is that when running the DataStax Community
>> AMI on Amazon we haven't seen the GC yet (it's been running for a week or
>> so). Just to be clear, another test on Amazon EC2 mentioned above (without
>> the Datastax AMI) shows the GC freezes.
>>
>> If any other information is needed, just let me know.
>>
>> Best regards,
>> Joel Samuelsson
>>
>
>

Mime
View raw message