cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogi Nerella <ynerella...@gmail.com>
Subject Re: Extremely long GC
Date Thu, 23 Jan 2014 23:08:34 GMT
Hi Joel,

One simple above log record will not help.   Please provide the entire
command line the process is started with including JVM options, the log
file showing all GC messages..
Is it possible for you to collect VerboseGC, which would print the before
and after statistics of the memory.

Version of java, version of Cassandra, availability of swap space, disk
space etc.,
CPU usage around the timeframe the garbage collection has happened.

What is the java -Xms (minimum memory) setting, try reducing it and see if
it helps.


Yogi




On Wed, Jan 22, 2014 at 11:12 PM, Joel Samuelsson <samuelsson.joel@gmail.com
> wrote:

> Here is one example. 12GB data, no load besides OpsCenter and perhaps 1-2
> requests per minute.
> INFO [ScheduledTasks:1] 2013-12-29 01:03:25,381 GCInspector.java (line
> 119) GC for ParNew: 426400 ms for 1 collections, 2253360864 used; max is
> 4114612224
>
>
> 2014/1/22 Yogi Nerella <ynerella999@gmail.com>
>
>> Hi,
>>
>> Can you share the GC logs for the systems you are running problems into?
>>
>> Yogi
>>
>>
>> On Wed, Jan 22, 2014 at 6:50 AM, Joel Samuelsson <
>> samuelsson.joel@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> We've been having problems with long GC pauses and can't seem to get rid
>>> of them.
>>>
>>> Our latest test is on a clean machine with Ubuntu 12.04 LTS, Java
>>> 1.7.0_45 and JNA installed.
>>> It is a single node cluster with most settings being default, the only
>>> things changed are ip-addresses, cluster name and partitioner (to
>>> RandomPartitioner).
>>> We are running Cassandra 2.0.4.
>>> We are running on a virtual machine with Xen.
>>> We have 16GB of ram and default memory settings for C* (i.e. heap size
>>> of 4GB). CPU specified as 8 cores by our provider.
>>>
>>> Right now, we have no data on the machine and no requests to it at all.
>>> Still we get ParNew GCs like the following:
>>> INFO [ScheduledTasks:1] 2014-01-18 10:54:42,286 GCInspector.java (line
>>> 116) GC for ParNew: 464 ms for 1 collections, 102838776 used; max is
>>> 4106223616
>>>
>>> While this may not be extremely long, on other machines with the same
>>> setup but some data (around 12GB) and around 10 read requests/s (i.e.
>>> basically no load) we have seen ParNew GC for 20 minutes or more. During
>>> this time, the machine goes down completely (I can't even ssh to it). The
>>> requests are mostly from OpsCenter and the rows requested are not extremely
>>> large (typically less than 1KB).
>>>
>>> We have tried a lot of different things to solve these issues since
>>> we've been having them for a long time including:
>>> - Upgrading Cassandra to new versions
>>> - Upgrading Java to new versions
>>> - Printing promotion failures in GC-log (no failures found!)
>>> - Different sizes of heap and heap space for different GC spaces (Eden
>>> etc.)
>>> - Different versions of Ubuntu
>>> - Running on Amazon EC2 instead of the provider we are using now (not
>>> with Datastax AMI)
>>>
>>> Something that may be a clue is that when running the DataStax Community
>>> AMI on Amazon we haven't seen the GC yet (it's been running for a week or
>>> so). Just to be clear, another test on Amazon EC2 mentioned above (without
>>> the Datastax AMI) shows the GC freezes.
>>>
>>> If any other information is needed, just let me know.
>>>
>>> Best regards,
>>> Joel Samuelsson
>>>
>>
>>
>

Mime
View raw message