cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Curdt <tcu...@apache.org>
Subject Re: Too many ParNew's
Date Wed, 09 Jun 2010 15:09:38 GMT
As promised on IIRC we also have collected some information as we are
seeing (probably) the same problem.

https://issues.apache.org/jira/browse/CASSANDRA-1177

On Wed, Jun 9, 2010 at 14:11, aaron morton <aaron@thelastpickle.com> wrote:
> May be related to CASSANDRA-1014
> https://issues.apache.org/jira/browse/CASSANDRA-1014
>
> And this discussion
> http://www.mail-archive.com/user@cassandra.apache.org/msg02923.html
>
> Aaron
>
> On 6 Jun 2010, at 08:14, Peter Schuller wrote:
>
>>>  INFO 17:54:18,567 GC for ParNew: 1522 ms, 69437384 reclaimed leaving 979692384
>>> used; max is 4424663040
>>>  INFO 17:54:22,567 GC for ParNew: 1989 ms, 69323576 reclaimed leaving 981439840
>>> used; max is 4424663040
>>>  INFO 17:54:26,187 GC for ParNew: 1337 ms, 69447160 reclaimed leaving 983903040
>>> used; max is 4424663040
>>>  INFO 17:54:29,489 GC for ParNew: 861 ms, 69507336 reclaimed leaving 986200344
u
>>> sed; max is 4424663040
>>>  INFO 17:54:32,238 GC for ParNew: 954 ms, 69245912 reclaimed leaving 987667920
u
>>> sed; max is 4424663040
>>>  INFO 17:54:36,053 GC for ParNew: 1242 ms, 69501648 reclaimed leaving 989972496
>>> used; max is 4424663040
>>>
>>> and this is affecting performance.
>>>
>>> This is on an 8Gb 4xCore machine with a 4Gb heap and mmapped i/o.
>>> KeyCaches are set to defaults and RowCaches are turned off.
>>
>> 1 second pauses sounds *really* excessive for young-generation GC:s,
>> especially with a current total heap size around 1 GB. Are you sure
>> the machine is not swapping? (Check with vmstat if it is actively
>> swapping in/out during this behavior.) This can happen even if you
>> have "free" memory, especially with mmap:ed memory competing competing
>> with the JVM heap. If you're swapping, and on Linux, you can try
>> decreasing the swappyness of the system by catting to
>> /proc/sys/vm/swappiness (IIRC; google it). This should prevent your
>> heap from being swapped out, at the expensive of less memory caching
>> the mmap:ed files.
>>
>> If it's a virtualized machine, make sure the host system is not
>> swapping or is otherwise severely overloaded...
>>
>> (Hypothesis based on gut feeling: If you're swapping I can imagine
>> old-space being swapped out. So the hot working set and allocations in
>> young gen might be fine, but copying surviving data from the young
>> generation into the old generation may trigger swapping, such that you
>> only exhibit excessive latencies during GC but mostly not otherwise.)
>>
>> --
>> / Peter Schuller
>
>

Mime
View raw message