cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Too many ParNew's
Date Wed, 09 Jun 2010 12:11:39 GMT
May be related to CASSANDRA-1014
https://issues.apache.org/jira/browse/CASSANDRA-1014

And this discussion
http://www.mail-archive.com/user@cassandra.apache.org/msg02923.html

Aaron

On 6 Jun 2010, at 08:14, Peter Schuller wrote:

>>  INFO 17:54:18,567 GC for ParNew: 1522 ms, 69437384 reclaimed leaving 979692384
>> used; max is 4424663040
>>  INFO 17:54:22,567 GC for ParNew: 1989 ms, 69323576 reclaimed leaving 981439840
>> used; max is 4424663040
>>  INFO 17:54:26,187 GC for ParNew: 1337 ms, 69447160 reclaimed leaving 983903040
>> used; max is 4424663040
>>  INFO 17:54:29,489 GC for ParNew: 861 ms, 69507336 reclaimed leaving 986200344 u
>> sed; max is 4424663040
>>  INFO 17:54:32,238 GC for ParNew: 954 ms, 69245912 reclaimed leaving 987667920 u
>> sed; max is 4424663040
>>  INFO 17:54:36,053 GC for ParNew: 1242 ms, 69501648 reclaimed leaving 989972496
>> used; max is 4424663040
>> 
>> and this is affecting performance.
>> 
>> This is on an 8Gb 4xCore machine with a 4Gb heap and mmapped i/o.
>> KeyCaches are set to defaults and RowCaches are turned off.
> 
> 1 second pauses sounds *really* excessive for young-generation GC:s,
> especially with a current total heap size around 1 GB. Are you sure
> the machine is not swapping? (Check with vmstat if it is actively
> swapping in/out during this behavior.) This can happen even if you
> have "free" memory, especially with mmap:ed memory competing competing
> with the JVM heap. If you're swapping, and on Linux, you can try
> decreasing the swappyness of the system by catting to
> /proc/sys/vm/swappiness (IIRC; google it). This should prevent your
> heap from being swapped out, at the expensive of less memory caching
> the mmap:ed files.
> 
> If it's a virtualized machine, make sure the host system is not
> swapping or is otherwise severely overloaded...
> 
> (Hypothesis based on gut feeling: If you're swapping I can imagine
> old-space being swapped out. So the hot working set and allocations in
> young gen might be fine, but copying surviving data from the young
> generation into the old generation may trigger swapping, such that you
> only exhibit excessive latencies during GC but mostly not otherwise.)
> 
> -- 
> / Peter Schuller


Mime
View raw message