cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: Too many ParNew's
Date Sat, 05 Jun 2010 20:14:33 GMT
>  INFO 17:54:18,567 GC for ParNew: 1522 ms, 69437384 reclaimed leaving 979692384
> used; max is 4424663040
>  INFO 17:54:22,567 GC for ParNew: 1989 ms, 69323576 reclaimed leaving 981439840
> used; max is 4424663040
>  INFO 17:54:26,187 GC for ParNew: 1337 ms, 69447160 reclaimed leaving 983903040
> used; max is 4424663040
>  INFO 17:54:29,489 GC for ParNew: 861 ms, 69507336 reclaimed leaving 986200344 u
> sed; max is 4424663040
>  INFO 17:54:32,238 GC for ParNew: 954 ms, 69245912 reclaimed leaving 987667920 u
> sed; max is 4424663040
>  INFO 17:54:36,053 GC for ParNew: 1242 ms, 69501648 reclaimed leaving 989972496
> used; max is 4424663040
>
> and this is affecting performance.
>
> This is on an 8Gb 4xCore machine with a 4Gb heap and mmapped i/o.
> KeyCaches are set to defaults and RowCaches are turned off.

1 second pauses sounds *really* excessive for young-generation GC:s,
especially with a current total heap size around 1 GB. Are you sure
the machine is not swapping? (Check with vmstat if it is actively
swapping in/out during this behavior.) This can happen even if you
have "free" memory, especially with mmap:ed memory competing competing
with the JVM heap. If you're swapping, and on Linux, you can try
decreasing the swappyness of the system by catting to
/proc/sys/vm/swappiness (IIRC; google it). This should prevent your
heap from being swapped out, at the expensive of less memory caching
the mmap:ed files.

If it's a virtualized machine, make sure the host system is not
swapping or is otherwise severely overloaded...

(Hypothesis based on gut feeling: If you're swapping I can imagine
old-space being swapped out. So the hot working set and allocations in
young gen might be fine, but copying surviving data from the young
generation into the old generation may trigger swapping, such that you
only exhibit excessive latencies during GC but mostly not otherwise.)

-- 
/ Peter Schuller

Mime
View raw message