cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Doubleday <daniel.double...@gmx.net>
Subject Re: Cassandra memory problem
Date Mon, 04 Jul 2011 16:33:49 GMT
Hi Sebastian,

one question: do you use jna.jar and do you see JNA mlockall successful in your logs.
There's that wild theory here that our problem might be related to mlockall and no swap. 
Maybe the JVM does some realloc stuff and the pinned pages are not cleared ... 

but that's really only wild guessing.

Also you are saying that on your servers res mem is not > max heap and the java process
is not swapping?

Thanks,
Daniel

On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote:

> It was among one of the issues we had. One of our hosts was using OpenJDK and we've switched
it to Sun and this part of the issue stabilized. The other issues we had were Heap going through
the roof and then OOM under load.
> 
> 
> On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday <daniel.doubleday@gmx.net> wrote:
> Just to make sure: 
> You were seeing that res mem was more than twice of max java heap and that did change
after you tweaked GC settings?
> 
> Note that I am not having a heap / gc problem. The VM itself thinks everything is golden.
> 
> On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:
> 
>> We had an issue like that a short while ago here. This was mainly happening under
heavy load and we managed to stabilize it by tweaking the Young/Old space ratio of the JVM
and by also tweaking the tenuring thresholds/survivor ratios. What kind of load to you have
on your systems? Mostly reads, writes?
>> 
>> SC
>> 
>> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <daniel.doubleday@gmx.net>
wrote:
>> Hi all,
>> 
>> we have a mem problem with cassandra. res goes up without bounds (well until the
os kills the process because we dont have swap)
>> 
>> I found a thread that's about the same problem but on OpenJDK: 
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>> 
>> We are on Debian with Sun JDK.
>> 
>> Resident mem is 7.4G while heap is restricted to 3G.
>> 
>> Anyone else is seeing this with Sun JDK?
>> 
>> Cheers,
>> Daniel
>> 
>> :/home/dd# java -version
>> java version "1.6.0_24"
>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>> 
>> :/home/dd# ps aux |grep java
>> cass     28201  9.5 46.8 372659544 7707172 ?   SLl  May24 5656:21 /usr/bin/java -ea
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3000M -Xmx3000M -Xmn400M ...
>> 
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
                                                                                         
                                          
>> 28201 cass      20   0  355g 7.4g 1.4g S    8 46.9   5656:25 java
>> 
>> 
>> 
>> 
> 
> 


Mime
View raw message