Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of hannes@eyealike.com designates
 209.85.213.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <B2C2565F-1A06-4BD0-BA82-29A84F9B767D@thelastpickle.com>
References: <BANLkTi=7aJrsqdenjDZ-S04zcuHPp==GoQ@mail.gmail.com>
	<loom.20110505T083328-985@post.gmane.org>
	<BANLkTikwbLLaFs1HsdZ_KVxSkoakkPpikw@mail.gmail.com>
	<B2C2565F-1A06-4BD0-BA82-29A84F9B767D@thelastpickle.com>
Date: Mon, 9 May 2011 10:17:12 -0700
Message-ID: <BANLkTikKrf7mJQ=Cj9C-Fbwzz986Lpas-w@mail.gmail.com>
Subject: Re: Native heap leaks?
From: Hannes Schmidt <hannes@eyealike.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, May 5, 2011 at 4:16 PM, aaron morton <aaron@thelastpickle.com> wrot=
e:
> Hannes,
> =A0 =A0 =A0 =A0To get a baseline of behaviour set disk_access to standard=
. You will probably want to keep it like that if you want better control ov=
er the memory on the box.

I'll do a test with standard and report back.

>
> =A0 =A0 =A0 =A0Also connect to the box with JConsole and look at the Perm=
Gen space used it is not included in the max heap space setting. You can al=
so check the heap usage there, running inside of 1G is very tricky.

PermGen is at 25M which doesn't explain the 700-1000M RSS overhead.
Nevertheless, I wasn't aware that PermGen isn't capped by -Xmx, so
thank you for pointing it out.

>
> =A0 =A0 =A0 =A0If you want to keep it inside of 2Gb trying setting the he=
ap max to 1.5G, use standard IO, disable caches, and use a low memtable thr=
eshold (it depends on how many CF's you have, try 32mb)

I'm not sure I follow. Besides the slowly increasing RSS size,
Cassandra works great for us with 1G. Don't the caches and memtables
live in the heap? I am not seeing any GC pressure at all so 1G should
be ok. Or do the caches and memtables have native components attached
to them like JNA-allocated memory or direct byte buffers?

>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5 May 2011, at 22:30, Hannes Schmidt wrote:
>
>> This was my first thought, too. We switched to mmap_index_only and
>> didn't see any change in behavior. Looking at the smaps file attached
>> to my original post, one can see that the mmapped index files take up
>> only a minuscule part of RSS.
>>
>> On Wed, May 4, 2011 at 11:37 PM, Oleg Anastasyev <oleganas@gmail.com> wr=
ote:
>>> Probably this is because of mmapped io access mode, which is enabled by=
 default
>>> in 64-bit VMs - RAM is occupied by data files.
>>> If you have such a tight memory reqs, you can turn on standard access m=
ode in
>>> storage-conf.xml, but dont expect it to work fast then:
>>> <!--
>>>
>>>
>>> =A0~ Access mode. =A0mmapped i/o is substantially faster, but only prac=
tical on
>>>
>>>
>>> =A0~ a 64bit machine (which notably does not include EC2 "small" instan=
ces)
>>>
>>>
>>> =A0~ or relatively small datasets. =A0"auto", the safe choice, will ena=
ble
>>>
>>>
>>> =A0~ mmapping on a 64bit JVM. =A0Other values are "mmap", "mmap_index_o=
nly"
>>>
>>>
>>> =A0~ (which may allow you to get part of the benefits of mmap on a 32bi=
t
>>>
>>>
>>> =A0~ machine by mmapping only index files) and "standard".
>>>
>>>
>>> =A0~ (The buffer size settings that follow only apply to standard,
>>>
>>>
>>> =A0~ non-mmapped i/o.)
>>>
>>>
>>> =A0-->
>>>
>>>
>>> =A0<DiskAccessMode>standard</DiskAccessMode>
>>>
>>>
>
>