Reading this, I tried it again (this time on a freshly formated node due to hd failure).

Before crashing, my data dir was only 7.7M big. Using nmap_indexonly (mlockall was successfull) on a 64bit machine.

Anything else I could try to get this to work? Ps. All the other nodes (and this node) run fine (even with a few 100 Gigs of data loaded) when I remove the jna.jar, nodetool  -h localhost info showing more than 800M of free heap. (of a total of 3256M)

Thanks,
Thibaut








On Thu, May 5, 2011 at 2:49 PM, Chris Burroughs <chris.burroughs@gmail.com> wrote:
On 2011-05-05 06:30, Hannes Schmidt wrote:
> This was my first thought, too. We switched to mmap_index_only and
> didn't see any change in behavior. Looking at the smaps file attached
> to my original post, one can see that the mmapped index files take up
> only a minuscule part of RSS.

I have not looked into smaps before. But it actually seems odd that that
mmaped Index files are taking up so *little memory*.  Are they only a
few kb on disk?  Is this a snapshot taken shortly after the process
started or before the OOM killer is presumably about to come along.  How
long does it take to go from 1.1 G to 2.1 G resident?  Either way, it
would be worthwhile to set one node to standard io to make sure it's
really not mmap causing the problem.

Anyway, assuming it's not mmap, here are the other similar threads on
the topic.  Unfortunately none of them claim an obvious solution:

http://www.mail-archive.com/user@cassandra.apache.org/msg09279.html
http://www.mail-archive.com/user@cassandra.apache.org/msg08063.html
http://www.mail-archive.com/user@cassandra.apache.org/msg12036.html
http://mail.openjdk.java.net/pipermail/hotspot-dev/2011-April/004091.html