I faced a similar issue as described there : http://comments.gmane.org/gmane.comp.db.cassandra.user/11184.

I was running Cassandra 1.0.0 with a 3 node cluster on 3 t1.micro from Amazon EC2.

I have no error in cassandra logs, but an OOM in /var/log/kern.log which put one of my nodes down.

After some search I have seen two possible reason for it :

1 - I may be keeping too much keys cached (200 000 which is default). I read that now this cache is out of the heap, my heap is about 300 M, the HEAP_NEWSIZE is about 75M, the system use about 150M all this out of 600M

What happens when I try to cache a row key and my memory is already full ? How to choose the number of keys cached ? I saw that it is hard to calculate the amount of memory used by this cache. Should I start by caching 2,000 rows per CF and update my schema increasing progresively the amount of key cached if I see that it works fine ?

2 - I read a lot about mmap without understanding clearly if I should keep it "auto" or "mmap_index_only". Anyways, I didn't find the "disk_access_mode" option in my cassandra.yaml, which let me think that it was removed and I have no other choice to keep "auto" value. 

Thank you,