I have been going through the mailing list and compiling suggestions to address the swapping due to mmap issue.
1) Use JNA (done but)
Are these steps also required:
- Start Cassandra with CAP_IPC_LOCK (or as "root"). (not done)
grep Unevictable /proc/meminfo
- set /proc/sys/vm/swappiness to 0 (not done)
- set initial -Xms max -Xmx heapsize to same value(done)
2) use mmap_index_only
3) Disable swap completely
4) use 'standard' disk access mode and try mmap only if more performance is needed
I was wondering if there an order of preference for these options based on experience of folks who have tried these suggestions before I trial and error my way to what works for us otherwise I am assuming the above order of preference.
Our cluster Info:
Linux 2.6.18-194.32.1.el5.centos.plus #1 SMP x86_64 x86_64 x86_64 GNU/Linux
Cassandra v 0.7.2 , 4 nodes (JNA enabled), 16GB RAM, 8GB heapsize(initial/max computed by cassandra-env.sh) , 5 GB swap
Swap space got swamped on all four nodes with a light write only load of around 5-10 MBps at load size of about 5-6 GB on each node. Heap usage was not high(around 50%). We are running a 4 node hadoop cluster as client(4 mappers). The nodes were responding well but the swapping is sending alerts and waking up our sys admin and we want to configure this correctly before it becomes a problem.
INFO [main] 2011-03-14 12:36:20,276 AbstractCassandraDaemon.java (line 97) Heap size: 8328839168/8329887744
INFO [main] 2011-03-14 12:36:24,141 CLibrary.java (line 106) JNA mlockall successful
INFO [main] 2011-03-14 12:36:24,151 DatabaseDescriptor.java (line 121) Loading settings from file:/usr/local/apache-cassandra-0.7.2/conf/cassandra.yaml
INFO [main] 2011-03-14 12:36:24,308 DatabaseDescriptor.java (line 181) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap