BTW: Are you sure system doing wrong? System may save some pages to swap not removing them from RAM simply to have possibility to remove them later fast if needed.
We have 6 node cluster (cassandra 0.8.10). On one node i increase java heap size to 6GB, and now at this node begin grows swap, but system have about 3GB of free memory:
total used free shared buffers cached
Mem: 24733664 21702812 3030852 0 6792 13794724
-/+ buffers/cache: 7901296 16832368
Swap: 1998840 2352 1996488
And swap space slowly grows, but i misunderstand why?
PS: We have JNA mlock, and set vm.swappiness = 0
PS: OS ubuntu 10.0.4(2.6.32-40-generic)