First off, I'm curious what hardware (system specs) you're running this on?
Secondly, here are some observations:
* You're not running the newest JDK7, I can tell by your stack-size. Consider getting the newest.
* Cassandra 2.0.2 has a lot of improvements, consider upgrading. We noticed improved heap usage compared to 2.0.2
* Have you simply tried decreasing the size of your row cache? Tried 256MB?
* Do you have JNA installed? Otherwise, you're not getting off-heap usage for these caches which seems likely. Check your cassandra.log to verify JNA operation.
* Your NewGen is too small. See your heap peaks? This is because short-lived memory is being put into OldGen, which only gets cleaned up during fullGC. You should set your NewGen to about 25-30% of your total heapsize. Many objects are short-lived, and CMS GC is significantly more efficient if the shorter-lived objects never get promoted to OldGen; you'll get more concurrent, non-blocking GC. If you're not using JNA (per above) row-cache and key-cache is still on-heap, so you want your NewGen to be >= twice as large as the size of these combined caches. You should never so those crazy heap spikes, your caches are essentially overflowing into OldGen (with JNA).