I have a 4 node cluster with lots JVM memory and lots of system memory that slows down when I'm doing lots of writes.

Running DataStax charts I see my read and write latency rise from 50-100 u-secs to 1500-4500 u-secs.  This is across a 12 hour data load during which time the applied load is high but fairly constant (500-700 writes/sec).

I'm trying to understand the slowdown: there is no memory pressure, I've run every option under nodetool to look for bottlenecks (tpstats, compactionStats, etc) and see none.  I'm running with keycache and have about 98% hits.

What can I check next?

Brian Tarbox