In my test program when I increase load then I keep getting few "write timeout" from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out.
Here is my Cassandra details:
Ring of 3 nodes with RF=3
Node configuration: 24 core + 64GB RAM + 2TB
"write_request_timeout_in_ms: 5000", rest of Cassandra.yaml configuration is default
I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node.
My test program creates connection to all three Cassandra nodes and sends read+write request randomly.
Any idea what should I look for?