I have the following puzzle:
Storage proxy write latency ~235ms
CF write latency <1 ms

I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly distributed.
The client connects to a node and inserts a key with ConsistencyLevel.ONE
If it happen to be a local write operation is fast, same speed as in one node setup. JMX shows write latency <1 ms
If it happens to be a remote insert StorageProxy sends it to a proper node. This operation is slow. JMX shows write latency ~ 235ms.
In the same time, on remote node JMX shows same <1ms write latency. So it's not remote node being sluggish, it's something else.
There are no pending tasks on remote node - JMX counters are always zero, network is 1Gb, idle. So I can't blame it.

I profiled Cassandra server in JProfiler, could not find a thing. All this extra time is spent inside QuorumResponseHandler waiting for the condition to signal. Which should happen as soon as response is received.

There is one pooled TCP connection open to remote host. Hardly a bottleneck, ThreadPoolExecutors looks OK.

Any ideas why write latency it is so high?