incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Katkov <ikat...@gmail.com>
Subject Storage proxy write latency is too high
Date Thu, 01 Oct 2009 22:51:08 GMT
Hi,

I have the following puzzle:
Storage proxy write latency ~235ms
CF write latency <1 ms

I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly distributed.
The client connects to a node and inserts a key with ConsistencyLevel.ONE
If it happen to be a local write operation is fast, same speed as in one
node setup. JMX shows write latency <1 ms
If it happens to be a remote insert StorageProxy sends it to a proper node.
This operation is slow. JMX shows write latency ~ 235ms.
In the same time, on remote node JMX shows same <1ms write latency. So it's
not remote node being sluggish, it's something else.
There are no pending tasks on remote node - JMX counters are always zero,
network is 1Gb, idle. So I can't blame it.


I profiled Cassandra server in JProfiler, could not find a thing. All this
extra time is spent inside QuorumResponseHandler waiting for the condition
to signal. Which should happen as soon as response is received.

There is one pooled TCP connection open to remote host. Hardly a bottleneck,
ThreadPoolExecutors looks OK.

Any ideas why write latency it is so high?

Mime
View raw message