cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wayne <>
Subject Read Latency
Date Wed, 06 Oct 2010 13:55:31 GMT
I have been seeing some strange trends in read latency that I wanted to
throw out there to find some explanations. We are running .6.5 in a 10 node
cluster rf=3. We find that the read latency reported by the cfstats is
always about 1/4 of the actual time it takes to get the data back to the
python client. We are not using any higher level clients, and we usually are
doing Quorum reads (rf=3). If we ask for 1 copy (vs. 2 for Quorum) it is
around 2x the time reported in cfstats. This is true whether we have a .8 ms
read or a 5000 ms read. It is always around 4x the time for a Quorum read
and 2x the time for a single value read. This tells me that much of the time
waiting for a read has nothing to do with disk random read latency. This is
contrary to what is expected.

What is that extra time being used for? Waiting 2 ms for a read value to the
client when the value is retrieved in 1ms leaves 1ms that is unexplainable.
Is the node being requested by the client doing some "work" that equal the
time spent by the node actually serving up the data? Is this the thrift
server packaging up the response to the client?

Are reads really more CPU bound? We have lower end CPUs in our nodes, is
that part of the cause?

What is cfstats actually reporting? Is it not really reporting on* ALL* of
the time required to service a read? I assume is not reporting the time to
send the result to the requesting node.

How much of this time is network time? Would Infiniband or a lower latency
network architecture reduce any of these times? If we want to reduce a 2 ms
read to a 1ms read what will help us get there? We have cached keys which
then gives us a cfstats read latency < 1ms (~.85) but it still takes 2ms to
get to the client (single read).

Why does a quorum read double everything? It seems quorum reads are
serialized and not parallel. Is that true and if so why? Obviously it takes
more time to get 2 values and compare then get one value but if that is
always 2x+ then the adjustable consistency of Cassandra comes at a very high

Any other suggestions for decreasing read latency? Faster disks don't seem
as useful as faster CPUs. We have worked hard to reduce the cfstats reported
read latency and have been successful. How can we reduce the time from there
to the requesting client? What is the anatomy of a read from client request
to result? Where does the time usually go and what can help speed each step
up? Caching is the obvious answer but assume we are already caching what can
be cached (keys).

Thanks in advance for any advice or explanations anyone might have.

View raw message