cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thunder Stumpges <thunder.stump...@gmail.com>
Subject delay/stall processing reads
Date Thu, 02 Jan 2014 22:05:30 GMT
Hi all,

I am seeing a read operation delay in our small (3 node) cluster where I am
testing. The "normal" latency for these operations is < 2ms as recorded by
our load client. This holds easily beyond several hundred qps. However
there are times when all incoming queries (on a node-by-node basis) are
stalled anywhere from ~100-500ms, and then all "clear" and return at the
same time. This behavior is independent of the amount of load applied; Just
more queries get stalled at higher loads :). It seems like a "stall"
condition happens maybe every 30 seconds or so.

All of our column families are using replication factor 3 on 3 nodes, and
read consistency level ONE. The column family used for the test is not used
elsewhere, and is essentially empty.

We have verified at the network level that the request is "received" by the
machine, however the TRACE in cassandra does not even start for several
hundred milliseconds. All cassandra trace events happen very fast (< 2ms).
Once the query is complete, the response is sent promptly.

Here is a sample:

3 requests started at varying times, all returning at the same time (log
from load client)
2014-01-02 20:32:12,752 [14]  Start 12:32:12.612 End 12:32:12.752 Duration:
140.4811ms
2014-01-02 20:32:12,752 [12]  Start 12:32:12.456 End 12:32:12.752 Duration:
296.5098ms
2014-01-02 20:32:12,752 [7]    Start 12:32:12.316 End 12:32:12.752
Duration: 436.93ms

These three requests each hit the same server which was stalled in starting
those requests. All TRACE events in cassandra have the same timestamp, and
align with the END timestamp above.

Could this be garbage collection? It seems odd that the "delay" never seems
to happen in the middle of a request, only before it begins (at least
according to the trace events) I'm not sure how to proceed in
troubleshooting this.

Any additional information needed or suggestions on how to troubleshoot?
Many thanks in advance!
Thunder

P.S. Install information:
Datastax Community Cassandra v2.0.2 (3 nodes)
Datastax .net client (latest)
Cluster is production and running cluster-wide load ~500rps read, ~200rps
write

Mime
View raw message