cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Burroughs (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-8035) 2.0.x repair causes large increasein client latency even for small datase
Date Wed, 01 Oct 2014 16:08:33 GMT
Chris Burroughs created CASSANDRA-8035:
------------------------------------------

             Summary: 2.0.x repair causes large increasein client latency even for small datase
                 Key: CASSANDRA-8035
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8035
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: c-2.0.10, 3 nodes per @ DCs.  Load < 50 MB
            Reporter: Chris Burroughs


Running repair causes a significnat increase in client latency even when the total amount
of data per node is very small.

Each node 900 req/s and during normal operations the 99p Client Request Lantecy is less than
4 ms and usually less than 1ms.  During repair the latency increases to within 4-10ms on all
nodes.  I am unable to find any resource based explantion for this.  Several graphs are attached
to summarize.

 * Client Request Latency goes up significantly.
 * Local keyspace read latency is flat.  I interpret this to mean that it's purly coordinator
overhead that's causing the slowdown.
 * Row cache hit rate is unaffected ( and is very high).  Between these two metrics I don't
think there is any doubt that virtually all reads are being satisfied in memory.
 * There is plenty of available cpu.  Aggregate cpu used (mostly nic) did go up during this.

Having more/larger keyspaces seems to make it worse.  Having two keyspaces on this cluster
(still with total size << RAM) caused larger increases in latency which would have made
for better graphs but it pushed the cluster well outsid of SLAs and we needed to move the
second keyspace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message