cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Read time get worse during dynamic snitch reset
Date Mon, 11 Apr 2011 21:26:03 GMT
The reset interval clears the latency tracked for each node so a bad node will be read from
again. The scores for each node are then updated every 100ms (default) using the last 100
responses from a node. 

How long does the bad performance last for?

What CL are you reading at ? At Quorum with RF 4 the read request will be sent to 3 nodes,
ordered by proximity and wellness according to the dynamic snitch. (for background recent
discussion on dynamic snitch http://www.mail-archive.com/user@cassandra.apache.org/msg12089.html)

You can take a look at the weights and timings used by the DynamicSnitch in JConsole under
o.a.c.db.DynamicSnitchEndpoint . Also at DEBUG log level you will be able to see which nodes
the request is sent to. 

My guess is the DynamicSnitch is doing the right thing and the slow down is a node with a
problem getting back into the list of nodes used for your read. It's then moved down the list
as it's bad performance is noticed.

Hope that helps
Aaron
 

On 12 Apr 2011, at 01:28, shimi wrote:

> I finally upgraded 0.6.x to 0.7.4.  The nodes are running with the new version for several
days across 2 data centers.
> I noticed that the read time in some of the nodes increase by x50-60 every ten minutes.
> There was no indication in the logs for something that happen at the same time. The only
thing that I know that is running every 10 minutes is the dynamic snitch reset.
> So I changed dynamic_snitch_reset_interval_in_ms to 20 minutes and now I have the problem
once in every 20 minutes.
> 
> I am running all nodes with:
> replica_placement_strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
>       strategy_options:
>         DC1 : 2
>         DC2 : 2
>       replication_factor: 4
> 
> (DC1 and DC2 are taken from the ips)
> Does anyone familiar with this kind of behavior?
> 
> Shimi
> 


Mime
View raw message