cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kirill Bogdanov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6683) BADNESS_THRESHOLD does not working correctly with DynamicEndpointSnitch
Date Wed, 12 Feb 2014 08:48:21 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898928#comment-13898928
] 

Kirill Bogdanov commented on CASSANDRA-6683:
--------------------------------------------

Thank you for your answer.

I have came across this part of code in DES because I have observed suboptimal choice of nodes
in my configuration and started to investigate it. 

This is my config: 
* PropertyFileSnitch
* dynamic_snitch_badness_threshold 0.1, 
* 4 DCs, 
* keyspace with replication quota 1 for each DC. 
* Read repair and speculative_retry are disabled for my tables.
* Performing read operations with consistency TWO

I am observing that local DC that serves read request has about the same probability of asking
any of the 3 remote replicas to confirm consistency TWO regardless of their score (is that
correct?). 
Since all nodes are in different DCs {{subsnitch.sortByProximity}} places local node at the
start of the list (first) but does not sort other remote DCs.
After {{subsnitch.sortByProximity}} addresses list with scores may look something like that:
- DC1: 0.1 (first)
- DC2: 0.7 
- DC3: 0.2
- DC4: 0.2

Since we are not calling {{sortByProximityWithScore}} we returning this list to {{AbstractReadExecutor
getReadExecutor}} where {{consistencyLevel.filterForQuery}} (based on consistency TWO) picks
up first 2 addresses from the list. As a result we are sending read request to suboptimal
DC2.

By implementing my change ({{Math.abs()}}) I am seeing ~15% read throughput improvement in
my setup with cassandra stress tool.

Due to my limited knowledge of Cassandra internals I am probably wrong to blame DES and BADNESS_THRESHOLD,
but I would greatly appreciate if you could point out what is the correct behaviour in the
situation above and which module is responsible for sorting nodes by the scores.

Thank you.

> BADNESS_THRESHOLD does not working correctly with DynamicEndpointSnitch
> -----------------------------------------------------------------------
>
>                 Key: CASSANDRA-6683
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6683
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Linux 3.8.0-33-generic
>            Reporter: Kirill Bogdanov
>              Labels: snitch
>             Fix For: 2.0.6
>
>
> There is a problem in *DynamicEndpointSnitch.java* in sortByProximityWithBadness()
> Before calling sortByProximityWithScore we comparing each nodes score ratios to the badness
threshold.
> {code}
> if ((first - next) / first >  BADNESS_THRESHOLD)
>             {
>                 sortByProximityWithScore(address, addresses);
>                 return;
>             }
> {code}
> This is not always the correct comparison because *first* score can be less than *next*
 score and in that case we will compare a negative number with positive.
> The solution is to compute absolute value of the ratio:
> {code}
> if (Math.abs((first - next) / first) > BADNESS_THRESHOLD)
> {code}
> This issue causing an incorrect sorting of DCs based on their performance and affects
performance of the snitch.
> Thanks.
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message