cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kirill Bogdanov (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-6683) BADNESS_THRESHOLD does not working correctly with DynamicEndpointSnitch
Date Wed, 12 Feb 2014 08:48:21 GMT


Kirill Bogdanov commented on CASSANDRA-6683:

Thank you for your answer.

I have came across this part of code in DES because I have observed suboptimal choice of nodes
in my configuration and started to investigate it. 

This is my config: 
* PropertyFileSnitch
* dynamic_snitch_badness_threshold 0.1, 
* 4 DCs, 
* keyspace with replication quota 1 for each DC. 
* Read repair and speculative_retry are disabled for my tables.
* Performing read operations with consistency TWO

I am observing that local DC that serves read request has about the same probability of asking
any of the 3 remote replicas to confirm consistency TWO regardless of their score (is that
Since all nodes are in different DCs {{subsnitch.sortByProximity}} places local node at the
start of the list (first) but does not sort other remote DCs.
After {{subsnitch.sortByProximity}} addresses list with scores may look something like that:
- DC1: 0.1 (first)
- DC2: 0.7 
- DC3: 0.2
- DC4: 0.2

Since we are not calling {{sortByProximityWithScore}} we returning this list to {{AbstractReadExecutor
getReadExecutor}} where {{consistencyLevel.filterForQuery}} (based on consistency TWO) picks
up first 2 addresses from the list. As a result we are sending read request to suboptimal

By implementing my change ({{Math.abs()}}) I am seeing ~15% read throughput improvement in
my setup with cassandra stress tool.

Due to my limited knowledge of Cassandra internals I am probably wrong to blame DES and BADNESS_THRESHOLD,
but I would greatly appreciate if you could point out what is the correct behaviour in the
situation above and which module is responsible for sorting nodes by the scores.

Thank you.

> BADNESS_THRESHOLD does not working correctly with DynamicEndpointSnitch
> -----------------------------------------------------------------------
>                 Key: CASSANDRA-6683
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Linux 3.8.0-33-generic
>            Reporter: Kirill Bogdanov
>              Labels: snitch
>             Fix For: 2.0.6
> There is a problem in ** in sortByProximityWithBadness()
> Before calling sortByProximityWithScore we comparing each nodes score ratios to the badness
> {code}
> if ((first - next) / first >  BADNESS_THRESHOLD)
>             {
>                 sortByProximityWithScore(address, addresses);
>                 return;
>             }
> {code}
> This is not always the correct comparison because *first* score can be less than *next*
 score and in that case we will compare a negative number with positive.
> The solution is to compute absolute value of the ratio:
> {code}
> if (Math.abs((first - next) / first) > BADNESS_THRESHOLD)
> {code}
> This issue causing an incorrect sorting of DCs based on their performance and affects
performance of the snitch.
> Thanks.

This message was sent by Atlassian JIRA

View raw message