Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 9 Jun 2016 01:57:21 +0000 (UTC)
From: "Dikang Gu (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12703077.1395574963000.35158.1465437441114@Atlassian.JIRA>
In-Reply-To: <JIRA.12703077.1395574963000@Atlassian.JIRA>
References: <JIRA.12703077.1395574963000@Atlassian.JIRA> <JIRA.12703077.1395574963175@arcas>
Subject: [jira] [Commented] (CASSANDRA-6908) Dynamic endpoint snitch
 destabilizes cluster under heavy load
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 09 Jun 2016 01:57:23 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-6908?page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D=
15321773#comment-15321773 ]=20

Dikang Gu commented on CASSANDRA-6908:
--------------------------------------

We had a similar problem and we worked around it by setting the dynamic_sni=
tch_badness_threshold to be 50, which dropped the P99 latency by 10X.


> Dynamic endpoint snitch destabilizes cluster under heavy load
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-6908
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6908
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Configuration
>            Reporter: Bart=C5=82omiej Roma=C5=84ski
>            Assignee: Brandon Williams
>         Attachments: as-dynamic-snitch-disabled.png
>
>
> We observe that with dynamic snitch disabled our cluster is much more sta=
ble than with dynamic snitch enabled.
> We've got a 15 nodes cluster with pretty strong machines (2xE5-2620, 64 G=
B RAM, 2x480 GB SSD). We mostly do reads (about 300k/s).
> We use Astyanax on client side with TOKEN_AWARE option enabled. It automa=
tically direct read queries to one of the nodes responsible the given token=
.
> In that case with dynamic snitch disabled Cassandra always handles read l=
ocally. With dynamic snitch enabled Cassandra very often decides to proxy t=
he read to some other node. This causes much higher CPU usage and produces =
much more garbage what results in more often GC pauses (young generation fi=
lls up quicker). By "much higher" and "much more" I mean 1.5-2x.
> I'm aware that higher dynamic_snitch_badness_threshold value should solve=
 that issue. The default value is 0.1. I've looked at scores exposed in JMX=
 and the problem is that our values seemed to be completely random. They ar=
e between usually 0.5 and 2.0, but changes randomly every time I hit refres=
h.
> Of course, I can set dynamic_snitch_badness_threshold to 5.0 or something=
 like that, but the result will be similar to simply disabling the dynamic =
switch at all (that's what we done).
> I've tried to understand what's the logic behind these scores and I'm not=
 sure if I get the idea...
> It's a sum (without any multipliers) of two components:
> - ratio of recent given node latency to recent average node latency
> - something called 'severity', what, if I analyzed the code correctly, is=
 a result of BackgroundActivityMonitor.getIOWait() - it's a ratio of "iowai=
t" CPU time to the whole CPU time as reported in /proc/stats (the ratio is =
multiplied by 100)
> In our case the second value is something around 0-2% but varies quite he=
avily every second.
> What's the idea behind simply adding this two values without any multipli=
ers (e.g the second one is in percentage while the first one is not)? Are w=
e sure this is the best possible way of calculating the final score?
> Is there a way too force Cassandra to use (much) longer samples? In our c=
ase we probably need that to get stable values. The 'severity' is calculate=
d for each second. The mean latency is calculated based on some magic, hard=
coded values (ALPHA =3D 0.75, WINDOW_SIZE =3D 100).=20
> Am I right that there's no way to tune that without hacking the code?
> I'm aware that there's dynamic_snitch_update_interval_in_ms property in t=
he config file, but that only determines how often the scores are recalcula=
ted not how long samples are taken. Is that correct?
> To sum up, It would be really nice to have more control over dynamic snit=
ch behavior or at least have the official option to disable it described in=
 the default config file (it took me some time to discover that we can just=
 disable it instead of hacking with dynamic_snitch_badness_threshold=3D1000=
).
> Currently for some scenarios (like ours - optimized cluster, token aware =
client, heavy load) it causes more harm than good.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)