cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shalom sagges <shalomsag...@gmail.com>
Subject Re: A Question About Hints
Date Mon, 04 Mar 2019 21:21:03 GMT
Everyone really should move off of the 2.x versions just like you are doing.

Tell me about it... But since there are a lot of groups involved, these
things take time unfortunately.


Thanks for your assistance Kenneth


On Mon, Mar 4, 2019 at 11:04 PM Kenneth Brotman
<kenbrotman@yahoo.com.invalid> wrote:

> Since you are in the process of upgrading, I’d do nothing on the settings
> right now.  But if you wanted to do something on the settings in the
> meantime, based on my read of the information available, I’d maybe double
> the default settings. The upgrade will help a lot of things as you know.
>
>
>
> Everyone really should move off of the 2.x versions just like you are
> doing.
>
>
>
> *From:* shalom sagges [mailto:shalomsagges@gmail.com]
> *Sent:* Monday, March 04, 2019 12:34 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> See my comments inline.
>
>
>
> Do the 8 nodes clusters have the problem too?
>
> Yes
>
>
>
> To the same extent?
>
> It depends on the throughput, but basically the smaller clusters get low
> throughput, so the problem is naturally smaller.
>
>
>
> Is it any cluster across multi-DC’s?
>
> Yes
>
>
>
> Do all the clusters use nodes with similar specs?
>
> All nodes have similar specs within a cluster but different specs on
> different clusters.
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
> Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the
> process of upgrading to 3.11.4
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> I guess you can say it about almost any parameter you change :)
>
>
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
> The entire article is very good, but I wonder if it's still valid since it
> was created around 4 years ago.
>
>
>
> Thanks!
>
>
>
>
>
>
>
>
>
> On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman <kenbrotman@yahoocom.invalid>
> wrote:
>
> Makes sense  If you have time and don’t mind, could you answer the
> following:
>
> Do the 8 nodes clusters have the problem too?
>
> To the same extent?
>
> Is it just the clusters with the large node count?
>
> Is it any cluster across multi-DC’s?
>
> Do all the clusters use nodes with similar specs?
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
>
>
>
>
>
>
> *From:* shalom sagges [mailto:shalomsagges@gmail.com]
> *Sent:* Monday, March 04, 2019 11:04 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> It varies...
>
> Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
>
> Both settings are on default.
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
> That's the plan, but I thought I might first get some valuable information
> from someone in the community that has already experienced in this type of
> change.
>
>
>
> Thanks!
>
>
>
> On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman
> <kenbrotman@yahoo.com.invalid> wrote:
>
> It sounds like your use case might be appropriate for tuning those two
> settings some.
>
>
>
> How many nodes are in the cluster?
>
> Are both settings definitely on the default values currently?
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
>
>
> Then of course share your results with us.
>
>
>
> *From:* shalom sagges [mailto:shalomsagges@gmail.com]
> *Sent:* Monday, March 04, 2019 9:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> Hi Kenneth,
>
>
>
> The concern is that in some cases, hints accumulate on nodes, and it takes
> a while until they are delivered (multi DCs).
>
> I see that whenever there are  a lot of hints in play,like after a rolling
> restart, the cluster works harder. That's why I want to decrease the hints
> delivery time.
>
> I didn't want to change the configuration blindly and thought the
> community might have some experience on this subject.
>
>
>
> I went over the cassandra.yaml file but didn't find any information on
> optimizing these attributes, just that the max_throttle is divided between
> nodes in the cluster and that I should increase the
> max_hints_delivery_threads because I have multi-dc deployments.
>
>
>
> # Maximum throttle in KBs per second, per delivery thread.  This will be
> # reduced proportionally to the number of nodes in the cluster.  (If there
> # are two nodes in the cluster, each delivery thread will use the maximum
> # rate; if there are three, each will throttle to half of the maximum,
> # since we expect two nodes to be delivering hints simultaneously)
> hinted_handoff_throttle_in_kb: 1024
>
> # Number of threads with which to deliver hints;
> # Consider increasing this number when you have multi-dc deployments, since
> # cross-dc handoff tends to be slower
> max_hints_delivery_threads: 2
>
>
>
>
>
> Thanks for your help!
>
>
>
>
>
> On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman
> <kenbrotman@yahoo.com.invalid> wrote:
>
> What is the concern?  Why are you looking there?  The casssandra.yml file
> has some notes about it.  Did you read them?
>
>
>
> *From:* shalom sagges [mailto:shalomsagges@gmail.com]
> *Sent:* Monday, March 04, 2019 7:22 AM
> *To:* user@cassandra.apache.org
> *Subject:* A Question About Hints
>
>
>
> Hi All,
>
>
>
> Does anyone know what is the most optimal hints configuration (multiple
> DCs) in terms of
>
> max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
>
> If it's different for various use cases, is there a rule of thumb I can
> work with?
>
>
>
> I found this post but it's quite old:
>
> http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/
>
>
>
> Thanks!
>
>

Mime
View raw message