cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Brotman" <kenbrot...@yahoo.com.INVALID>
Subject RE: A Question About Hints
Date Mon, 04 Mar 2019 19:37:19 GMT
Makes sense.  If you have time and don’t mind, could you answer the following:

Do the 8 nodes clusters have the problem too? 

To the same extent?  

Is it just the clusters with the large node count? 

Is it any cluster across multi-DC’s?

Do all the clusters use nodes with similar specs?

 

The version of Cassandra you are on can make a difference.  What version are you on?

 

Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: https://www.youtube.com/watch?v=uN4FtAjYmLU
where he briefly mentions you can get into trouble if you go to fast or two slow?



BTW, I thought the comments at the end of the article you mentioned were really good.

 

 

 

From: shalom sagges [mailto:shalomsagges@gmail.com] 
Sent: Monday, March 04, 2019 11:04 AM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

It varies...

Some clusters have 48 nodes, others 24 nodes and some 8 nodes. 

Both settings are on default. 

 

I’d try making a single conservative change to one or the other, measure and reassess. 
Then do same to other setting.

That's the plan, but I thought I might first get some valuable information from someone in
the community that has already experienced in this type of change. 

 

Thanks!

 

On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman <kenbrotman@yahoo.com.invalid> wrote:

It sounds like your use case might be appropriate for tuning those two settings some. 

 

How many nodes are in the cluster?

Are both settings definitely on the default values currently?

 

I’d try making a single conservative change to one or the other, measure and reassess. 
Then do same to other setting.

 

Then of course share your results with us.

 

From: shalom sagges [mailto:shalomsagges@gmail.com] 
Sent: Monday, March 04, 2019 9:54 AM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

Hi Kenneth, 

 

The concern is that in some cases, hints accumulate on nodes, and it takes a while until they
are delivered (multi DCs). 

I see that whenever there are  a lot of hints in play,like after a rolling restart, the cluster
works harder. That's why I want to decrease the hints delivery time. 

I didn't want to change the configuration blindly and thought the community might have some
experience on this subject. 

 

I went over the cassandra.yaml file but didn't find any information on optimizing these attributes,
just that the max_throttle is divided between nodes in the cluster and that I should increase
the max_hints_delivery_threads because I have multi-dc deployments.   

 

# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously)
hinted_handoff_throttle_in_kb: 1024

# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2

 

 

Thanks for your help!

 

 

On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman <kenbrotman@yahoo.com.invalid> wrote:

What is the concern?  Why are you looking there?  The casssandra.yml file has some notes about
it.  Did you read them?

 

From: shalom sagges [mailto:shalomsagges@gmail.com] 
Sent: Monday, March 04, 2019 7:22 AM
To: user@cassandra.apache.org
Subject: A Question About Hints

 

Hi All,

 

Does anyone know what is the most optimal hints configuration (multiple DCs) in terms of 

max_hints_delivery_threads and hinted_handoff_throttle_in_kb? 

If it's different for various use cases, is there a rule of thumb I can work with?

 

I found this post but it's quite old:

http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/

 

Thanks!


Mime
View raw message