From user-return-63353-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Mon Mar 4 18:27:58 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BA981180627 for ; Mon, 4 Mar 2019 19:27:57 +0100 (CET) Received: (qmail 30937 invoked by uid 500); 4 Mar 2019 18:27:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 30927 invoked by uid 99); 4 Mar 2019 18:27:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Mar 2019 18:27:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id C500FC215D for ; Mon, 4 Mar 2019 18:27:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.799 X-Spam-Level: ** X-Spam-Status: No, score=2.799 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Eo2vmsvIn6UJ for ; Mon, 4 Mar 2019 18:27:48 +0000 (UTC) Received: from sonic313-11.consmr.mail.ne1.yahoo.com (sonic313-11.consmr.mail.ne1.yahoo.com [66.163.185.34]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id C5F075FDAA for ; Mon, 4 Mar 2019 18:27:47 +0000 (UTC) X-YMail-OSG: 7DBVD00VM1k4Slvi4khoH36K9qQm6B.Cew9b4l5ggZJVATp9VaarGlUik3h9PHj ep9DQpRDEvbGsb9VUVXxsA4j6jychgKU4nhMDC9PO9vlPuxlWCSVpZ_NZKTNU3cLq_IcrE1PWvao YLPUrUuoPWJ47d_txNT_M4Dm.z6berS93sILHVlB8Uw7YpHm5TRk.1CkoXAESKW.tP1Ps_Tppg8G sCx5CxThYoEEiI9iRYWyTzd4cSYeRj0XMx73DcpAMsWwSSJJw20s2WB0Hiy2bHceloDZrSVFerPE 2scVFrk9tiDCirJSpZEM4aKx5mhSN5Oq4RGD6aAaZwjMz5brxFe43UmACH6NrSowKyKmticmrV0p N._GjJFiozEqxxM.yc7GPRU1e1Xe9j.WdC_sJsXX0f2iDrjaNboX6C4P7Sflu.NXe4Wo124jHHSE 5f24PO58hviUSUv6k2KNfAwrIAxG3hDbsQqZuStQ48ULn5KfFeliNqtkK3ELBhHqP_tUsSGLJ11l rARjgUz7RFWInM7BIImWT2O3ullWR8q9PmwR9IATW6Ol8XkqCt5bHOyXYSuK7RR0K2TxlyGOp_Ui j2KcdpBMwoI_zE0pKEQOw4zlTn3Ryu22UyReiNf6d1X9Gd2ArF57K8ErK5_DRqhXtCm4CTIBWeJV j3gKSS6.AGWzfAGQC_SN1xBpQG3CIBh.CrsI_CzJETxYfvrlsWk99NftLumMuau6eCwJNLGx7fg2 JE2P87XNjgCQqkuisuDWNg8FgUblToeDrlwh.iLwNR0ts0hlwYuVuXun4x8s2BQsMl7BYv4d._97 rFz.1ezPo4x9bbFZcRz6v8bgHHTewrmg_MMebvks5QBUYqDmJMFy5xRobA7uX_0DaOdB74j0V_Er hgGQdZry.0V3GavM1l.CUY2LcNSns.oRpE.AerZEZYFO7K2xhhgm8fZrd5Ko4bJEJGh5P7g4j05m 7_3TiMuLHYE.TySM7QKWIelFhPydyw.6vNl6hbJO9ArNimBDTC8Ku_TpHkXvoe_dNQXX2nNySayr YbBohdICqyGpWuwQctMh8pA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.ne1.yahoo.com with HTTP; Mon, 4 Mar 2019 18:27:41 +0000 Received: from c-24-10-87-155.hsd1.ca.comcast.net (EHLO i7) ([24.10.87.155]) by smtp419.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 9fcf1972f9814512ef431ad330acc9b3 for ; Mon, 04 Mar 2019 18:27:40 +0000 (UTC) From: "Kenneth Brotman" To: References: <000001d4d2a9$820245c0$8606d140$@yahoo.com> In-Reply-To: Subject: RE: A Question About Hints Date: Mon, 4 Mar 2019 10:27:37 -0800 Message-ID: <000001d4d2b7$f2236100$d66a2300$@yahoo.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0001_01D4D274.E4029200" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQIA2PmJ81aFc9ZxlCnWhRlYzxc/IQGVFU1CAOPGmM2lj/VaYA== Content-Language: en-us ------=_NextPart_000_0001_01D4D274.E4029200 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable It sounds like your use case might be appropriate for tuning those two = settings some.=20 =20 How many nodes are in the cluster? Are both settings definitely on the default values currently? =20 I=E2=80=99d try making a single conservative change to one or the other, = measure and reassess. Then do same to other setting. =20 Then of course share your results with us. =20 From: shalom sagges [mailto:shalomsagges@gmail.com]=20 Sent: Monday, March 04, 2019 9:54 AM To: user@cassandra.apache.org Subject: Re: A Question About Hints =20 Hi Kenneth,=20 =20 The concern is that in some cases, hints accumulate on nodes, and it = takes a while until they are delivered (multi DCs).=20 I see that whenever there are a lot of hints in play,like after a = rolling restart, the cluster works harder. That's why I want to decrease = the hints delivery time.=20 I didn't want to change the configuration blindly and thought the = community might have some experience on this subject.=20 =20 I went over the cassandra.yaml file but didn't find any information on = optimizing these attributes, just that the max_throttle is divided = between nodes in the cluster and that I should increase the = max_hints_delivery_threads because I have multi-dc deployments. =20 =20 # Maximum throttle in KBs per second, per delivery thread. This will be # reduced proportionally to the number of nodes in the cluster. (If = there # are two nodes in the cluster, each delivery thread will use the = maximum # rate; if there are three, each will throttle to half of the maximum, # since we expect two nodes to be delivering hints simultaneously.) hinted_handoff_throttle_in_kb: 1024 # Number of threads with which to deliver hints; # Consider increasing this number when you have multi-dc deployments, = since # cross-dc handoff tends to be slower max_hints_delivery_threads: 2 =20 =20 Thanks for your help! =20 =20 On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman = wrote: What is the concern? Why are you looking there? The casssandra.yml = file has some notes about it. Did you read them? =20 From: shalom sagges [mailto:shalomsagges@gmail.com]=20 Sent: Monday, March 04, 2019 7:22 AM To: user@cassandra.apache.org Subject: A Question About Hints =20 Hi All, =20 Does anyone know what is the most optimal hints configuration (multiple = DCs) in terms of=20 max_hints_delivery_threads and hinted_handoff_throttle_in_kb?=20 If it's different for various use cases, is there a rule of thumb I can = work with? =20 I found this post but it's quite old: http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/ =20 Thanks! ------=_NextPart_000_0001_01D4D274.E4029200 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

It sounds like your use case might be appropriate for tuning those = two settings some.

 

How many nodes are in the cluster?

Are both settings definitely on the default values = currently?

 

I=E2=80=99d try making a single conservative change to one or the = other, measure and reassess.=C2=A0 Then do same to other = setting.

 

Then of course share your results with us.

 

From:= = shalom sagges [mailto:shalomsagges@gmail.com]
Sent: Monday, = March 04, 2019 9:54 AM
To: = user@cassandra.apache.org
Subject: Re: A Question About = Hints

 

Hi Kenneth,

 

The concern is that in some cases, hints accumulate on = nodes, and it takes a while until they are delivered (multi DCs). =

I see that whenever there = are  a lot of hints in play,like after a rolling restart, the = cluster works harder. That's why I want to decrease the hints delivery = time.

I didn't want to = change the configuration blindly and thought the community might have = some experience on this subject.

 

I = went over the cassandra.yaml file but didn't find any information on = optimizing these attributes, just that the max_throttle is divided = between nodes in the cluster and that I should increase the = max_hints_delivery_threads because I have multi-dc = deployments.  

 

# = Maximum throttle in KBs per second, per delivery thread.  This will = be
# reduced proportionally to the number of nodes in the = cluster.  (If there
# are two nodes in the cluster, each = delivery thread will use the maximum
# rate; if there are three, each = will throttle to half of the maximum,
# since we expect two nodes to = be delivering hints simultaneously.)
hinted_handoff_throttle_in_kb: = 1024

# Number of threads with which to deliver hints;
# = Consider increasing this number when you have multi-dc deployments, = since
# cross-dc handoff tends to be = slower
max_hints_delivery_threads: 2

 

 

Thanks for your help!

 

 

On = Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman = <kenbrotman@yahoo.com.invalid> = wrote:

What is the concern?  Why are you looking there?  The = casssandra.yml file has some notes about it.  Did you read = them?

 

From:= = shalom sagges [mailto:shalomsagges@gmail.com]
Sent: Monday, = March 04, 2019 7:22 AM
To: user@cassandra.apache.org
Subject: A = Question About Hints

 <= /o:p>

Hi = All,

 <= /o:p>

Does anyone = know what is the most optimal hints configuration (multiple DCs) in = terms of

max_hints_de= livery_threads and hinted_handoff_throttle_in_kb? =

If it's = different for various use cases, is there a rule of thumb I can work = with?

 <= /o:p>

I found = this post but it's quite old:

 <= /o:p>

Thanks!=

------=_NextPart_000_0001_01D4D274.E4029200--