Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D0531D73B for ; Tue, 19 Mar 2013 07:15:14 +0000 (UTC) Received: (qmail 74783 invoked by uid 500); 19 Mar 2013 07:15:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 74689 invoked by uid 500); 19 Mar 2013 07:15:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74663 invoked by uid 99); 19 Mar 2013 07:15:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Mar 2013 07:15:11 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a59.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Mar 2013 07:15:07 +0000 Received: from homiemail-a59.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTP id 70383564057 for ; Tue, 19 Mar 2013 00:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=OQrwDQAtnYRf5Qgn48KuoBu6mh E=; b=Gy4CMi62JWMq0vZ8fVGwEYMn5QM1Kt4eMr/D7J60Te6WKV9i/vNDoYFj5I zEvx+/lvb2IZG8uFRGuEYLFoS552Ldz9YzosAJYARwqSX479YeqSvJKqtX2TB3kg c2lnNJdTogLcOTA+CTZcZ557RVKwZULd1pDK0jsYAJmI+vqCk= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a59.g.dreamhost.com (Postfix) with ESMTPSA id BB6DD564061 for ; Tue, 19 Mar 2013 00:14:44 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_A32C39D5-D83F-4C02-A106-15DECF133074" Message-Id: <7475747E-253A-4A04-B3E4-A163DC4425F8@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: hinted handoff disabling trade-offs Date: Tue, 19 Mar 2013 20:14:43 +1300 References: <9B6E672A-20AB-4F02-AA7D-103A563AD1A8@barracuda.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_A32C39D5-D83F-4C02-A106-15DECF133074 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > I think I understand what it means for > application-level data, but the part I'm not entirely sure about is > what it could mean for Cassandra internals. Internally it means the write will not be retries to nodes that were = either down or did not ack before rpc_timeout. That's all.=20 If you are doing thing with read_repair_chance =3D=3D 0 and CL ONE you = are in a very eventually consistent world. The only thing that will = guarantee consistency for you now is running nodetool repair.=20 =20 >=20 >=20 > My cluster is under heavy write load. I'm considering disabling Hinted > Handoffs so the nodes recover quicker in case compactions begin to > back up. If the node cluster is approaching capacity, then ultimately the thing = to do is add more nodes. The only things to do are disable the commit = log and use a lower CL. =20 If it's approaching capacity you will start to see pending mutations = back up, maybe some dropped mutations and the maybe an increase in the = difference between the latency reported in the proxyhistograms and the = cfhistograms or cfstats.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 16/03/2013, at 4:50 PM, Matt Kap wrote: > Thanks Aaron. >=20 > I am using CL=3DONE. read_repair_chance=3D0. The part which I'm = wondering > about is what happens to the internal Cassandra writes if Hinted > Handoffs are disabled. I think I understand what it means for > application-level data, but the part I'm not entirely sure about is > what it could mean for Cassandra internals. >=20 > My cluster is under heavy write load. I'm considering disabling Hinted > Handoffs so the nodes recover quicker in case compactions begin to > back up. >=20 > On Wed, Mar 6, 2013 at 2:06 AM, aaron morton = wrote: >> The advantage of HH is that it reduces the probability of a = DigestMismatch >> when using a CL > ONE. A DigestMismatch means the read has to run a = second >> time before returning to the client. >>=20 >>> - No risk of hinted-handoffs building up >>> - No risk of hinted-handoffs flooding a node that just came up >>=20 >> See the yaml config settings for the max hint window and the = throttling. >>=20 >>> Can anyone suggest any other factors that I'm missing here. = Specifically >>> reasons >>> not to do this. >>=20 >> If you are doing this for performance first make sure your data model = is >> efficient, that you are doing the most efficient reads (see my = presentation >> here = http://www.datastax.com/events/cassandrasummit2012/presentations), and >> your caching is bang on. Then consider if you can tune the CL, and if = your >> client is token aware so it directs traffic to a node that has it. >>=20 >> Cheers >>=20 >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> New Zealand >>=20 >> @aaronmorton >> http://www.thelastpickle.com >>=20 >> On 4/03/2013, at 9:19 PM, Michael Kjellman = wrote: >>=20 >> Also, if you have enough hints being created that its significantly >> impacting your heap I have a feeling things are going to get out of = sync >> very quickly. >>=20 >> On Mar 4, 2013, at 9:17 PM, "Wz1975" wrote: >>=20 >> Why do you think disabling hinted handoff will improve memory usage? >>=20 >>=20 >> Thanks. >> -Wei >>=20 >> Sent from my Samsung smartphone on AT&T >>=20 >>=20 >> -------- Original message -------- >> Subject: Re: hinted handoff disabling trade-offs >> From: Michael Kjellman >> To: "user@cassandra.apache.org" >> CC: >>=20 >>=20 >> Repair is slow. >>=20 >> On Mar 4, 2013, at 8:07 PM, "Matt Kap" wrote: >>=20 >>> I am looking to get a second opinion about disabling = hinted-handoffs. I >>> have an application that can tolerate a fair amount of inconsistency >>> (advertising domain), and so I'm weighting the pros and cons of = hinted >>> handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon. >>>=20 >>> Pros of disabling hinted handoffs: >>> - Reduces heap >>> - Improves GC performance >>> - No risk of hinted-handoffs building up >>> - No risk of hinted-handoffs flooding a node that just came up >>>=20 >>> Cons >>> - Some writes can be lost, at least until repair runs >>>=20 >>> Can anyone suggest any other factors that I'm missing here. = Specifically >>> reasons >>> not to do this. >>>=20 >>> Cheers! >>> -Matt >>=20 >> Copy, by Barracuda, helps you store, protect, and share all your = amazing >> things. Start today: www.copy.com. >>=20 >>=20 >> ---------------------------------- >> Copy, by Barracuda, helps you store, protect, and share all your = amazing >> things. Start today: www.copy.com. >> =AD=AD >>=20 >>=20 >=20 >=20 >=20 > --=20 > www.calcmachine.com - easy online calculator. --Apple-Mail=_A32C39D5-D83F-4C02-A106-15DECF133074 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1  I think I understand what it means = for
application-level data, but the part I'm not entirely sure about = is
what it could mean for Cassandra internals.Internally = it means the write will not be retries to nodes that were either down or = did not ack before rpc_timeout. That's all. 

If = you are doing thing with read_repair_chance =3D=3D 0 and CL ONE you are = in a very eventually consistent world. The only thing that will = guarantee consistency for you now is running nodetool = repair. 

 

My cluster is under heavy write load. I'm considering = disabling Hinted
Handoffs so the nodes recover quicker in case = compactions begin to
back up.
If the node cluster is = approaching capacity, then ultimately the thing to do is add more nodes. = The only things to do are disable the commit log and use a lower CL. =  

If it's approaching capacity you will = start to see pending mutations back up, maybe some dropped mutations and = the maybe an increase in the difference between the latency reported in = the proxyhistograms and the cfhistograms or = cfstats. 

Cheers

http://www.thelastpickle.com

On 16/03/2013, at 4:50 PM, Matt Kap <matvey1414@gmail.com> = wrote:

Thanks Aaron.

I am using CL=3DONE. = read_repair_chance=3D0. The part which I'm wondering
about is what = happens to the internal Cassandra writes if Hinted
Handoffs are = disabled. I think I understand what it means for
application-level = data, but the part I'm not entirely sure about is
what it could mean = for Cassandra internals.

My cluster is under heavy write load. = I'm considering disabling Hinted
Handoffs so the nodes recover = quicker in case compactions begin to
back up.

On Wed, Mar 6, = 2013 at 2:06 AM, aaron morton <aaron@thelastpickle.com> = wrote:
The advantage of HH is that it = reduces the probability of a DigestMismatch
when using a CL > ONE. = A DigestMismatch means the read has to run a second
time before = returning to the client.

- No risk of = hinted-handoffs building up
- No risk of hinted-handoffs flooding a = node that just came up

See the yaml config settings = for the max hint window and the throttling.

Can anyone suggest any other factors that I'm missing = here. Specifically
reasons
not to do this.

If = you are doing this for performance first make sure your data model = is
efficient, that you are doing the most efficient reads (see my = presentation
here = http://www.datastax.com/events/cassandrasummit2012/presentations), = and
your caching is bang on. Then consider if you can tune the CL, = and if your
client is token aware so it directs traffic to a node = that has it.

Cheers

-----------------
Aaron = Morton
Freelance Cassandra Developer
New = Zealand

@aaronmorton
http://www.thelastpickle.com
=
On 4/03/2013, at 9:19 PM, Michael Kjellman = <mkjellman@barracuda.com> wrote:

Also, if you have enough = hints being created that its significantly
impacting your heap I have = a feeling things are going to get out of sync
very quickly.

On = Mar 4, 2013, at 9:17 PM, "Wz1975" <wz1975@YAHOO.COM> = wrote:

Why do you think disabling hinted handoff will improve = memory usage?


Thanks.
-Wei

Sent from my Samsung = smartphone on AT&T


-------- Original message = --------
Subject: Re: hinted handoff disabling trade-offs
From: = Michael Kjellman <mkjellman@barracuda.com>
To: = "user@cassandra.apache.org" = <user@cassandra.apache.org>
CC:


Repair is = slow.

On Mar 4, 2013, at 8:07 PM, "Matt Kap" = <matvey1414@gmail.com> wrote:

I = am looking to get a second opinion about disabling hinted-handoffs. = I
have an application that can tolerate a fair amount of = inconsistency
(advertising domain), and so I'm weighting the pros and = cons of hinted
handoffs. I'm running Cassandra 1.0, looking to = upgrade to 1.1 soon.

Pros of disabling hinted handoffs:
- = Reduces heap
- Improves GC performance
- No risk of = hinted-handoffs building up
- No risk of hinted-handoffs flooding a = node that just came up

Cons
- Some writes can be lost, at = least until repair runs

Can anyone suggest any other factors that = I'm missing here. Specifically
reasons
not to do = this.

Cheers!
-Matt

Copy, by Barracuda, = helps you store, protect, and share all your amazing
things. Start = today: = www.copy.com.


----------------------------------
Copy, by = Barracuda, helps you store, protect, and share all your = amazing
things. Start today: www.copy.com.
=  =AD=AD





--
www.calcmachine.com - easy = online calculator.

= --Apple-Mail=_A32C39D5-D83F-4C02-A106-15DECF133074--