incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: hinted handoff disabling trade-offs
Date Tue, 19 Mar 2013 07:14:43 GMT
>  I think I understand what it means for
> application-level data, but the part I'm not entirely sure about is
> what it could mean for Cassandra internals.
Internally it means the write will not be retries to nodes that were either down or did not
ack before rpc_timeout. That's all. 

If you are doing thing with read_repair_chance == 0 and CL ONE you are in a very eventually
consistent world. The only thing that will guarantee consistency for you now is running nodetool
repair. 

 
> 
> 
> My cluster is under heavy write load. I'm considering disabling Hinted
> Handoffs so the nodes recover quicker in case compactions begin to
> back up.
If the node cluster is approaching capacity, then ultimately the thing to do is add more nodes.
The only things to do are disable the commit log and use a lower CL.  

If it's approaching capacity you will start to see pending mutations back up, maybe some dropped
mutations and the maybe an increase in the difference between the latency reported in the
proxyhistograms and the cfhistograms or cfstats. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/03/2013, at 4:50 PM, Matt Kap <matvey1414@gmail.com> wrote:

> Thanks Aaron.
> 
> I am using CL=ONE. read_repair_chance=0. The part which I'm wondering
> about is what happens to the internal Cassandra writes if Hinted
> Handoffs are disabled. I think I understand what it means for
> application-level data, but the part I'm not entirely sure about is
> what it could mean for Cassandra internals.
> 
> My cluster is under heavy write load. I'm considering disabling Hinted
> Handoffs so the nodes recover quicker in case compactions begin to
> back up.
> 
> On Wed, Mar 6, 2013 at 2:06 AM, aaron morton <aaron@thelastpickle.com> wrote:
>> The advantage of HH is that it reduces the probability of a DigestMismatch
>> when using a CL > ONE. A DigestMismatch means the read has to run a second
>> time before returning to the client.
>> 
>>> - No risk of hinted-handoffs building up
>>> - No risk of hinted-handoffs flooding a node that just came up
>> 
>> See the yaml config settings for the max hint window and the throttling.
>> 
>>> Can anyone suggest any other factors that I'm missing here. Specifically
>>> reasons
>>> not to do this.
>> 
>> If you are doing this for performance first make sure your data model is
>> efficient, that you are doing the most efficient reads (see my presentation
>> here http://www.datastax.com/events/cassandrasummit2012/presentations), and
>> your caching is bang on. Then consider if you can tune the CL, and if your
>> client is token aware so it directs traffic to a node that has it.
>> 
>> Cheers
>> 
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>> 
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 4/03/2013, at 9:19 PM, Michael Kjellman <mkjellman@barracuda.com> wrote:
>> 
>> Also, if you have enough hints being created that its significantly
>> impacting your heap I have a feeling things are going to get out of sync
>> very quickly.
>> 
>> On Mar 4, 2013, at 9:17 PM, "Wz1975" <wz1975@YAHOO.COM> wrote:
>> 
>> Why do you think disabling hinted handoff will improve memory usage?
>> 
>> 
>> Thanks.
>> -Wei
>> 
>> Sent from my Samsung smartphone on AT&T
>> 
>> 
>> -------- Original message --------
>> Subject: Re: hinted handoff disabling trade-offs
>> From: Michael Kjellman <mkjellman@barracuda.com>
>> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
>> CC:
>> 
>> 
>> Repair is slow.
>> 
>> On Mar 4, 2013, at 8:07 PM, "Matt Kap" <matvey1414@gmail.com> wrote:
>> 
>>> I am looking to get a second opinion about disabling hinted-handoffs. I
>>> have an application that can tolerate a fair amount of inconsistency
>>> (advertising domain), and so I'm weighting the pros and cons of hinted
>>> handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.
>>> 
>>> Pros of disabling hinted handoffs:
>>> - Reduces heap
>>> - Improves GC performance
>>> - No risk of hinted-handoffs building up
>>> - No risk of hinted-handoffs flooding a node that just came up
>>> 
>>> Cons
>>> - Some writes can be lost, at least until repair runs
>>> 
>>> Can anyone suggest any other factors that I'm missing here. Specifically
>>> reasons
>>> not to do this.
>>> 
>>> Cheers!
>>> -Matt
>> 
>> Copy, by Barracuda, helps you store, protect, and share all your amazing
>> things. Start today: www.copy.com.
>> 
>> 
>> ----------------------------------
>> Copy, by Barracuda, helps you store, protect, and share all your amazing
>> things. Start today: www.copy.com.
>>  ­­
>> 
>> 
> 
> 
> 
> -- 
> www.calcmachine.com - easy online calculator.


Mime
View raw message