incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan King <r...@twitter.com>
Subject Re: thinking about dropping hinted handoff
Date Tue, 23 Feb 2010 00:57:39 GMT
On Mon, Feb 22, 2010 at 2:05 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
> On Mon, Feb 22, 2010 at 1:53 PM, Ryan King <ryan@twitter.com> wrote:
>> So, after having some more experience with HH, I've reformed my
>> opinion. I think we have 3 options:
>>
>> 1. Make the natural endpoints responsible for the hints.
>> 2. Make a random node responsible for hints.
>> 3. Get rid of HH.
>>
>> #1 reduces the "surprising effects in a small cluster" problem by
>> adding a marginal amount of resource demands to nodes that already
>> have the data we need.
>>
>> #2 will spread the load out. We had a node die last week and decided
>> to leave it down so that we could learn about the effects of this
>> situation. We eventually ended up killing the next node on the ring
>> with all the hints (I think there some improvements to this in 0.6,
>> but I don't know if they'll be enough). So, even on a large cluster
>> (ours is currently 45 nodes), HH can have surprising effects on nodes
>> that neighbor a node that's down. Picking either a random node or
>> using the coordinator node for the hint would spread the load out.
>>
>> #3 is, I think, the right answer. It make our system simpler and it
>> makes the behavior in failure conditions more predictable and safe.
>
> This is a good summary of the options.
>
> Why do you find 3 more compelling than 1?  Yes, it's simpler, but 1
> would not require a large change to the exiting code, so perhaps we
> need a better case than that to justify removing a feature that
> already (mostly) works.

I think I find it more compelling because we're currently experiencing
pain related to HH. I'd be ok with keeping it as long as we can make
the effects of a node down be less drastic.

-ryan

Mime
View raw message