incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: thinking about dropping hinted handoff
Date Mon, 22 Feb 2010 22:05:52 GMT
On Mon, Feb 22, 2010 at 1:53 PM, Ryan King <ryan@twitter.com> wrote:
> So, after having some more experience with HH, I've reformed my
> opinion. I think we have 3 options:
>
> 1. Make the natural endpoints responsible for the hints.
> 2. Make a random node responsible for hints.
> 3. Get rid of HH.
>
> #1 reduces the "surprising effects in a small cluster" problem by
> adding a marginal amount of resource demands to nodes that already
> have the data we need.
>
> #2 will spread the load out. We had a node die last week and decided
> to leave it down so that we could learn about the effects of this
> situation. We eventually ended up killing the next node on the ring
> with all the hints (I think there some improvements to this in 0.6,
> but I don't know if they'll be enough). So, even on a large cluster
> (ours is currently 45 nodes), HH can have surprising effects on nodes
> that neighbor a node that's down. Picking either a random node or
> using the coordinator node for the hint would spread the load out.
>
> #3 is, I think, the right answer. It make our system simpler and it
> makes the behavior in failure conditions more predictable and safe.

This is a good summary of the options.

Why do you find 3 more compelling than 1?  Yes, it's simpler, but 1
would not require a large change to the exiting code, so perhaps we
need a better case than that to justify removing a feature that
already (mostly) works.

-Jonathan

Mime
View raw message