incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan King <r...@twitter.com>
Subject Re: thinking about dropping hinted handoff
Date Mon, 22 Feb 2010 19:53:04 GMT
On Wed, Jan 27, 2010 at 12:02 PM, Ryan King <ryan@twitter.com> wrote:
> On Wed, Jan 27, 2010 at 11:49 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>> On Wed, Jan 27, 2010 at 1:48 PM, Stu Hood <stu.hood@rackspace.com> wrote:
>>>> The HH code currently tries to send the hints to nodes other than the
>>>> natural endpoints. If small-scale performance is a problem, we could
>>>> make the natural endpoints be responsible for the hints. This reduces
>>>> durability a bit, but might be a decent tradeoff.
>>> The other interesting benefit is that the hint would not need to store the actual
content of the change, since the natural endpoints will already be storing copies. The hints
would just need to encode the fact that a given (key,name1[,name2]) changed.
>>
>> Right, I think that's what Ryan was getting at.
>
> Indeed. Like I said, this change only helps with the "surprising
> effects on a small cluster" problem, but if that's enough, perhaps we
> should do it.

So, after having some more experience with HH, I've reformed my
opinion. I think we have 3 options:

1. Make the natural endpoints responsible for the hints.
2. Make a random node responsible for hints.
3. Get rid of HH.

#1 reduces the "surprising effects in a small cluster" problem by
adding a marginal amount of resource demands to nodes that already
have the data we need.

#2 will spread the load out. We had a node die last week and decided
to leave it down so that we could learn about the effects of this
situation. We eventually ended up killing the next node on the ring
with all the hints (I think there some improvements to this in 0.6,
but I don't know if they'll be enough). So, even on a large cluster
(ours is currently 45 nodes), HH can have surprising effects on nodes
that neighbor a node that's down. Picking either a random node or
using the coordinator node for the hint would spread the load out.

#3 is, I think, the right answer. It make our system simpler and it
makes the behavior in failure conditions more predictable and safe.

What do you all think?

-ryan

Mime
View raw message