incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Alex <joe.m.a...@gmail.com>
Subject Re: Question on Hinted Handoff
Date Tue, 02 Nov 2010 01:28:09 GMT
My findings - would be nice if somebody can please verify.
Critical for our eval to verify HintedHandOff, ReadRepair and
AntiEntropy works as we think it does

Node 1, 2, 3, 4
RF=3

All nodes up - Node 2 is responsible for key 1005
Write CL=ONE, Insert key 1005, value=123 in Node 1
Node 2, 3, 4 gets data
Read CL=ONE  Read on all 4 nodes gets value 123

Node 2 is down now
Write CL=ONE, Insert key 1005, value=A123 in Node 1
Node 3, 4 gets data
Node 4 adds Hint for Node 2
Read CL=ONE  Read on all 3 Node 1, 3, 4 gets value A123.

Node 2 is brought back up
Hint for 1 row is Handed off by Node 4
Read CL=ONE  Read on Node 1, 2, gets old value 123.
See log entries:
Received responses in DataRepairHandler : ID : 72
FROM:/Node 3
TYPE:RESPONSE_STAGE
VERB:READ_RESPONSE
Received responses in DataRepairHandler : ID : 72
FROM:/Node 4
TYPE:RESPONSE_STAGE
VERB:READ_RESPONSE

Read CL=ONE  Read on Node 3, 4, gets new value A123.
Most of the times after a Read on Node 3 or 4 Node 1 and 2 start
showing the latest updated A123 (updated when Node 2 was down)

My expectation was even though Node 2 was down key written to Node 3
or 4 should be updated in Node 2 using Hint and the subsequent reads
to Node 1 or Node 2 itself should have got the latest value



On Mon, Nov 1, 2010 at 4:06 PM, Joe Alex <joe.m.alex@gmail.com> wrote:
> To keep the question simple,
> If an insert or remove Key happens when the responsible Node is down
> (RF=3) what is the expected behavior when the Node comes back up ?
>
> For example Key 1005 was removed when Node 2 was down. When Node 2
> came back up it started showing back ?
>
> On Mon, Nov 1, 2010 at 2:22 PM, Joe Alex <joe.m.alex@gmail.com> wrote:
>> I am running cassandra 0.6.6
>> 4 nodes with RF=3
>> Have set the InitialTokens manually
>> Loaded around 4 million records
>>
>> Had a question why the following is happening
>>
>> Node 4 was down when a new key 1005 was added (value 123).
>> Node 2 which is responsible for the key added a Hint for Node 4
>> Node 4 was brought back up and noticed the Hints Handed off and data
>> started showing up in Node 4
>> Noticed a ReadRepair also happenning
>> All fine so far
>>
>> did a get and the value is 123
>> Node 2 returned the data, with background digest checks on Node 3 and
>> Node 4 (RF=3)
>>
>> Now Node 2 (responsible for key 1005) was taken down
>> Key 1005 value was updated to A123 (ApplyRowMutation on Node 3 and Node 4).
>> Node 4 added a hint for Node 2
>>
>> did a get and the value is A123
>> Node 3 returned the data, with background digest checks on Node 4
>> (RF=3 and Node 2 is down)
>>
>> Now Node 2 is back up
>> Hints were handed off by Node 4
>>
>> did a get and the value is the old value 123
>> Node 2 returned the data, with background digest checks on Node 3 and
>> Node 4 (RF=3)
>>
>> Was expecting the latest write wins - A123 written on Node 4 to be in Node 2.
>> Any ideas ?
>>
>> Now if Node 2 is down the old value A123 will be returned
>> Tried a repair when Node 2 was up and all Nodes got updated to the old data
>>
>

Mime
View raw message