cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fred Habash <fmhab...@gmail.com>
Subject Re: CL=LQ, RF=3: Can a Write be Lost If Two Nodes ACK'ing it Die
Date Fri, 03 May 2019 13:19:10 GMT
Thank you all.

So, please, bear with me for a second. I'm trying to figure out how can
data be totally lost under the above circumstances when nodes die in two
out of three racks.

You stated ....
"the replica may or many not have made its way to the third node '. Why
'may not'?

This is what I came up with ...

1. Write goes to coordinator in rac1
2. Local coordinator submits RF = 3 writes to all racks
3. Two nodes in rack 1 and 2 ack the write. Client is happy
4. Nodes massacre happens in rack 1 & 2 (infrastructure event)
5. Nodes in rack3 witness in increased in load as a result of cluster
shrinking
6. Coordinator in rack1 stores HH for the row for rack3 (either coordinator
slows down or rack3 node is overloaded).
7. Eventually, coordinator in rack1 dies and HH's are lost.
8. The row that was once ack'd to the app, is now gone gone.

Plausible?


On Thu, May 2, 2019 at 8:23 PM Avinash Mandava <avinash@vorstella.com>
wrote:

> Good catch, misread the detail.
>
> On Thu, May 2, 2019 at 4:56 PM Ben Slater <ben.slater@instaclustr.com>
> wrote:
>
>> Reading more carefully, it could actually be either way: quorum requires
>> that a majority of nodes complete and ack the write but still aims to write
>> to RF nodes (with the last replicate either written immediately or
>> eventually via hints or repairs). So, in the scenario outlined the replica
>> may or many not have made its way to the third node by the time the first
>> two replicas are lost. If there is a replica on the third node it can be
>> recovered to the other two nodes by either rebuild (actually replace) or
>> repair.
>>
>> Cheers
>> Ben
>>
>> ---
>>
>>
>> *Ben Slater**Chief Product Officer*
>>
>> <https://www.instaclustr.com/platform/>
>>
>> <https://www.facebook.com/instaclustr>
>> <https://twitter.com/instaclustr>
>> <https://www.linkedin.com/company/instaclustr>
>>
>> Read our latest technical blog posts here
>> <https://www.instaclustr.com/blog/>.
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information.  If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>>
>> On Fri, 3 May 2019 at 09:33, Avinash Mandava <avinash@vorstella.com>
>> wrote:
>>
>>> In scenario 2 it's lost, if both nodes die and get replaced entirely
>>> there's no history anywhere that the write ever happened, as it wouldn't be
>>> in commitlog, memtable, or sstable in node 3. Surviving that failure
>>> scenario of two nodes with same data simultaneously failing requires upping
>>> CL or RF, or spreading across 3 racks, if the situation you're trying to
>>> avoid is rack failure (which im guessing it is from the question setup)
>>>
>>> On Thu, May 2, 2019 at 2:25 PM Ben Slater <ben.slater@instaclustr.com>
>>> wrote:
>>>
>>>> In scenario 2, if the row has been written to node 3 it will be
>>>> replaced on the other nodes via rebuild or repair.
>>>>
>>>> ---
>>>>
>>>>
>>>> *Ben Slater**Chief Product Officer*
>>>>
>>>> <https://www.instaclustr.com/platform/>
>>>>
>>>> <https://www.facebook.com/instaclustr>
>>>> <https://twitter.com/instaclustr>
>>>> <https://www.linkedin.com/company/instaclustr>
>>>>
>>>> Read our latest technical blog posts here
>>>> <https://www.instaclustr.com/blog/>.
>>>>
>>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>>> (Australia) and Instaclustr Inc (USA).
>>>>
>>>> This email and any attachments may contain confidential and legally
>>>> privileged information.  If you are not the intended recipient, do not copy
>>>> or disclose its content, but please reply to this email immediately and
>>>> highlight the error to the sender and then immediately delete the message.
>>>>
>>>>
>>>> On Fri, 3 May 2019 at 00:54, Fd Habash <fmhabash@gmail.com> wrote:
>>>>
>>>>> C*: 2.2.8
>>>>>
>>>>> Write CL = LQ
>>>>>
>>>>> Kspace RF = 3
>>>>>
>>>>> Three racks
>>>>>
>>>>>
>>>>>
>>>>> A write gets received by node 1 in rack 1 at above specs. Node 1
>>>>> (rack1) & node 2 (rack2)  acknowledge it to the client.
>>>>>
>>>>>
>>>>>
>>>>> Within some unit of time, node 1 & 2 die. Either ….
>>>>>
>>>>>    - Scenario 1: C* process death: Row did not make it to sstable (it
>>>>>    is in commit log & was in memtable)
>>>>>    - Scenario 2: Node death: row may be have made to sstable, but
>>>>>    nodes are gone (will have to bootstrap to replace).
>>>>>
>>>>>
>>>>>
>>>>> Scenario 1: Row is not lost because once C* is restarted, commit log
>>>>> should replay the mutation.
>>>>>
>>>>>
>>>>>
>>>>> Scenario 2: row is gone forever? If these two nodes are replaced via
>>>>> bootstrapping, will they ever get the row back from node 3 (rack3) if
the
>>>>> write ever made it there?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----------------
>>>>> Thank you
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> www.vorstella.com
>>> 408 691 8402
>>>
>>
>
> --
> www.vorstella.com
> 408 691 8402
>


-- 

----------------------------------------
Thank you

Mime
View raw message