cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hiroyuki Yamada <mogwa...@gmail.com>
Subject Re: A cluster (RF=3) not recovering after two nodes are stopped
Date Fri, 26 Apr 2019 06:17:45 GMT
Hello,

Thank you for some feedbacks.

>Ben
Thank you.
I've tested with lower concurrency in my side, the issue still occurs.
We are using 3 x T3.xlarge instances for C* and small and separate instance
for the client program.
But if we tried with 1 host with 3 C* nodes, the issue didn't occur.

> Alok
We also thought so and tested with hints disabled, but it doesn't make any
difference. (the issue still occurs)

Thanks,
Hiro




On Fri, Apr 26, 2019 at 8:19 AM Alok Dwivedi <alok.dwivedi@instaclustr.com>
wrote:

> Could it be related to hinted hand offs being stored in Node1 and then
> attempted to be replayed in Node2 when it comes back causing more load as
> new mutations are also being applied from cassandra-stress at same time?
>
> Alok Dwivedi
> Senior Consultant
> https://www.instaclustr.com/
>
>
>
>
> On 26 Apr 2019, at 09:04, Ben Slater <ben.slater@instaclustr.com> wrote:
>
> In the absence of anyone else having any bright ideas - it still sounds to
> me like the kind of scenario that can occur in a heavily overloaded
> cluster. I would try again with a lower load.
>
> What size machines are you using for stress client and the nodes? Are they
> all on separate machines?
>
> Cheers
> Ben
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
> <https://www.instaclustr.com/platform/>
>
> <https://www.facebook.com/instaclustr>   <https://twitter.com/instaclustr>
>    <https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Thu, 25 Apr 2019 at 17:26, Hiroyuki Yamada <mogwaing@gmail.com> wrote:
>
>> Hello,
>>
>> Sorry again.
>> We found yet another weird thing in this.
>> If we stop nodes with systemctl or just kill (TERM), it causes the
>> problem,
>> but if we kill -9, it doesn't cause the problem.
>>
>> Thanks,
>> Hiro
>>
>> On Wed, Apr 24, 2019 at 11:31 PM Hiroyuki Yamada <mogwaing@gmail.com>
>> wrote:
>>
>>> Sorry, I didn't write the version and the configurations.
>>> I've tested with C* 3.11.4, and
>>> the configurations are mostly set to default except for the replication
>>> factor and listen_address for proper networking.
>>>
>>> Thanks,
>>> Hiro
>>>
>>> On Wed, Apr 24, 2019 at 5:12 PM Hiroyuki Yamada <mogwaing@gmail.com>
>>> wrote:
>>>
>>>> Hello Ben,
>>>>
>>>> Thank you for the quick reply.
>>>> I haven't tried that case, but it does't recover even if I stopped the
>>>> stress.
>>>>
>>>> Thanks,
>>>> Hiro
>>>>
>>>> On Wed, Apr 24, 2019 at 3:36 PM Ben Slater <ben.slater@instaclustr.com>
>>>> wrote:
>>>>
>>>>> Is it possible that stress is overloading node 1 so it’s not
>>>>> recovering state properly when node 2 comes up? Have you tried running
with
>>>>> a lower load (say 2 or 3 threads)?
>>>>>
>>>>> Cheers
>>>>> Ben
>>>>>
>>>>> ---
>>>>>
>>>>>
>>>>> *Ben Slater*
>>>>> *Chief Product Officer*
>>>>>
>>>>>
>>>>> <https://www.facebook.com/instaclustr>
>>>>> <https://twitter.com/instaclustr>
>>>>> <https://www.linkedin.com/company/instaclustr>
>>>>>
>>>>> Read our latest technical blog posts here
>>>>> <https://www.instaclustr.com/blog/>.
>>>>>
>>>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>>>> (Australia) and Instaclustr Inc (USA).
>>>>>
>>>>> This email and any attachments may contain confidential and legally
>>>>> privileged information.  If you are not the intended recipient, do not
copy
>>>>> or disclose its content, but please reply to this email immediately and
>>>>> highlight the error to the sender and then immediately delete the message.
>>>>>
>>>>>
>>>>> On Wed, 24 Apr 2019 at 16:28, Hiroyuki Yamada <mogwaing@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I faced a weird issue when recovering a cluster after two nodes are
>>>>>> stopped.
>>>>>> It is easily reproduce-able and looks like a bug or an issue to fix,
>>>>>> so let me write down the steps to reproduce.
>>>>>>
>>>>>> === STEPS TO REPRODUCE ===
>>>>>> * Create a 3-node cluster with RF=3
>>>>>>    - node1(seed), node2, node3
>>>>>> * Start requests to the cluster with cassandra-stress (it continues
>>>>>> until the end)
>>>>>>    - what we did: cassandra-stress mixed cl=QUORUM duration=10m
>>>>>> -errors ignore -node node1,node2,node3 -rate threads\>=16
>>>>>> threads\<=256
>>>>>> * Stop node3 normally (with systemctl stop)
>>>>>>    - the system is still available because the quorum of nodes is
>>>>>> still available
>>>>>> * Stop node2 normally (with systemctl stop)
>>>>>>    - the system is NOT available after it's stopped.
>>>>>>    - the client gets `UnavailableException: Not enough replicas
>>>>>> available for query at consistency QUORUM`
>>>>>>    - the client gets errors right away (so few ms)
>>>>>>    - so far it's all expected
>>>>>> * Wait for 1 mins
>>>>>> * Bring up node2
>>>>>>    - The issue happens here.
>>>>>>    - the client gets ReadTimeoutException` or WriteTimeoutException
>>>>>> depending on if the request is read or write even after the node2
is
>>>>>> up
>>>>>>    - the client gets errors after about 5000ms or 2000ms, which are
>>>>>> request timeout for write and read request
>>>>>>    - what node1 reports with `nodetool status` and what node2 reports
>>>>>> are not consistent. (node2 thinks node1 is down)
>>>>>>    - It takes very long time to recover from its state
>>>>>> === STEPS TO REPRODUCE ===
>>>>>>
>>>>>> Is it supposed to happen ?
>>>>>> If we don't start cassandra-stress, it's all fine.
>>>>>>
>>>>>> Some workarounds we found to recover the state are the followings:
>>>>>> * Restarting node1 and it recovers its state right after it's
>>>>>> restarted
>>>>>> * Setting lower value in dynamic_snitch_reset_interval_in_ms (to
60000
>>>>>> or something)
>>>>>>
>>>>>> I don't think either of them is a really good solution.
>>>>>> Can anyone explain what is going on and what is the best way to make
>>>>>> it not happen or recover ?
>>>>>>
>>>>>> Thanks,
>>>>>> Hiro
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>>>
>>>>>>
>

Mime
View raw message