curator-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuri Tceretian (JIRA)" <>
Subject [jira] [Updated] (CURATOR-504) Race conditions in LeaderLatch after reconnecting to ensemble
Date Tue, 05 Feb 2019 20:13:00 GMT


Yuri Tceretian updated CURATOR-504:
    Attachment: XP91JuD048Nl_8h9NZpH01QZJMfCLewjfd2eQNfOsR6GuApPNVJlkonWQLBR8pjltdpp1gtUMsE1VOgQmXn95fL68Ha80taHb8hAsY0mtcfXbGwv8iBIrefoAook18a3wZ3o3JnV7JkyO_QDP2UMMH44Jr7WX50Rs6JyPRtEU15aHVUzLXAViK6BqidYAEhUmqS7iZYWXRAg6wcHO9ViWznti2-jIZIcfWCRJ5Z8N7CI_KrX.png

> Race conditions in LeaderLatch after reconnecting to ensemble
> -------------------------------------------------------------
>                 Key: CURATOR-504
>                 URL:
>             Project: Apache Curator
>          Issue Type: Bug
>    Affects Versions: 4.1.0
>            Reporter: Yuri Tceretian
>            Assignee: Jordan Zimmerman
>            Priority: Minor
>         Attachments: 51868597-65791000-231c-11e9-9bfa-1def62bc3ea1.png, Screen Shot 2019-01-31
at 10.26.59 PM.png, XP91JuD048Nl_8h9NZpH01QZJMfCLewjfd2eQNfOsR6GuApPNVJlkonWQLBR8pjltdpp1gtUMsE1VOgQmXn95fL68Ha80taHb8hAsY0mtcfXbGwv8iBIrefoAook18a3wZ3o3JnV7JkyO_QDP2UMMH44Jr7WX50Rs6JyPRtEU15aHVUzLXAViK6BqidYAEhUmqS7iZYWXRAg6wcHO9ViWznti2-jIZIcfWCRJ5Z8N7CI
(1).png, XP91JuD048Nl_8h9NZpH01QZJMfCLewjfd2eQNfOsR6GuApPNVJlkonWQLBR8pjltdpp1gtUMsE1VOgQmXn95fL68Ha80taHb8hAsY0mtcfXbGwv8iBIrefoAook18a3wZ3o3JnV7JkyO_QDP2UMMH44Jr7WX50Rs6JyPRtEU15aHVUzLXAViK6BqidYAEhUmqS7iZYWXRAg6wcHO9ViWznti2-jIZIcfWCRJ5Z8N7CI_KrX.png
> We use LeaderLatch in a lot of places in our system and when ZooKeeper ensemble is unstable
and clients are reconnecting to logs are full of messages like the following:
> {{[2017-08-31 19:18:34,562][ERROR][]
Can't find our node. Resetting. Index: -1 {}}}
> According to the [implementation|],
this can happen in two cases:
>  * When internal state `ourPath` is null
>  * When the list of latches does not have the expected one.
> I believe we hit the first condition because of races that occur after client reconnects
to ZooKeeper.
>  * Client reconnects to ZooKeeper and LeaderLatch gets the event and calls reset method
which set the internal state (`ourPath`) to null, removes old latch and creates a new one.
This happens in thread "Curator-ConnectionStateManager-0".
>  * Almost simultaneously, LeaderLatch gets another even NodeDeleted ([here|])
and tries to re-read the list of latches and check leadership. This happens in the thread
> Therefore, sometimes there is a situation when method `checkLeadership` is called when
`ourPath` is null.
> Below is an approximate diagram of what happens:
> !51868597-65791000-231c-11e9-9bfa-1def62bc3ea1.png|width=1261,height=150!

This message was sent by Atlassian JIRA

View raw message