nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Bende (Jira)" <>
Subject [jira] [Resolved] (NIFI-6589) Leader Election should cache results obtained from ZooKeeper
Date Wed, 02 Oct 2019 14:57:01 GMT


Bryan Bende resolved NIFI-6589.
    Resolution: Fixed

> Leader Election should cache results obtained from ZooKeeper
> ------------------------------------------------------------
>                 Key: NIFI-6589
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Minor
>             Fix For: 1.10.0
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
> In order to determine which node in a cluster is the Cluster Coordinator, a node must
make a request to ZooKeeper. That means that if we have N nodes in a cluster, then we must
ask ZooKeeper for each request at least (N+1) times (and no more than N+2) who is the Cluster
Coordinator. This is done because when the request comes in, the node must determine whether
or not it is the Cluster Coordinator. If so, it must replicate the request to each node. If
not, it must forward the request to the Cluster Coordinator, which will then do so. When the
request is replicated, it will again check if it is the cluster coordinator. If we instead
cache the result of querying ZooKeeper for a short period of time, say 1 minute, we can dramatically
decrease the number of times that we hit ZooKeeper. If the Coordinator / Primary Node changes
in the mean time, it should still be notified of the change asynchronously.
> The polling is done currently because we've seen situations where the asynchronous notification
did not happen. But if we update the code so that we cache the results, this means that we
will also update the code for caching results of which node is Primary Node. This is a benefit
as well, because currently we don't poll for this and as a result, if we do happen to miss
the notification, we could theoretically have 2 nodes running processors should only run on
Primary Node.

This message was sent by Atlassian Jira

View raw message