nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <>
Subject [jira] [Commented] (NIFI-6849) On startup, NiFi should be more liberal about what it's willing to inherit from cluster
Date Thu, 09 Apr 2020 15:18:00 GMT


ASF subversion and git services commented on NIFI-6849:

Commit 683b1d995200384dcde8ec269163bc57f25b23ce in nifi's branch refs/heads/master from Mark
[;h=683b1d9 ]

NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if
there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's
flow is inherited.
 - Refactored Flow Synchronization to make code cleaner
 - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local
flow is empty.
 - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord,
so that we have the ability to read records without already knowing the Queue objects. Updated
StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not
yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow
uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING'
- Added additional log statements to aid in debugging

NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance

NIFI-6849: Addressed review feedback

NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck
class. Fixed logic that prevented users/groups/policies from being forcibly inherited during

This closes #3891

> On startup, NiFi should be more liberal about what it's willing to inherit from cluster
> ---------------------------------------------------------------------------------------
>                 Key: NIFI-6849
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>          Time Spent: 3.5h
>  Remaining Estimate: 0h
> On startup, if an instance is configured to be clustered, the instance must connect to
the Cluster Coordinator and download the existing cluster flow, access policies, and users
and groups. The instance then performs some checks to ensure that the local flow matches the
cluster's flow. If it doesn't, then the node fails to startup and logs errors that the local
flow is different than the cluster's flow.
> This was done in order to facilitate debugging. If a particular node is not behaving
as expected, a user is able to disconnect the node from the cluster and make modifications
to the node. If the node is then restarted, it may not be desirable to lose those changes.
> However, in the vast majority of cases (probably over 98% of the time), what the user
really wants is for the node to just join back to the cluster and inherit the cluster's flow
- especially if the node just disconnected because it failed to make a modification. This
is also problematic with how the Users, Groups, and Policies are inherited.
> As a result, we should make the following modifications:
> 1) If Users, Groups, or Access Policies cannot be inherited, continue to fail, unless
the flow is empty. If the flow is empty, it doesn't really make sense to retain the authorizations'
configuration because they don't really apply to anything. As a result, if the flow is empty,
just inherit whatever the cluster has. But first, make a backup of the existing policies,
users, and groups, so that users can manually revert if they do end up needing to.
> 2) If the flow differs from the cluster flow, check the proposed flow to see if it removes
any existing connections. If it does remove a connection, and that connection has data queued
locally, continue to fail. Otherwise, create a backup of the flow and replace the node's flow
with the cluster flow.

This message was sent by Atlassian Jira

View raw message