nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <>
Subject Re: Upgrade from 1.0.0 to 1.1.1, cluster config. under heavy load, nodes do not connect
Date Thu, 26 Jan 2017 14:16:17 GMT

NiFi provides an embedded ZooKeeper server for convenience, mostly for 'testing and evaluation'
types of
purposes. For any sort of production or very high-volume flows, I would strongly encourage
you to move ZooKeeper
to its own servers. You will certainly see a lot of problems when trying to interact with
ZooKeeper if the box that
ZooKeeper is running on is under heavy load - either CPU-wise or I/O-wise.


> On Jan 26, 2017, at 7:26 AM, bmichaud <> wrote:
> On Monday, I stood up a cluster with the same configuration as was done
> successfully in 1.0.0 a three-server NiFi cluster. Before I started the
> cluster, I cleaned out all zookeeper state and data from the old cluster,
> but kept the same flow intact, connected to Kafka to pull data from a topic.
> This was a performance environment, and there was heavy load on that kafka
> topic, so it was immediately busy. 
> My strong belief is that, due to the volume of data that the flow needed to
> process during the election process, the election of a coordinator never
> occurred, and, to this day, each node remains disconnected from the others,
> although they are running independently.
> Could this be a defect in NiFi or Zookeeper? What would you suggest that I
> do to resolve this issue?
> All servers in the cluster are configured in the following manner:
> nifi.cluster.node.address=server1 
> nifi.zookeeper.connect.string=server1:2181,server2:2181,server3:2181 
> server.1=server1:2888:3888 
> server.2=server2:2888:3888 
> server.3=server3:2888:3888 
> state-management.xml: 
>    <cluster-provider>
>        <id>zk-provider</id>
> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>        <property name="Connect
> String">server1:2181,server2:2181,server3:2181</property>
>        <property name="Root Node">/nifi</property>
>        <property name="Session Timeout">10 seconds</property>
>        <property name="Access Control">Open</property>
>    </cluster-provider>
> Let me know if you need additional information, please.
> --
> View this message in context:
> Sent from the Apache NiFi Developer List mailing list archive at

View raw message