nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Clustering Best Practices?
Date Thu, 20 Apr 2017 12:52:16 GMT
Jim,

I would offer you a few bits of advice. First, NiFi relies on ZooKeeper to coordinate which
node is responsible
to act as the Cluster Coordinator and which node should be the Primary Node. NiFi does allow
you to start
and embedded ZooKeeper, but for production use, it is recommended that you use an external
ZooKeeper
running on different boxes.

Secondly, the default timeouts used for clustering (the "nifi.cluster.node.connection.timeout"
and
"nifi.cluster.node.read.timeout" properties in nifi.properties) are set to 5 seconds. This
is okay for a "getting started"
type of cluster. However, if you start processing large numbers of FlowFiles, the JVM's garbage
collection can sometimes
cause some fairly lengthy pauses. This could cause timeouts. So I would recommend increasing
that to 10-15 seconds
or more.

Finally, the "nifi.zookeeper.connect.timeout" and "nifi.zookeeper.session.timeout" default
to 3 seconds, as this is what
the ZooKeeper default is. However, I've heard some people indicate that they saw frequent
ZooKeeper timeouts, which caused
the Primary Node and Cluster Coordinator to change frequently. Changing the value to 5 or
10 seconds was much better.

Thanks
-Mark

> On Apr 20, 2017, at 7:01 AM, James McMahon <jsmcmahon3@gmail.com> wrote:
> 
> Good morning. I have established an initial single-threaded NiFi server instance for
my customers. It works well, but I anticipate increasing usage as groups learn more about
it. I also want to move beyond our single threaded-ness.
> 
> I would like to take the next step in the evolution of our NiFi capability, implementing
a clustered NiFi server configuration to help me address the following requirements:
> 1. increase our fault tolerance
> 2. permit our configuration to scale to peak processing demands during bulk data loads
and as more customers begin to leverage our NiFi instance
> 3. permit our configuration to load balance
> 
> I do intend to begin by reading through the clustering sections in the NiFi Sys Admin
guide. I am also interested in hearing from our user community, particularly regarding clustering
"best practices" and practical insights based on your experiences. Thanks in advance for any
insights you are willing to share.  -Jim


Mime
View raw message