nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martijn Dekkers" <mart...@dekkers.org.uk>
Subject Re: Number and size of nodes
Date Thu, 30 May 2019 09:20:55 GMT
In clusters, odd numbers of nodes are generally preferred (depending on clustering implementation)
to avoid (to an extent) split-brain scenarios and generally manage quorum. I stand to be corrected,
but in the current implementation in NiFi I don't think this is an issue. 

Additional nodes will give you increased IO throughput for most cases. IO will, in most cases,
be your bottleneck.
 Core/thread count per node will have an impact on scheduling. Matt Clarke wrote an excellent
article on thread usage in NiFi: https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor.html

"Optimal" Cluster design will come down to your anticipated use-cases. Having said that, most
run-of-the-mill "decent" systems will deliver great performance for most systems. If your
needs are more towards the "we *must* have very high performance", or "we *must* process x
messages per second" to a degree of business criticality, you should probably make sure you
design your flow and then design and implement a system to meet the needs of that flow. 

On Thu, 30 May 2019, at 09:52, Christian Andreasen wrote:
> We are planning to build a NiFi cluster and have two questions that we hope you could
help us answer.
>  1. When having our NiFi cluster configured to run with an external Zookeeper cluster
(i.e. not using the default embedded ZK mode) is it then still best practice to have an odd
number of NiFi nodes? If so, why is that?
>  2. Keeping all other things constant, is there then any advantage of running a setup
with 3 NiFi nodes each having 8 cores compared to a setup with 6 nodes each having 4 cores?
> Any input much appreciated.
> 
> Thanks,
> Christian

Mime
View raw message