nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noel Alex Makumuli <alexnoe...@gmail.com>
Subject Improving Nifi Performance
Date Thu, 17 Aug 2017 23:05:48 GMT
Hello guys,

Ever since I tried Apache Nifi, I was amazed with its powerful features and
perfomance.
Every day, I would like to understand and learn more of Nifi and I try to
improve my knowledge of by reading articles and following on the forums.
The applications of nifi have simplied a lot of work as ordinary user and
developer [ I have yet to explore the advance use cases, and i will soon]

Nifi is doing a fantastic job and I am wondering if there anything which I
can
do to improve its performance from my current settings.

I have thousands and thousands [ more than many 20million+] files which need
ETL and I am wondering what are the two cents of valuable advice would
advice
me on how i can improve the performance [ meaning transform more files per
day/ hour]. The files size ranges from 1MB to 60MB.

My set up is very simple. Everything reside in the same location [
location].
However, i need to know if there is anything i can do to improve the
performance. I am sure there one or two twiks I can do achieve the best
performance. Please kindly do share some valuable tips for me.

<code>
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Nifi 1.3

Server RAM 128 GB
</code>
File open limits..
/etc/security/limits.conf
<code>
*         hard    nofile      500000
*         soft    nofile      500000
root      hard    nofile      500000
root      soft    nofile      500000

session required pam_limits.so
</code>
Anything in particular i am doing wrong in particular..?

Please find the attached files in mail.
-- 
NOEL ALEX MAKUMULI

Mime
View raw message