nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raveendran, Varsha" <varsha.raveend...@siemens.com>
Subject RE: Configure duration to push to RPG in AWS (IoT flow)
Date Tue, 02 May 2017 07:00:53 GMT
Hi Laim,

Thanks for replying.

Do you mean to change Run Scheduling for all processors to 0s? That’s how we have it now.
Also, I see the bottleneck primarily at when the TCP processor is sending the flow files to
the RPG. It starts queuing up though it is just one flow file in the queue. Why does it queue
up?
Am I missing something?

Regards,
Varsha


From: Lee Laim [mailto:lee.laim@gmail.com]
Sent: Tuesday, May 02, 2017 12:14 PM
To: users@nifi.apache.org
Subject: Re: Configure duration to push to RPG in AWS (IoT flow)

Hi Varsha,

Based on your description, It might be worth checking the 'run scheduling' to make sure the
processors are scheduled properly.  I've changed the scheduling for debug purposes, and have
forgotten to changing it back to '0 seconds'

Lee


On May 1, 2017, at 10:38 PM, Raveendran, Varsha <varsha.raveendran@siemens.com<mailto:varsha.raveendran@siemens.com>>
wrote:
Hi Aldrin,

We changed our flow to the following and still see no improvement.

RPi with one instance of MiNiFi

-       A ListenTCP processor

-       RPG connecting to EC2

EC2 with one NiFi server

-       Input Port

-       SplitText, ReplaceText – to transform the data into CQL statements

-       PutCassandra

Without MiNiFi, we are able to transfer data at the rate of 100kB/s to NiFi server.

From the TCP client we are sending 4000 packets, each of size 100bytes. From the MiNiFi logs,
we observe that the values are sent in batches with a time gap of atleast 60 s. Is that the
normal behavior of MiNiFi?
Can we not have a continuous stream of data going to EC2? Another point that I want to mention
is that the device has only 512MB RAM. Could this be a factor?

We have observed that the CPU utilization goes up 99% at times for Minifi process.
Any pointers in optimizing the flow would help.

Regards,
Varsha


From: Aldrin Piri [mailto:aldrinpiri@gmail.com]
Sent: Tuesday, May 02, 2017 1:25 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Configure duration to push to RPG in AWS (IoT flow)

Hi Varsha,

When you are talking about flows, are these separate instances of MiNiFi or different parts
of your overall configuration?  Is there a reason the collected data is written to and then
read from disk?  I/O costs are amplified when running off of SD cards which can be quite slow.
 Not that this accounts for the 50s/event but does provide a point for consideration.  Have
you been able to test network transfer rates to your EC2 NiFi instance outside of MiNiFi?

What is the rate of ingest to the TCP processor both in volume? In number of events?

Would you be able to share your configuration?

Have run similar flows on RPis and equivalents and had good throughput and seems like something
has gone slightly awry.

On Sat, Apr 29, 2017 at 5:30 AM, Raveendran, Varsha <varsha.raveendran@siemens.com<mailto:varsha.raveendran@siemens.com>>
wrote:
Hello All,

I have a MiNiFi flow running on Raspberry Pi. The flow has a ListenTCP that pushes the events
from a client to a folder using PutFile. Another flow takes files from a folder using ListFile
and FetchFile and pushes it to a remote process group pointing to a NiFi server on EC2. From
the logs on the Raspberry Pi we observed that the queue between FetchFile and RPG is never
empty. The Nifi server in EC2 receives messages every 50 seconds. This is slow for a real
time application that is expecting atleast 1 flow file every second.

Any thoughts on how to improve the performance in MiNiFi to push messages faster to the Input
port on EC2?

Regards,
Varsha

Registered Office: 130 Pandurang Budhkar Marg, Worli, Mumbai – 400018; Corporate Identity
number: L28920MH1957PLC010839; Tel.: +91 (22) 3967 7000<tel:+91%2022%203967%207000>;
Fax: +91 22 3967 7500<tel:+91%2022%203967%207500>;
Contact / Email: www.siemens.co.in/contact<http://www.siemens.co.in/contact>; Website:
www.siemens.co.in<http://www.siemens.co.in>. Sales Offices: Ahmedabad, Bengaluru, Bhopal,
Bhubaneswar, Chandigarh, Chennai, Coimbatore, Gurgaon, Hyderabad, Jaipur, Jamshedpur, Kharghar,
Kolkata, Lucknow, Kochi, Mumbai, Nagpur, Navi Mumbai, New Delhi,


Registered Office: 130 Pandurang Budhkar Marg, Worli, Mumbai – 400018; Corporate Identity
number: L28920MH1957PLC010839; Tel.: +91 (22) 3967 7000; Fax: +91 22 3967 7500;
Contact / Email: www.siemens.co.in/contact<http://www.siemens.co.in/contact>; Website:
www.siemens.co.in<http://www.siemens.co.in>. Sales Offices: Ahmedabad, Bengaluru, Bhopal,
Bhubaneswar, Chandigarh, Chennai, Coimbatore, Gurgaon, Hyderabad, Jaipur, Jamshedpur, Kharghar,
Kolkata, Lucknow, Kochi, Mumbai, Nagpur, Navi Mumbai, New Delhi,

Registered Office: 130 Pandurang Budhkar Marg, Worli, Mumbai – 400018; Corporate Identity
number: L28920MH1957PLC010839; Tel.: +91 (22) 3967 7000; Fax: +91 22 3967 7500;
Contact / Email: www.siemens.co.in/contact; Website: www.siemens.co.in. Sales Offices: Ahmedabad,
Bengaluru, Bhopal, Bhubaneswar, Chandigarh, Chennai, Coimbatore, Gurgaon, Hyderabad, Jaipur,
Jamshedpur, Kharghar, Kolkata, Lucknow, Kochi, Mumbai, Nagpur, Navi Mumbai, New Delhi,
Mime
View raw message