nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kumiko Yada <Kumiko.Y...@ds-iq.com>
Subject Best way to process the process the requests in batch
Date Fri, 27 May 2016 00:34:30 GMT
Hello,

We implemented the custom process that are similar to the InvokeHTTP that the part of URL
can be replace with the Context Data List, then write the weather to the flowfile.  For example,
URL to get the weather feed have to include the zip code in URL, and the ZIP code is {0} in
the URL and replaced the zip code from the Context Data List property.

URL
http://example{0}/weather<http://example%7b0%7d/weather>

Context Data List:
00000
11111
22222

Processor with make the following requests:
http://example{0}/weather<http://example%7b0%7d/weather>

http://example00000/weather
http://example11111/weather
http://example22222/weather

This processor is processed in one request at a time and have a perf issue.  I'd like to modify
to process in batches.  What are the best way to process in batches?  And also, would the
Nifi keep track how many requests the processor is processed?  If so, how the Nifi keep track
this and how long the Nifi keep track of data?  I'd like to add the quota priorities in this
processor to keep track of quota.  For example, if the weather feeds can be requested only
100 requests a day, I don't want to processor to executed once the quota is reached.

Thanks
Kumiko

Mime
View raw message