nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shaine Berube <shaine.ber...@perfectsearchcorp.com>
Subject Limiting a queue
Date Wed, 08 Jun 2016 14:40:54 GMT
Hello all,

I'm kind of new to developing Nifi, though I've been doing some pretty in
depth stuff and some advanced database queries.  My question is in
regarding the queues between processor, I want to limit a queue to... say
2000, how would I go about doing that?  Or better yet, how would I tell the
processor generating the queue to only put a max of 2000 files into the
queue?

Allow me to explain with a scenario:
We are doing data migration from one database to another.
-Processor A is generating a queue consumed by Processor B
-Processor A is taking configuration and generating SQL queries in 1000
record chunks so that Processor B can insert them into a new database.
Given the size of the source database, Processor A can potentially generate
hundreds of thousands of files.

Is there a way for Processor A to check it's down stream queue for the
queue size?  How would I get Processor A to only put 2000 files into the
queue at any given time, so that Processor A can continue running but wait
for room in the queue?

Thank you in advance.

-- 
*Shaine Berube*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message