nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Gresock <jgres...@gmail.com>
Subject Re: Queue prioritization with backpressure
Date Fri, 30 Sep 2016 10:01:31 GMT
I forgot to mention the bigger problem with the way this prioritization
works.  If three queues lead to a single component, and all 3 are full due
to back pressure, it is easily observable that once Queue A starts to
drain, it will continue to drain as long as it has incoming flow files.
This means that if there's a constant flow of incoming flow files to Queue
A, the other two queues will never be drained (at least, that's my theory
based on an hour of observation).

I'm going to create a JIRA ticket to track this.

On Thu, Sep 29, 2016 at 4:02 PM, Joe Gresock <jgresock@gmail.com> wrote:

> In my flow, I receive flow files from multiple sources that eventually get
> funneled into one processor.  The connections use OldestFirstPrioritizer,
> and while there is no backpressure, the cross-queue prioritization appears
> to work as designed (i.e., each queue that leads into the funnel appears to
> be fairly drained, resulting in essentially a round robin distribution
> among the funneled queues).
>
> However, as soon as backpressure engages, NiFi appears to want to fully
> drain one of the funneled queues, leaving the others untouched.  This
> happens even when the undrained queues have flow files older than the ones
> that are drained first.  Is this known behavior in NiFi?  If so, do you
> guys think it should be addressed?
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can
> do all this through him who gives me strength.    *-Philippians 4:12-13*
>



-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message