nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Wicks (pwicks)" <pwi...@micron.com>
Subject RE: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times
Date Wed, 24 Jul 2019 13:18:02 GMT
Thanks for the input Mark, I can definitely see that being valuable.  That leads to some new
ideas.

We could have an, "Expiration Strategy" drop down.  Options might be:

 - Lineage Expiration (default, keeps backwards compat.)
 - Queue Expiration
 - Back Pressure + Queue Expiration (which would follow the rules Mark described).

As for the order, I'm not confident I remember how it works right now.

Thanks,
  Peter


-----Original Message-----
From: Mark Bean <mark.o.bean@gmail.com> 
Sent: Wednesday, July 24, 2019 7:08 AM
To: dev@nifi.apache.org
Subject: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times

On a similar note, we recently had a case where it would be desirable for the flowfile expiration
to kick in only after the flowfile size/count back pressure limits have been reached. In other
words, once a back pressure
(size) limit is reached, it would be desirable to then remove flowfiles - beginning with the
oldest first - until the back pressure limit is no longer in violation.

Thanks,
Mark

On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <pwicks@micron.com>
wrote:

> I was thinking it would be nice to expire FlowFile's based on their 
> time in queue, in addition to the current option of their total 
> lineage time (as in, have both options available).
> Any thoughts on pros/cons of having this available?
>
> Thanks,
>   Peter
>
Mime
View raw message