nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Ghigiarelli <briang...@gmail.com>
Subject Re: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times
Date Wed, 24 Jul 2019 13:31:06 GMT
To add to the back pressure management options, we wrote a custom
"RouteOnBackPressure" processor [1] that would allow us to re-route the
incoming flow after outgoing queues hit their backpressure thresholds. The
processor itself doesn't handle anything with expiration (though output
queues could), and requires our third-party NAR bundle [2], but it works
well for handling surges.

[1]
https://github.com/Asymmetrik/nifi-nar-bundles/blob/develop/nifi-asymmetrik-standard-bundle/nifi-asymmetrik-standard-processors/src/main/java/com/asymmetrik/nifi/processors/RouteOnBackPressure.java
[2] https://github.com/Asymmetrik/nifi-nar-bundles

On Wed, Jul 24, 2019 at 9:18 AM Peter Wicks (pwicks) <pwicks@micron.com>
wrote:

> Thanks for the input Mark, I can definitely see that being valuable.  That
> leads to some new ideas.
>
> We could have an, "Expiration Strategy" drop down.  Options might be:
>
>  - Lineage Expiration (default, keeps backwards compat.)
>  - Queue Expiration
>  - Back Pressure + Queue Expiration (which would follow the rules Mark
> described).
>
> As for the order, I'm not confident I remember how it works right now.
>
> Thanks,
>   Peter
>
>
> -----Original Message-----
> From: Mark Bean <mark.o.bean@gmail.com>
> Sent: Wednesday, July 24, 2019 7:08 AM
> To: dev@nifi.apache.org
> Subject: [EXT] Re: FlowFile Expiration - Lineage vs Queue Times
>
> On a similar note, we recently had a case where it would be desirable for
> the flowfile expiration to kick in only after the flowfile size/count back
> pressure limits have been reached. In other words, once a back pressure
> (size) limit is reached, it would be desirable to then remove flowfiles -
> beginning with the oldest first - until the back pressure limit is no
> longer in violation.
>
> Thanks,
> Mark
>
> On Tue, Jul 23, 2019 at 3:41 PM Peter Wicks (pwicks) <pwicks@micron.com>
> wrote:
>
> > I was thinking it would be nice to expire FlowFile's based on their
> > time in queue, in addition to the current option of their total
> > lineage time (as in, have both options available).
> > Any thoughts on pros/cons of having this available?
> >
> > Thanks,
> >   Peter
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message