nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Midhun Mohan <midhun.mo...@esginc.us>
Subject Re: Queue Prioritization
Date Sun, 01 Mar 2020 15:57:02 GMT
So I would like to know your suggestions on how to implement wait and
notify.

In my case messages will flow one after another

If a failure occurs I need to retry the failure before the orginal ones,
and notify users if it fails again.


On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <emanueol@gmail.com> wrote:

> Hi,
>
> Invokehttp have 4 exits
> - success
> - error retry -> loop/count retries (updateattribite) -> if retries > max
> then logmessage(error) flow stops here ff discarded  Otherwise loop back to
> invokehttp.
> - error type 2 (bad password etc, )
> - error type 3 (connectivity problems) you may want also to loop this one
> with some delay , count retries and exit /log/stop after max.
>
> I just highly advise prior to invokehttp to have an updateattribute
> deleting all attributes that don't start with some prefix.. to be sure your
> ff looping doesnt keep adfing different errors, attributes wriiten by
> invoke http.
>
> As good practice we prefix name all control attributes with flow_xxxxxxx so
> our "cleanup," Updateattribute before Invokehttp have special regex sayng
> "delete all attributes that don't start with flow_
> (If you interested tomorrow when im back in office i csn share the regex a
> bit tricky).
>
> Hope this helps, i just would love all processors to be as good as
> invokehttp providing hard multipke exit points for specifics + all the nice
> write attributes invokehttp adds that ptovide high details on errors or
> success (atttibutes created/added by invokehttp to ff are prefixed with
> invoke.xxxxx
>
> Cheers,
> Emanuel O.
>
>
>
>
>
>
> On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <midhun.mohan@esginc.us> wrote:
>
> > Hi my use case is
> >
> > When a invokehttp processor fails I need to stop the incoming message and
> > retry the failed message 2 times and if it is success start the execution
> > normally or if it fails again need to notify admins and stop the main
> > invoke http processor.
> >
> >
> >
> > Hope you can get the scenario from this
> >
> > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <eduardo.fontes@gmail.com>
> > wrote:
> >
> > > Hi Midhun,
> > >
> > > You can use DistributedMapCacheServer which starts a cache service
> within
> > > NiFi's process, but I not recommend in production. Instead you could
> use
> > > RedisConnectionPoolService.
> > >
> > > To use solution Wait/Notify the NiFi must know the number of flowfiles
> to
> > > process in one queue before start another queue. If it is a batch/on
> > demand
> > > flow I think it's easier. If it is a continuos/streamming flow you must
> > > define a window, by number of flowfiles or by time, to switch between
> > > queues.
> > >
> > > Please let me know what is your case.
> > >
> > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <midhun.mohan@esginc.us>
> > > wrote:
> > >
> > > > I think wait and notify can solve my problem. I checked the
> documents ,
> > > > which is bit unclear to me. Can anyone help me?
> > > >
> > > > Just to know few things like whether we need to configure a cache
> > server,
> > > > if so which one is best
> > > > And also getting confused on how to configure wait and notify signals
> > > >
> > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> > eduardo.fontes@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi! Maybe you should try another approach, like Wait/Notify or
> > > configure
> > > > > prioritizer just before the funnel, depending on your demand.
> > > > >
> > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> > midhun.mohan@esginc.us>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > > How to prioritize one queue before a funnel?
> > > > > >
> > > > > >  I need to make sure one queue is emptied before another queue
is
> > > taken
> > > > > > over for executing, I tried with a priority attribute but it
> fails
> > > > when a
> > > > > > queue with less priority has messages before the queue with
high
> > > > > priority.
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Midhun Mohan
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message