nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Midhun Mohan <midhun.mo...@esginc.us>
Subject Re: Queue Prioritization
Date Mon, 02 Mar 2020 12:35:40 GMT
Hi Eduardo,
Can you please send the Image  again, I am not able to open it

On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes <eduardo.fontes@gmail.com>
wrote:

> Hi Midhun,
>
> I think this flow will do what you want.
>
> [image: image.png]
>
> Be advised that use of Wait/Notify will slow down the average throughput
> of your flow because the aditional steps with the Cache.
>
>
>
>
> On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <midhun.mohan@esginc.us>
> wrote:
>
>> So I would like to know your suggestions on how to implement wait and
>> notify.
>>
>> In my case messages will flow one after another
>>
>> If a failure occurs I need to retry the failure before the orginal ones,
>> and notify users if it fails again.
>>
>>
>> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <emanueol@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > Invokehttp have 4 exits
>> > - success
>> > - error retry -> loop/count retries (updateattribite) -> if retries >
>> max
>> > then logmessage(error) flow stops here ff discarded  Otherwise loop
>> back to
>> > invokehttp.
>> > - error type 2 (bad password etc, )
>> > - error type 3 (connectivity problems) you may want also to loop this
>> one
>> > with some delay , count retries and exit /log/stop after max.
>> >
>> > I just highly advise prior to invokehttp to have an updateattribute
>> > deleting all attributes that don't start with some prefix.. to be sure
>> your
>> > ff looping doesnt keep adfing different errors, attributes wriiten by
>> > invoke http.
>> >
>> > As good practice we prefix name all control attributes with
>> flow_xxxxxxx so
>> > our "cleanup," Updateattribute before Invokehttp have special regex
>> sayng
>> > "delete all attributes that don't start with flow_
>> > (If you interested tomorrow when im back in office i csn share the
>> regex a
>> > bit tricky).
>> >
>> > Hope this helps, i just would love all processors to be as good as
>> > invokehttp providing hard multipke exit points for specifics + all the
>> nice
>> > write attributes invokehttp adds that ptovide high details on errors or
>> > success (atttibutes created/added by invokehttp to ff are prefixed with
>> > invoke.xxxxx
>> >
>> > Cheers,
>> > Emanuel O.
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <midhun.mohan@esginc.us> wrote:
>> >
>> > > Hi my use case is
>> > >
>> > > When a invokehttp processor fails I need to stop the incoming message
>> and
>> > > retry the failed message 2 times and if it is success start the
>> execution
>> > > normally or if it fails again need to notify admins and stop the main
>> > > invoke http processor.
>> > >
>> > >
>> > >
>> > > Hope you can get the scenario from this
>> > >
>> > > On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <eduardo.fontes@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Hi Midhun,
>> > > >
>> > > > You can use DistributedMapCacheServer which starts a cache service
>> > within
>> > > > NiFi's process, but I not recommend in production. Instead you could
>> > use
>> > > > RedisConnectionPoolService.
>> > > >
>> > > > To use solution Wait/Notify the NiFi must know the number of
>> flowfiles
>> > to
>> > > > process in one queue before start another queue. If it is a batch/on
>> > > demand
>> > > > flow I think it's easier. If it is a continuos/streamming flow you
>> must
>> > > > define a window, by number of flowfiles or by time, to switch
>> between
>> > > > queues.
>> > > >
>> > > > Please let me know what is your case.
>> > > >
>> > > > On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
>> midhun.mohan@esginc.us>
>> > > > wrote:
>> > > >
>> > > > > I think wait and notify can solve my problem. I checked the
>> > documents ,
>> > > > > which is bit unclear to me. Can anyone help me?
>> > > > >
>> > > > > Just to know few things like whether we need to configure a cache
>> > > server,
>> > > > > if so which one is best
>> > > > > And also getting confused on how to configure wait and notify
>> signals
>> > > > >
>> > > > > On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
>> > > eduardo.fontes@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi! Maybe you should try another approach, like Wait/Notify
or
>> > > > configure
>> > > > > > prioritizer just before the funnel, depending on your demand.
>> > > > > >
>> > > > > > On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
>> > > midhun.mohan@esginc.us>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > > How to prioritize one queue before a funnel?
>> > > > > > >
>> > > > > > >  I need to make sure one queue is emptied before another
>> queue is
>> > > > taken
>> > > > > > > over for executing, I tried with a priority attribute
but it
>> > fails
>> > > > > when a
>> > > > > > > queue with less priority has messages before the queue
with
>> high
>> > > > > > priority.
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Midhun Mohan
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

-- 


Regards,
Midhun Mohan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message