nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Endre Kovács <ekov...@cloudera.com.INVALID>
Subject Re: Queue Prioritization
Date Mon, 02 Mar 2020 12:37:11 GMT
Please upload it to somewhere and link it here. ASF mailing lists are striping the attachments/images.

> On 2 Mar 2020, at 13:35, Midhun Mohan <midhun.mohan@esginc.us> wrote:
> 
> Hi Eduardo,
> Can you please send the Image  again, I am not able to open it
> 
> On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes <eduardo.fontes@gmail.com <mailto:eduardo.fontes@gmail.com>>
> wrote:
> 
>> Hi Midhun,
>> 
>> I think this flow will do what you want.
>> 
>> [image: image.png]
>> 
>> Be advised that use of Wait/Notify will slow down the average throughput
>> of your flow because the aditional steps with the Cache.
>> 
>> 
>> 
>> 
>> On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan <midhun.mohan@esginc.us>
>> wrote:
>> 
>>> So I would like to know your suggestions on how to implement wait and
>>> notify.
>>> 
>>> In my case messages will flow one after another
>>> 
>>> If a failure occurs I need to retry the failure before the orginal ones,
>>> and notify users if it fails again.
>>> 
>>> 
>>> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira, <emanueol@gmail.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> Invokehttp have 4 exits
>>>> - success
>>>> - error retry -> loop/count retries (updateattribite) -> if retries
>
>>> max
>>>> then logmessage(error) flow stops here ff discarded  Otherwise loop
>>> back to
>>>> invokehttp.
>>>> - error type 2 (bad password etc, )
>>>> - error type 3 (connectivity problems) you may want also to loop this
>>> one
>>>> with some delay , count retries and exit /log/stop after max.
>>>> 
>>>> I just highly advise prior to invokehttp to have an updateattribute
>>>> deleting all attributes that don't start with some prefix.. to be sure
>>> your
>>>> ff looping doesnt keep adfing different errors, attributes wriiten by
>>>> invoke http.
>>>> 
>>>> As good practice we prefix name all control attributes with
>>> flow_xxxxxxx so
>>>> our "cleanup," Updateattribute before Invokehttp have special regex
>>> sayng
>>>> "delete all attributes that don't start with flow_
>>>> (If you interested tomorrow when im back in office i csn share the
>>> regex a
>>>> bit tricky).
>>>> 
>>>> Hope this helps, i just would love all processors to be as good as
>>>> invokehttp providing hard multipke exit points for specifics + all the
>>> nice
>>>> write attributes invokehttp adds that ptovide high details on errors or
>>>> success (atttibutes created/added by invokehttp to ff are prefixed with
>>>> invoke.xxxxx
>>>> 
>>>> Cheers,
>>>> Emanuel O.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Sun, 1 Mar 2020, 12:42 Midhun Mohan, <midhun.mohan@esginc.us> wrote:
>>>> 
>>>>> Hi my use case is
>>>>> 
>>>>> When a invokehttp processor fails I need to stop the incoming message
>>> and
>>>>> retry the failed message 2 times and if it is success start the
>>> execution
>>>>> normally or if it fails again need to notify admins and stop the main
>>>>> invoke http processor.
>>>>> 
>>>>> 
>>>>> 
>>>>> Hope you can get the scenario from this
>>>>> 
>>>>> On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, <eduardo.fontes@gmail.com
>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Hi Midhun,
>>>>>> 
>>>>>> You can use DistributedMapCacheServer which starts a cache service
>>>> within
>>>>>> NiFi's process, but I not recommend in production. Instead you could
>>>> use
>>>>>> RedisConnectionPoolService.
>>>>>> 
>>>>>> To use solution Wait/Notify the NiFi must know the number of
>>> flowfiles
>>>> to
>>>>>> process in one queue before start another queue. If it is a batch/on
>>>>> demand
>>>>>> flow I think it's easier. If it is a continuos/streamming flow you
>>> must
>>>>>> define a window, by number of flowfiles or by time, to switch
>>> between
>>>>>> queues.
>>>>>> 
>>>>>> Please let me know what is your case.
>>>>>> 
>>>>>> On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
>>> midhun.mohan@esginc.us>
>>>>>> wrote:
>>>>>> 
>>>>>>> I think wait and notify can solve my problem. I checked the
>>>> documents ,
>>>>>>> which is bit unclear to me. Can anyone help me?
>>>>>>> 
>>>>>>> Just to know few things like whether we need to configure a cache
>>>>> server,
>>>>>>> if so which one is best
>>>>>>> And also getting confused on how to configure wait and notify
>>> signals
>>>>>>> 
>>>>>>> On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
>>>>> eduardo.fontes@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi! Maybe you should try another approach, like Wait/Notify
or
>>>>>> configure
>>>>>>>> prioritizer just before the funnel, depending on your demand.
>>>>>>>> 
>>>>>>>> On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
>>>>> midhun.mohan@esginc.us>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> How to prioritize one queue before a funnel?
>>>>>>>>> 
>>>>>>>>> I need to make sure one queue is emptied before another
>>> queue is
>>>>>> taken
>>>>>>>>> over for executing, I tried with a priority attribute
but it
>>>> fails
>>>>>>> when a
>>>>>>>>> queue with less priority has messages before the queue
with
>>> high
>>>>>>>> priority.
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Midhun Mohan
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> -- 
> 
> 
> Regards,
> Midhun Mohan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message