nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Wicks (pwicks)" <pwi...@micron.com>
Subject RE: Discuss: Process Group as a Process / "Single Threaded" Process Group
Date Thu, 16 Feb 2017 04:43:03 GMT
This is good stuff Koji, there is so much new stuff going into NiFi it's hard to keep up.

I'm a little disappointed to see that you have to use a Distributed Map Cache. This makes
sense for many scenarios, but I wonder what other options there are (of course with extra
coding), specifically for non-clustered environments.  Any ideas?

-----Original Message-----
From: Koji Kawamura [mailto:ijokarumawak@gmail.com] 
Sent: Wednesday, February 15, 2017 9:35 PM
To: dev@nifi.apache.org
Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process Group

Hi Peter, Joe

Recently, I've worked on NIFI-3452 "Add Wait processor Wait Mode property".
Using Wait/Notify combined with Back-pressure FlowFile count 1, it's possible to limit only
one FlowFile to be processed in a part of flow.
It's already merged into the master. Example is available here:
https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67
https://issues.apache.org/jira/browse/NIFI-3452

It may not be an ideal solution, but it will work for your use case.

Thanks,
Koji

On Thu, Feb 16, 2017 at 1:31 PM, Joe Witt <joe.witt@gmail.com> wrote:
> Peter
>
> We have a feature proposal on the nifi wiki for process groups as 
> functions.  I think we could update that to cover your case as well 
> https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Proces
> s+Groups with the notion of a single input to the 'function/group' at 
> a time.
> I guess it has a little different intent than you're needing though in 
> that you're not saying you need it to be referenceable like a function 
> but rather act more like a gate.  I wonder if the Wait/Notify 
> processors now on master would help your case.
>
> The request as asked is a bit outside the spirit of flow based 
> programming fundamentals that NiFi is built on so I'm hopeful we can 
> find another path that gets you the result you need.
>
> Thanks
> Joe
>
> On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) 
> <pwicks@micron.com> wrote:
>> I've been throwing around some ideas for how to migrate processes that would work
great in NiFi if you only processed one FlowFile at a time, but if there are multiple Flow
Files in the Flow processing through various Processes it would cause issues.
>>
>> One of the concepts I've been playing with is a Process Group that acts more like
a 1 threaded processor. One (and only one) FLowFile goes in to the Process Group, 0+ FlowFiles
come out (Output port would be optional, kind of like deciding whether or not to auto end
a relationship or continue it, but more manual); but no more than 1 FlowFile is ever pulled
in at a time.  The processor would have an input port, and so long as the Process Group has
1+ Flow Files in it, no new files will flow into the Process Group through the input port.
 I've been digging around the code, and it doesn't look to difficult to pull off.
>>
>>
>> -          New property on ProcessGroup, accompanying UI/validation
>>
>> o   Validation would probably need to verify there was 1 Input Port if enabled.
>>
>> -          Updated Logic in LocalPort's onTrigger code to check the parent container
and see if it has this setting enabled, and if it does how many total FlowFile's are currently
in queue. Only transfer in a new FlowFile if 0 flow files in queue.
>>
>>
>> Thoughts?
>>
>> One specific scenario I've come across is a database table that will be truncated
at the start of a flow, populated with a FlowFile's contents, updated by a second step to
remove duplicates, and then merged into the final table using a third step.  Of course if
I could use a separate table for each flow file this wouldn't be an issue, but I've run into
constraints where I'm limited to a single table.
>>
Mime
View raw message