nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: Process Group singleton
Date Tue, 27 Dec 2016 16:02:38 GMT
Hi Yari,

I think your assumptions may be a little off.  You can use input/output
ports to help traverse the nested levels of PGs.  To help illustrate, I
created a template [1] you can evaluate as to one way this could be managed
for your scenario.  As noted, the number of connections can become
unwieldy, so organization is key with large numbers of groups making use of
a central PG of functionality! :)

Please let us know if that clarifies or if there are any additional
questions.


[1] https://gist.github.com/apiri/69b40d46d29a15641962dae649093298

Thanks!
--aldrin


On Tue, Dec 27, 2016 at 10:40 AM, Yari Marchetti <
yari.marchetti@buongiorno.com> wrote:

> Hi Aldrin,
> the problem I see with your suggestion, at least in my specific case, it's
> that it requires to have all the referencing groups (in your example A,B)
> inside the same process group (otherwise you cannot link them, right?) but
> that's not the case for us because we've many of them so they're organised
> in a hierarchy of PG for better manageability and control (Please let me
> know if my assumptions are correct, because I may be totally off :) That's
> why I was looking for something that could be referenced from anywhere
> (like a RPG).
>
> Yari
>
> On 27 December 2016 at 16:14, Aldrin Piri <aldrinpiri@gmail.com> wrote:
>
>> Hi Yari,
>>
>> The concept you are looking for has been discussed a bit and a draft
>> feature proposal was created for referenceable process groups [1], although
>> is a common pattern and implementable with the current functionality
>> provided; common cases of this are standard tagging or enrichment sets of
>> functionality.
>>
>> What you want to attempt seems like you wouldn't necessarily need to do
>> an RPG (unless you were looking to cut down on connection clutter), but
>> could perform routing to a single process group via attributes and tagging
>> and then using a fan out to send data back to its respective group.
>>
>> Flow of data would be something such as:
>>
>> * Source Group {A, B, ...}
>> * Update Attribute
>>  - Tag Data as Source Group {A, B, ...} (source)
>> * Singleton Process group input port for common processing
>> * Route on Attribute (source)
>>  - Create relationships for each of the groups that are transmitting data
>> to this common block
>> * Source Group
>>
>>
>> [1] https://cwiki.apache.org/confluence/display/NIFI/Referen
>> ce-able+Process+Groups
>>
>> On Tue, Dec 27, 2016 at 9:49 AM, Yari Marchetti <
>> yari.marchetti@buongiorno.com> wrote:
>>
>>> Hello,
>>> we're actively using Nifi on production but there's a recurring problem
>>> popping out every now and then: maintenance of process groups created from
>>> templates, as we're spending quite some time looking for all the instances
>>> of a specific template and updating them as required.
>>>
>>> What we would like to do it's something like a singleton instance of a
>>> process group that could just be referenced by all other the flows in Nifi
>>> without having different instances of a template. Is there any way to
>>> achieve this? The only thing that came to my mind is using a RPG to the
>>> local cluster (which it's made up of 3 servers), don't know though if it
>>> would work...
>>>
>>> Thanks,
>>> Yari
>>>
>>
>>
>

Mime
View raw message