Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CAEFA200BEA for ; Tue, 27 Dec 2016 17:03:29 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C99FF160B31; Tue, 27 Dec 2016 16:03:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C5287160B23 for ; Tue, 27 Dec 2016 17:03:28 +0100 (CET) Received: (qmail 50409 invoked by uid 500); 27 Dec 2016 16:03:27 -0000 Mailing-List: contact users-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@nifi.apache.org Delivered-To: mailing list users@nifi.apache.org Received: (qmail 50398 invoked by uid 99); 27 Dec 2016 16:03:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Dec 2016 16:03:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6D65A1A02DB for ; Tue, 27 Dec 2016 16:03:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id xDhPDL0XIB-z for ; Tue, 27 Dec 2016 16:03:26 +0000 (UTC) Received: from mail-yw0-f174.google.com (mail-yw0-f174.google.com [209.85.161.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id EB7A65F1BA for ; Tue, 27 Dec 2016 16:03:25 +0000 (UTC) Received: by mail-yw0-f174.google.com with SMTP id v81so69293070ywb.2 for ; Tue, 27 Dec 2016 08:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=6DUy0i4RsnNPD+6FuuGpBeTufUeF9Bj4ZS+v2bzdOXI=; b=eZY41VozN0xYt/ELjE4HYr0M+SCqzHfJdmm4on41cUArKz4opDN4e8ERwi/oOFfJ42 97/fQOk8uXvrMhhUVvtBrcx6IwryGSkda3+hDZnfsy8aolp/hb/b4POupdotzxqOIrqH e52vtl3UGqvh4mNNAsdf7Ga6Z71HZ6lUPA3IXSE/KfqCtiYkLZ5oxxcYWpGcI7O/wicC zEzIBDzLtoPM3XQzAZU+4HQpB8zK+7bD9X2qU7ddAJqwLTb2XyZz8+YBGFyIc9GpbuBB +xnOBxyRZdPyThGTO4amJy+3TZTTpOd5kdwR5wNdYi6LghLwVF+uxpFlacf9fCn2Ktw4 fGUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=6DUy0i4RsnNPD+6FuuGpBeTufUeF9Bj4ZS+v2bzdOXI=; b=HC6aMOPITGyp0gJ8MC4GO8uZAiat+lEcBkv/RchSQINo71zexCLHFb4+VM3kFaKx/V OtM56Hhma8VNfBZkqsHPkBrHiM9sLQtO+2PsFqAJOhff6TfLuChgm55XaNECswb22235 QNS4RTLD/BXbDWTbfDTnVoDtzXaVfc6iEUd8e7zyO5zLHPi7TimlIEsygKy6wSOTKKOw FGkh9uEyLlQBTgrhpfq6HltppP9j+Pjwt59hk0wx3e8vGOerYSdSq60y5VBCeOvxwgxr 9Se/WpNTqRKqpdgDzPGgbCoWke0YvRVsmvGEgab6mVamrAIm9z/3vKdiFv1EdlL8D/jF IwpQ== X-Gm-Message-State: AIkVDXIJai4JWfHunjIwVx81VSXCdHLxSWK8wvv7r0xoUPtLY/7jdYlX1o89gw6K8Mc3PRYRFWt16uxsYZdxcg== X-Received: by 10.129.128.3 with SMTP id q3mr22062634ywf.27.1482854598749; Tue, 27 Dec 2016 08:03:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.161.74 with HTTP; Tue, 27 Dec 2016 08:02:38 -0800 (PST) In-Reply-To: References: From: Aldrin Piri Date: Tue, 27 Dec 2016 11:02:38 -0500 Message-ID: Subject: Re: Process Group singleton To: users@nifi.apache.org Content-Type: multipart/alternative; boundary=94eb2c030ec49ccd300544a5fdd2 archived-at: Tue, 27 Dec 2016 16:03:30 -0000 --94eb2c030ec49ccd300544a5fdd2 Content-Type: text/plain; charset=UTF-8 Hi Yari, I think your assumptions may be a little off. You can use input/output ports to help traverse the nested levels of PGs. To help illustrate, I created a template [1] you can evaluate as to one way this could be managed for your scenario. As noted, the number of connections can become unwieldy, so organization is key with large numbers of groups making use of a central PG of functionality! :) Please let us know if that clarifies or if there are any additional questions. [1] https://gist.github.com/apiri/69b40d46d29a15641962dae649093298 Thanks! --aldrin On Tue, Dec 27, 2016 at 10:40 AM, Yari Marchetti < yari.marchetti@buongiorno.com> wrote: > Hi Aldrin, > the problem I see with your suggestion, at least in my specific case, it's > that it requires to have all the referencing groups (in your example A,B) > inside the same process group (otherwise you cannot link them, right?) but > that's not the case for us because we've many of them so they're organised > in a hierarchy of PG for better manageability and control (Please let me > know if my assumptions are correct, because I may be totally off :) That's > why I was looking for something that could be referenced from anywhere > (like a RPG). > > Yari > > On 27 December 2016 at 16:14, Aldrin Piri wrote: > >> Hi Yari, >> >> The concept you are looking for has been discussed a bit and a draft >> feature proposal was created for referenceable process groups [1], although >> is a common pattern and implementable with the current functionality >> provided; common cases of this are standard tagging or enrichment sets of >> functionality. >> >> What you want to attempt seems like you wouldn't necessarily need to do >> an RPG (unless you were looking to cut down on connection clutter), but >> could perform routing to a single process group via attributes and tagging >> and then using a fan out to send data back to its respective group. >> >> Flow of data would be something such as: >> >> * Source Group {A, B, ...} >> * Update Attribute >> - Tag Data as Source Group {A, B, ...} (source) >> * Singleton Process group input port for common processing >> * Route on Attribute (source) >> - Create relationships for each of the groups that are transmitting data >> to this common block >> * Source Group >> >> >> [1] https://cwiki.apache.org/confluence/display/NIFI/Referen >> ce-able+Process+Groups >> >> On Tue, Dec 27, 2016 at 9:49 AM, Yari Marchetti < >> yari.marchetti@buongiorno.com> wrote: >> >>> Hello, >>> we're actively using Nifi on production but there's a recurring problem >>> popping out every now and then: maintenance of process groups created from >>> templates, as we're spending quite some time looking for all the instances >>> of a specific template and updating them as required. >>> >>> What we would like to do it's something like a singleton instance of a >>> process group that could just be referenced by all other the flows in Nifi >>> without having different instances of a template. Is there any way to >>> achieve this? The only thing that came to my mind is using a RPG to the >>> local cluster (which it's made up of 3 servers), don't know though if it >>> would work... >>> >>> Thanks, >>> Yari >>> >> >> > --94eb2c030ec49ccd300544a5fdd2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Yari,

I think your assumptions may b= e a little off.=C2=A0 You can use input/output ports to help traverse the n= ested levels of PGs.=C2=A0 To help illustrate, I created a template [1] you= can evaluate as to one way this could be managed for your scenario.=C2=A0 = As noted, the number of connections can become unwieldy, so organization is= key with large numbers of groups making use of a central PG of functionali= ty! :)

Please let us know if that clarifies or if = there are any additional questions.


Thanks!
=
--aldrin


On Tue, Dec 27, 2016 at 10:40 AM, Yari Marchetti <yari.marchetti@buongiorno.com> wrote:
Hi Aldrin,
the problem I see wi= th your suggestion, at least in my specific case, it's that it requires= to have all the referencing groups (in your example A,B) inside the same p= rocess group (otherwise you cannot link them, right?) but that's not th= e case for us because we've many of them so they're organised in a = hierarchy of PG for better manageability and control (Please let me know if= my assumptions are correct, because I may be totally off :) That's why= I was looking for something that could be referenced from anywhere (like a= RPG).=C2=A0

<= /div>
Yari

On 27 December 2016 at 16:14, Aldri= n Piri <aldrinpiri@gmail.com> wrote:
Hi Yari,

The concept you = are looking for has been discussed a bit and a draft feature proposal was c= reated for referenceable process groups [1], although is a common pattern a= nd implementable with the current functionality provided; common cases of t= his are standard tagging or enrichment sets of functionality.
What you want to attempt seems like you wouldn't necessaril= y need to do an RPG (unless you were looking to cut down on connection clut= ter), but could perform routing to a single process group via attributes an= d tagging and then using a fan out to send data back to its respective grou= p.

Flow of data would be something such as:
<= div>
* Source Group {A, B, ...}
* Update Attribute= =C2=A0
=C2=A0- Tag Data as Source Group {A, B, ...} (source)
* Singleton Process group input port for common processing
= * Route on Attribute (source)
=C2=A0- Create relationships for ea= ch of the groups that are transmitting data to this common block =C2=A0=C2= =A0
* Source Group



On Tue, Dec 27, 2016 at 9:49 AM, Yari Marchetti <yari.marchetti@buongiorno.com> wrote:
H= ello,
we're actively using Nifi o= n production but there's a recurring problem popping out every now and = then: maintenance of process groups created from templates, as we're sp= ending quite some time looking for all the instances of a specific template= and updating them as required.=C2=A0
=
What we would like to do it's= something like a singleton instance of a process group that could just be = referenced by all other the flows in Nifi without having different instance= s of a template. Is there any way to achieve this? The only thing that came= to my mind is using a RPG to the local cluster (which it's made up of = 3 servers), don't know though if it would work...

Thanks,
Yari



--94eb2c030ec49ccd300544a5fdd2--