Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D7CEA200BCF for ; Mon, 5 Dec 2016 10:05:42 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D65FB160B17; Mon, 5 Dec 2016 09:05:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 07E41160AF9 for ; Mon, 5 Dec 2016 10:05:41 +0100 (CET) Received: (qmail 22527 invoked by uid 500); 5 Dec 2016 09:05:41 -0000 Mailing-List: contact users-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@nifi.apache.org Delivered-To: mailing list users@nifi.apache.org Received: (qmail 22517 invoked by uid 99); 5 Dec 2016 09:05:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Dec 2016 09:05:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id AD43E1803A7 for ; Mon, 5 Dec 2016 09:05:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1 X-Spam-Level: X-Spam-Status: No, score=-1 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RP_MATCHES_RCVD=-2.999, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id n-pmuj-WDAwg for ; Mon, 5 Dec 2016 09:05:38 +0000 (UTC) Received: from exchange.systel-sa.com (exchange.systel-sa.com [37.58.143.234]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 434905FDAC for ; Mon, 5 Dec 2016 08:56:24 +0000 (UTC) Received: from SIGMA.systel.local (192.17.1.42) by exchange.systel-sa.com (10.0.0.7) with Microsoft SMTP Server (TLS) id 8.3.264.0; Mon, 5 Dec 2016 09:55:35 +0100 Received: from [192.168.5.52] (192.168.5.52) by SIGMA.systel.local (192.17.1.42) with Microsoft SMTP Server id 8.3.264.0; Mon, 5 Dec 2016 09:55:35 +0100 From: De Mezzo Benoit Subject: Re: multiple flows and cluster To: "users@nifi.apache.org" References: <9c49dfdd-ea91-e766-ffaf-5f8a89d0fec2@systel-sa.com> Message-ID: Date: Mon, 5 Dec 2016 09:55:34 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------65E16F82E99C4C978F1637B3" archived-at: Mon, 05 Dec 2016 09:05:43 -0000 --------------65E16F82E99C4C978F1637B3 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Thanks! Sorry for the delay! What do you mean by creating microclusters? Is this a Nifi functionnality (ie. hierarchical cluster of clusters) ? Or is it a cluster design that must be created thoroughly? Benoit. Le 26/11/2016 à 02:13, Andy LoPresto a écrit : > Benoit, > > Every NiFi node can run many disparate flows, and you can separate > them into distinct process groups for logical divisions. > > You can trigger these flows in a few ways. The first, you could set > the first processor in the flow to be triggered by event receipt > rather than schedule driven so that it doesn't run until a unit of > data enters the flow. The other thing you could do is use the REST API > to enable/disable individual processors/flows and trigger that API > invocation via cron or some other scheduling system external to NiFi. > Finally, you could use an ExecuteScript processor to perform more > complex custom logic to determine which flows should be running at any > given time (and then use the REST API to enable/disable them). > > If you really find that these changes cannot be synchronized across > the single flow residing on a NiFi cluster, you could segment and > subdivide your cluster into microclusters where each shares a partial > flow (grouped by performance considerations or logical concepts). > > Andy LoPresto > alopresto@apache.org > alopresto.apache@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 25, 2016, at 04:13, De Mezzo Benoit > wrote: > >> Hi all, >> >> Here is my problem: I need to run many small etl flow (say a thousand) >> at flow-dedicated times. As theses flows may be added, removed, changed >> all the time, they can not be easily merged within a big-meta flow to be >> run in the Nifi cluster. >> >> I would like to know if there a way to run multiple flows within a nifi >> cluster? >> >> Or may be there is a way to generate, per flow, a small standalone >> binary (jar?) to run only once a flow against an embedded Nifi engine >> (each jar will be managed by a task scheduling cluster) ? >> >> Thanks ! >> >> Benoit. >> --------------65E16F82E99C4C978F1637B3 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 8bit
Thanks! Sorry for the delay!

What do you mean by creating microclusters? Is this a Nifi functionnality (ie. hierarchical cluster of clusters) ? Or is it a cluster design that must be created thoroughly?

Benoit.



Le 26/11/2016 à 02:13, Andy LoPresto a écrit :
Benoit,

Every NiFi node can run many disparate flows, and you can separate them into distinct process groups for logical divisions. 

You can trigger these flows in a few ways. The first, you could set the first processor in the flow to be triggered by event receipt rather than schedule driven so that it doesn't run until a unit of data enters the flow. The other thing you could do is use the REST API to enable/disable individual processors/flows and trigger that API invocation via cron or some other scheduling system external to NiFi. Finally, you could use an ExecuteScript processor to perform more complex custom logic to determine which flows should be running at any given time (and then use the REST API to enable/disable them).

If you really find that these changes cannot be synchronized across the single flow residing on a NiFi cluster, you could segment and subdivide your cluster into microclusters where each shares a partial flow (grouped by performance considerations or logical concepts). 

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 25, 2016, at 04:13, De Mezzo Benoit <b.demezzo@systel-sa.com> wrote:

Hi all,

Here is my problem: I need to run many small etl flow (say a thousand)
at flow-dedicated times. As theses flows may be added, removed, changed
all the time, they can not be easily merged within a big-meta flow to be
run in the Nifi cluster.

I would like to know if there a way to run multiple flows within a nifi
cluster?

Or may be there is a way to generate, per flow, a small standalone
binary (jar?) to run only once a flow against an embedded Nifi engine
(each jar will be managed by a task scheduling cluster) ?

Thanks !

Benoit.


--------------65E16F82E99C4C978F1637B3--