nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From De Mezzo Benoit <b.deme...@systel-sa.com>
Subject Re: multiple flows and cluster
Date Mon, 05 Dec 2016 08:55:34 GMT
Thanks! Sorry for the delay!

What do you mean by creating microclusters? Is this a Nifi
functionnality (ie. hierarchical cluster of clusters) ? Or is it a
cluster design that must be created thoroughly?

Benoit.



Le 26/11/2016 à 02:13, Andy LoPresto a écrit :
> Benoit,
>
> Every NiFi node can run many disparate flows, and you can separate
> them into distinct process groups for logical divisions. 
>
> You can trigger these flows in a few ways. The first, you could set
> the first processor in the flow to be triggered by event receipt
> rather than schedule driven so that it doesn't run until a unit of
> data enters the flow. The other thing you could do is use the REST API
> to enable/disable individual processors/flows and trigger that API
> invocation via cron or some other scheduling system external to NiFi.
> Finally, you could use an ExecuteScript processor to perform more
> complex custom logic to determine which flows should be running at any
> given time (and then use the REST API to enable/disable them).
>
> If you really find that these changes cannot be synchronized across
> the single flow residing on a NiFi cluster, you could segment and
> subdivide your cluster into microclusters where each shares a partial
> flow (grouped by performance considerations or logical concepts). 
>
> Andy LoPresto
> alopresto@apache.org <mailto:alopresto@apache.org>
> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 25, 2016, at 04:13, De Mezzo Benoit <b.demezzo@systel-sa.com
> <mailto:b.demezzo@systel-sa.com>> wrote:
>
>> Hi all,
>>
>> Here is my problem: I need to run many small etl flow (say a thousand)
>> at flow-dedicated times. As theses flows may be added, removed, changed
>> all the time, they can not be easily merged within a big-meta flow to be
>> run in the Nifi cluster.
>>
>> I would like to know if there a way to run multiple flows within a nifi
>> cluster?
>>
>> Or may be there is a way to generate, per flow, a small standalone
>> binary (jar?) to run only once a flow against an embedded Nifi engine
>> (each jar will be managed by a task scheduling cluster) ?
>>
>> Thanks !
>>
>> Benoit.
>>


Mime
View raw message