nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Does NiFi support Hot Deploy?
Date Tue, 16 Feb 2016 02:45:18 GMT
Jeff

Clustered NiFis (multiple nodes) is quite common.  You can have
hundreds of processors representing a quite large number of distinct
flows on there.  Templates are the mechanism by which flows can be
shared among clusters and between dev/prod environments.  We have some
important issues to tackle to make templates as powerful as they
should be such as having environment variable mapping and
consistent/repeatable serialization which lends to better version
control of templates.

Thanks
Joe

On Mon, Feb 15, 2016 at 9:39 PM, Jeff - Data Bean Australia
<databean.au@gmail.com> wrote:
> Thanks Joe for the clarification. And it does make sense to me with runtime
> reliability and change requirement in enterprise environment.
>
> I know that there are solutions that put multiple NiFi instances working
> together. How common a solution like this in real world? Should we,
> generally speaking, prefer simple flow.xml configuration and combine NiFi
> instances together for more complex scenarios?
>
> On Tue, Feb 16, 2016 at 3:16 PM, Joe Witt <joe.witt@gmail.com> wrote:
>>
>> Jeff,
>>
>> Regarding hot deploy of new dataflows:
>> Absolutely.  The ability to have interactive command and control is a
>> key feature/concept of NiFi so you can certainly create, alter, remove
>> dataflows while the systems is running by design.  This isn't just for
>> a person controlling the flow through  UI but also for external
>> systems to have automated interactions with NiFi through its REST API
>> which can do things like change the flow, alter priorities, etc..
>>
>> Regarding hot deploy of new code/extensions:
>> We've avoided live deploy of new extensions to this point.  Largely
>> due to the understanding that while adding new extensions at runtime
>> is pretty doable it is less reliable/clear to change/update.  That
>> said, we're trending toward this registry and it would back versions
>> of extensions at which point maybe this becomes more reasonable.
>>
>> Anyway, in the mean time one option may be the fact that the
>> ExecuteScript,InvokeScriptedProcessor processors do support live
>> alteration of the code behind them.  While clearly not a complete
>> solution this may help with some of your cases.
>>
>> Thanks
>> Joe
>>
>>
>>
>> On Mon, Feb 15, 2016 at 9:10 PM, Jeff - Data Bean Australia
>> <databean.au@gmail.com> wrote:
>> > Here is my use case: I have a quite complex Data Flow system implemented
>> > by
>> > NiFi and keep streaming data to another system. There is part of the
>> > system
>> > I might want to update on-the-fly without turning down the whole NiFi
>> > platform. And occasionally I might want to add a new processor, or
>> > upgrade
>> > an existing one, then bring up this processor right away.
>> >
>> > Can I do it in NiFi now?
>> >
>> > Thanks,
>> > Jeff
>> >
>> > --
>> > Data Bean - A Big Data Solution Provider in Australia.
>
>
>
>
> --
> Data Bean - A Big Data Solution Provider in Australia.

Mime
View raw message