nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Primary Only Content Migration
Date Thu, 07 Jun 2018 18:46:59 GMT
Peter,

There really shouldn't be any non-source processors scheduled for
primary node only. We may even want to consider preventing that option
when the processor has an incoming connection to avoid creating any
confusion.

As long as you set source processors to primary node only then
everything should be ok... if primary node changes, the source
processor starts executing on the new primary node, and any flow files
it already produced on the old primary node will continue to be worked
off by the downstream processors on the old node until they are all
processed.

-Bryan



On Thu, Jun 7, 2018 at 1:55 PM, Peter Wicks (pwicks) <pwicks@micron.com> wrote:
> I'm sure many of you have the same situation, a flow that runs on a cluster, and at some
point merges back down to a primary only processor; your files sit there in the queue with
nowhere to go... We've used the work around of having a remote processor group that loops
the data back to the primary node for a while, but would really like a clean/simple solution.
This approach requires that users be able to put an input port on the root flow, and then
route the file back down, which is a nuisance.
>
> I have been thinking of adding either a processor that moves data between specific nodes
in a cluster, or a queue (?) option that will let users migrate the content of a flowfile
back to the master node. This would allow you to move data back to a primary very easily without
needing RPG's and input ports at the root level.
>
> All of my development work with NiFi has been focused on processors, so I'm not really
sure where I would start with this.  Thoughts?
>
> Thanks,
>   Peter

Mime
View raw message