nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: Can CaptureChangeMySQL be scheduled to all nodes instead of primary node?
Date Fri, 11 Oct 2019 05:12:05 GMT
Hi Lei,

I don't know any NiFi built-in feature to achieve that.
To distribute CaptureChangeMySQL load among nodes, I'd deploy separate
standalone NiFi (or even MiNiFi Java) in addition to the main NiFi
cluster for the main data flow.

For example, if there are 5 databases and 3 NiFi nodes, deploy a 3
node NiFi cluster with an InputPort.
And also, run standalone NiFi/MiNiFi processes on each node, too,
node-a (datasource 1 and 2), node-b (datasource 3 and 4), node-c
(datasource 5) then use RemoteProcessGroup to send captured data to
the main NiFi cluster.

This approach may be harder to maintain, but feasible.

Thanks,
Koji

On Wed, Oct 9, 2019 at 3:06 PM wanglei2@geekplus.com.cn
<wanglei2@geekplus.com.cn> wrote:
>
> I am using CaptureChangeMySQL to replicate the database.
> There are many data sources and so there're many  CaptureChangeMySQL processors.
> The CaptureChangeMySQL throws same slave id error  if scheduled on all nodes. So it can
only be scheduled on primary node. This causes  very heavy load on the primary node.
>
> Is there any method than i can  distribute the CaptureChangeMySQL processors to all nodes
instead of only to primary node?
>
> Thanks,
> Lei
>
> ________________________________
> wanglei2@geekplus.com.cn

Mime
View raw message