From dev-return-19896-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Fri Oct 11 05:12:50 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id AF8C5180626 for ; Fri, 11 Oct 2019 07:12:49 +0200 (CEST) Received: (qmail 35116 invoked by uid 500); 11 Oct 2019 05:12:46 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 34849 invoked by uid 99); 11 Oct 2019 05:12:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Oct 2019 05:12:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8D4A9C2270; Fri, 11 Oct 2019 05:12:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.2 X-Spam-Level: X-Spam-Status: No, score=-0.2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id vAtvFWbZySop; Fri, 11 Oct 2019 05:12:41 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.222.180; helo=mail-qk1-f180.google.com; envelope-from=ijokarumawak@gmail.com; receiver= Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 94D1BBC5BC; Fri, 11 Oct 2019 05:12:41 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id u184so7815275qkd.4; Thu, 10 Oct 2019 22:12:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lgTVqqHQmRyrDtQqvIioG4IU0uBExzn8uDKMrNCBC+o=; b=re0E2Hg/txPjv7j59Cs9/3N4nzryBUHYw+kOhhz2RIkc5V/nalCKFbkYgx6emvSm4y K81l2JAEH/KnYbuFM/R7avWQsLAujvpOXzNS/l18Z4hCG7liaoZ2Bnpk4STBbAhsSS8E +6+RUVaObJJLu4SlpDJ5MsvVh5lgBB9ZgJOLc2U8YKNL82YD5gqTZhYOP/rzPWEgNk/3 9gi3kdN1WrckywR48pj2WZAjf5ppnS38OsbwhKGZ223ZEyWUc/e6jhCrV26sAdXZh6CW 1BmkdeZE4CQBWhEa5JO9fPFWIg2mYIwSR18r246QXam0iDvOKHEtHY+zbGeGK8rVbX3c v1aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lgTVqqHQmRyrDtQqvIioG4IU0uBExzn8uDKMrNCBC+o=; b=k3YPAp7NFrF20FVd23l9P/0qeiNxumpgNkTsZL0kvHDgOcnI88/jQ3/ZNCzPTRzfcx MNq3U118C4Oh4wlFbL38gpb4j3SbAl8vo5dpGHGwb7cljn1D48st1bQW3FhSQbhsltP7 CarK374KsztDA8vKhQbFdZeuJqv8pbENLkOd39xtj03y4l4ecSVpyGHpmIPZNFPbsSzO TiIEQ1Xjg6Ul/fuVTFDnrALOwu7cK+pHQDt9VBbVZg5QhqcSf/Ebc+KQAGUApMOBfd+f FBpT9xL2CJeJmb/tjHEGwRYNEemXIQvwMPFZ8iyJXsrsvAbxUBXi9PjhDKszlXzI/Z87 7THA== X-Gm-Message-State: APjAAAWu9QyUWfOO9/ZIPxcivmNeAOxJU6vdMYPaVj8n5NJU2amTvkhX BwipBwyeXBi1OIXFGZuzB3o0HfSux4LrDCtB2Z2/1BOD X-Google-Smtp-Source: APXvYqysvrYJC8aNqBGcg0TVYnvy2VoP4SBqQvCn8q81SrmHCida3gGqCGYcNrYUovcDmMLDiQ9fMkC6hV9jUWLQfH4= X-Received: by 2002:a37:67d4:: with SMTP id b203mr7308750qkc.435.1570770760993; Thu, 10 Oct 2019 22:12:40 -0700 (PDT) MIME-Version: 1.0 References: <201910091358390700938@geekplus.com.cn> In-Reply-To: <201910091358390700938@geekplus.com.cn> From: Koji Kawamura Date: Fri, 11 Oct 2019 14:12:05 +0900 Message-ID: Subject: Re: Can CaptureChangeMySQL be scheduled to all nodes instead of primary node? To: users@nifi.apache.org Cc: dev Content-Type: text/plain; charset="UTF-8" Hi Lei, I don't know any NiFi built-in feature to achieve that. To distribute CaptureChangeMySQL load among nodes, I'd deploy separate standalone NiFi (or even MiNiFi Java) in addition to the main NiFi cluster for the main data flow. For example, if there are 5 databases and 3 NiFi nodes, deploy a 3 node NiFi cluster with an InputPort. And also, run standalone NiFi/MiNiFi processes on each node, too, node-a (datasource 1 and 2), node-b (datasource 3 and 4), node-c (datasource 5) then use RemoteProcessGroup to send captured data to the main NiFi cluster. This approach may be harder to maintain, but feasible. Thanks, Koji On Wed, Oct 9, 2019 at 3:06 PM wanglei2@geekplus.com.cn wrote: > > I am using CaptureChangeMySQL to replicate the database. > There are many data sources and so there're many CaptureChangeMySQL processors. > The CaptureChangeMySQL throws same slave id error if scheduled on all nodes. So it can only be scheduled on primary node. This causes very heavy load on the primary node. > > Is there any method than i can distribute the CaptureChangeMySQL processors to all nodes instead of only to primary node? > > Thanks, > Lei > > ________________________________ > wanglei2@geekplus.com.cn