From dev-return-18312-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Wed Oct 17 23:14:33 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1270918061A for ; Wed, 17 Oct 2018 23:14:32 +0200 (CEST) Received: (qmail 97306 invoked by uid 500); 17 Oct 2018 21:14:32 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 97294 invoked by uid 99); 17 Oct 2018 21:14:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Oct 2018 21:14:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0D320C5E79 for ; Wed, 17 Oct 2018 21:14:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.9 X-Spam-Level: * X-Spam-Status: No, score=1.9 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id XeUdl4miVA5x for ; Wed, 17 Oct 2018 21:14:29 +0000 (UTC) Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 341085F125 for ; Wed, 17 Oct 2018 21:14:28 +0000 (UTC) Received: by mail-ot1-f52.google.com with SMTP id c23so25898066otl.9 for ; Wed, 17 Oct 2018 14:14:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=vs8LeRS3Q7pu2zyTXXEUYCl+6kv8/ZItOhbquU8/qQc=; b=hAuGbNDvZvgQXIhAorFFVL6BMpNA577VZFZXT1iEMfz8YKJOk9imlfzzOodG61tfSt FozNCM2EcH+JTnYD7vGbVEqHvOoSazU7Yfse+9XixeDKwz1l4j2tt1oZfbxGyrbEUH9e SSjvb45+D91PZNay//TW5+DB2hz2ogiRgqmpyIyY8WzDGOieqdGvFMiOnalLctlZi+Em KyXuknHmoc+voopRbNuUMyJNlAOYsSn91WVneG4CMsPVdQokyjxfw9rYKZzqGvPNrycM xkBSG8VYLHWnFfB4x7Ig0DBeATsM5oAJ1SB+bs29GLTQgSR7/uDsrt6wye4iq8D6/yg6 bKHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=vs8LeRS3Q7pu2zyTXXEUYCl+6kv8/ZItOhbquU8/qQc=; b=h7OPVKQ+LuCDCQ1mFV962ro6ihLoxmhOmn9e04O3lCN0ZS/iOU63um+zMyNOeEh/so 6AS0O6JKSsB835vstGQmRqZfD7Bef12XYG6TyHyeZfbZBJ6Sf0v7Kc392XzQVZ8H0gP3 yUuK2bA14eD9ZGWqHb0ymiTdyEcRmboyTDJK1Pg2KTqCPDe/9l13O3x5gc1YAwv9SH+7 vtDy9dAk2PQhZv0fCD3j5wGC7s2cIJ4hrxJwXCnNucmcPbN7f0T21BqSvFQxdvYgw7H4 yI9iWP2T5R3tuKtdxxhEE7WwmD70h/iVUuXYSKH9Nn5dVCEoXItxa64LDubpGXuhJvoH Apow== X-Gm-Message-State: ABuFfogITp8iY6HzZ97fd5Tv5Jvzzmd10q0Q9PWLIubUlgzjBmX3O82l yIvFvPk3mW/LPM0IyyRPSgJxaGEJfplXOZ2KSPT49R1oklU= X-Google-Smtp-Source: ACcGV63Spxd0HnYFObs4bvL4ExQ5WCE7hrCPpLxfmAo0Yb2OS7bcfvDGo0nlvAd6CFmRHdoToNmg2u+y4l8kGER1P4c= X-Received: by 2002:a9d:7110:: with SMTP id n16mr16455530otj.305.1539810860633; Wed, 17 Oct 2018 14:14:20 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Mike Thomsen Date: Wed, 17 Oct 2018 17:14:09 -0400 Message-ID: Subject: Re: Scaling source processors in nifi horizontally. To: dev@nifi.apache.org Content-Type: multipart/alternative; boundary="0000000000005e923905787327e5" --0000000000005e923905787327e5 Content-Type: text/plain; charset="UTF-8" > may have to build some kind of tooling on top of it to monitor/provision new processor for newly added schemas etc. Could you elaborate on this part of your use case? On Wed, Oct 17, 2018 at 2:31 PM ashwin konale wrote: > Hi, > > I am experimenting with nifi for one of our usecases with plans of > extending it to various other data routing, ingestion usecases. Right now I > need to ingest data from mysql binlogs to hdfs/GCS. We have around 250 > different schemas and about 3000 tables to read data from. Volume of the > data flow ranges from 500 - 2000 messages per second in different schemas. > > Right now the problem is mysqlCDC processor can run in only one thread. To > overcome this issue I have two options. > > 1. Use primary node execution, so different processors for each of the > schemas. So eventually all processors which reads from mysql will run in > single node, which will be a bottleneck no matter how big my nifi cluster > is. > > 2. Another approach is to use multiple nifi instances to pull data and have > master nifi cluster for ingestion to various sinks. In this approach I will > have to manage all these small nifi instances, and may have to build some > kind of tooling on top of it to monitor/provision new processor for newly > added schemas etc. > > Is there any better way to achieve my usecase with nifi ? Please advice me > on the architechture. > > Looking forward for suggestion. > > - Ashwin > --0000000000005e923905787327e5--