From dev-return-17466-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Thu Jun 7 20:47:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DB827180663 for ; Thu, 7 Jun 2018 20:47:03 +0200 (CEST) Received: (qmail 85457 invoked by uid 500); 7 Jun 2018 18:47:02 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 85445 invoked by uid 99); 7 Jun 2018 18:47:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jun 2018 18:47:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 9671CCD577 for ; Thu, 7 Jun 2018 18:47:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.901 X-Spam-Level: X-Spam-Status: No, score=-0.901 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.79, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id GEuR_lf5h_bR for ; Thu, 7 Jun 2018 18:47:00 +0000 (UTC) Received: from mail-oi0-f48.google.com (mail-oi0-f48.google.com [209.85.218.48]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 831705F21E for ; Thu, 7 Jun 2018 18:47:00 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id c128-v6so9554106oig.11 for ; Thu, 07 Jun 2018 11:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-transfer-encoding; bh=gOExozvpaZJB8stYoQz2zIob70zSb6Ok52LNINsZ/ig=; b=k0619rAH8iRXQswyEKYPMPPktRDnmgRuC6u0bQZGrvZEikyBvMer7YwCMbeQTecXKy ywnFRzWTwcU2SvAoMpUTZawuNw0tWG+9tYg/uPhYWdy9opuGbl9Mx6EE4sirdV1by5tO GcBwJ5lefcxlfDAYF2fK800h3K8KuwrzCly14OdCGuLWMxCjuQOTMTQ9WE2BaW3RBsC9 UF5fSkTvFW49IPLZBgX3noWdyckT2OY8myEjZqkiU/BgSoWH4QpjmOuJPVeT7XftLHib fkoMy0NquzqahoGMiFxZPm0j5Co/j3y2yuuUl5zrZkjMtz88EuECpUCIHj5/wy1EcKic RhmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-transfer-encoding; bh=gOExozvpaZJB8stYoQz2zIob70zSb6Ok52LNINsZ/ig=; b=UmaLzNl90magTyEbYI/+a0ysyJcwowmzLagmqsyJzqEVDTXMOKAuwi6OzrOgMSvZ5d ly695QQ773dSLM9dUK1KjWMnzb4asrE7S5RW94eo1Bzj/7+KIqneeyxZO6ubSrrYgaiZ IhASR6yDyBKuZzKmSU9uTvizYSUyDLcDwG0xmRlexP6uehEiRDiWJ9kRiw71XUYEbXF9 b+gC0erlmu1842s2Gs7gJ2HqwC5hD6ool4gGuN/oCuYeTy7SeDEPRTtv1cxUoojFQhBy nNYdRR6jU2tV727GOKDjrArXgVZAcWUcuGHwqF+ux5g1zTK5nFVK4L7KzAishLE1gJSz V9Tw== X-Gm-Message-State: APt69E29w766gS11mpmiOmKO3rkr+/6HDaZ25TkcZfgain6ZBstorX92 Wen1gW3le6OwXBQ3rKHJ02BtEp57AOz1fu0vT1B6XA== X-Google-Smtp-Source: ADUXVKKGOMRQbWistcsmnro4Nj1HbxpAH1v/dUqzCs2qbYidRV47kT5OEiHarV7nNB/qt23f4mFsT3sHAoTeKXExls8= X-Received: by 2002:aca:ed53:: with SMTP id l80-v6mr1637548oih.281.1528397219746; Thu, 07 Jun 2018 11:46:59 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a4a:9686:0:0:0:0:0 with HTTP; Thu, 7 Jun 2018 11:46:59 -0700 (PDT) In-Reply-To: <833e2af107fc4f5e8a172987a1bb4e4d@SIWEX4B.sing.micron.com> References: <833e2af107fc4f5e8a172987a1bb4e4d@SIWEX4B.sing.micron.com> From: Bryan Bende Date: Thu, 7 Jun 2018 14:46:59 -0400 Message-ID: Subject: Re: Primary Only Content Migration To: dev@nifi.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Peter, There really shouldn't be any non-source processors scheduled for primary node only. We may even want to consider preventing that option when the processor has an incoming connection to avoid creating any confusion. As long as you set source processors to primary node only then everything should be ok... if primary node changes, the source processor starts executing on the new primary node, and any flow files it already produced on the old primary node will continue to be worked off by the downstream processors on the old node until they are all processed. -Bryan On Thu, Jun 7, 2018 at 1:55 PM, Peter Wicks (pwicks) wr= ote: > I'm sure many of you have the same situation, a flow that runs on a clust= er, and at some point merges back down to a primary only processor; your fi= les sit there in the queue with nowhere to go... We've used the work around= of having a remote processor group that loops the data back to the primary= node for a while, but would really like a clean/simple solution. This appr= oach requires that users be able to put an input port on the root flow, and= then route the file back down, which is a nuisance. > > I have been thinking of adding either a processor that moves data between= specific nodes in a cluster, or a queue (?) option that will let users mig= rate the content of a flowfile back to the master node. This would allow yo= u to move data back to a primary very easily without needing RPG's and inpu= t ports at the root level. > > All of my development work with NiFi has been focused on processors, so I= 'm not really sure where I would start with this. Thoughts? > > Thanks, > Peter