From dev-return-16823-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Tue Mar 27 15:56:53 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B4D9618064E for ; Tue, 27 Mar 2018 15:56:52 +0200 (CEST) Received: (qmail 44445 invoked by uid 500); 27 Mar 2018 13:56:51 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 44432 invoked by uid 99); 27 Mar 2018 13:56:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Mar 2018 13:56:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 81503C6A11 for ; Tue, 27 Mar 2018 13:56:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.898 X-Spam-Level: X-Spam-Status: No, score=0.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id EoLXTUWPhsdx for ; Tue, 27 Mar 2018 13:56:49 +0000 (UTC) Received: from mail-wr0-f178.google.com (mail-wr0-f178.google.com [209.85.128.178]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 867215F1B4 for ; Tue, 27 Mar 2018 13:56:49 +0000 (UTC) Received: by mail-wr0-f178.google.com with SMTP id u11so10201409wri.12 for ; Tue, 27 Mar 2018 06:56:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-transfer-encoding; bh=k52GETIOVWbxEQJtZ1oVHqHwBNQ6M3HzM9DywN/ifN4=; b=RmYDZduVx+xTFKhcCLnK0VBl9sMV/HwVyykjeClOoF9sDLHP5SvJOZojnSP9ZImnLY Ed5mt3zTFcf/uinmjWc3eMGGSB1w3xLtKu9Q8UAA+bGDzOdGwk2Pfno2fI/kE7mTJCuv H6XLrA+YbbQ6LlieFgzZXXMPjZNFTR8dZoOwpdWO9KReT+0vLspvnp9gWa6f3QZQI+HW RdhlS2XrMzp2aey4B7KKSdmS9E0746R6RBzBYUtLpyLhY0ASrqUcv1fUF1uYe7Pzjh/b oEbQP+uVdIkXp1v//GOSztdbKdTM9USWqZgMZNRXSntIZ66v7nzX7B3P3oTY//Zi/MrV IbxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-transfer-encoding; bh=k52GETIOVWbxEQJtZ1oVHqHwBNQ6M3HzM9DywN/ifN4=; b=SP8BhxKlutXZTGlIc298dfBhpgbEDBU2/Uhaa3afWEcTj7rl3R6Jru5csaIVdROrF1 vSAkQBZd4bMaJKMqKFMvo8TLi8SLzAe2OOBmNukG2XMcPB+c8bN4EJUIp8PGKHZ7hHgv o+d3zhsRcqQCgS3gVBJ9TSr8/uyx2R1tdjEUZbVLi7aFD796vle2CYZP04NRG6w19b3S ZCZ6eh3LHyeGUseAJTYkvDtYMx46yE+xs6q7lyM0pBfQkUyEC3ffVlPYn1EKTj3rVABr WOrhlH0rlE8wbVcV1Pc2N7Do7ZQJrIxcGLF/xTH9z7VXsWqS6X8nbgPmgxG0uWFzrJ5D PeiQ== X-Gm-Message-State: AElRT7EkaYjwyiGh705wVx34tQJfJP4i+Rb6HnRO97ZkGm6PC2Fv35hC Fa4BgGu7EJOVqaC5JtHF6w81qdkmxPFh1wD97zCGgg== X-Google-Smtp-Source: AIpwx4+8B3D75eh+HpoY5y5d77+INvIyCfDW90BYQYWcYJ4nlp6vFEBq5W3MszcsNsBVrfHg4XvH53L/BoioKTSyHBI= X-Received: by 10.223.190.138 with SMTP id i10mr7367145wrh.40.1522159008148; Tue, 27 Mar 2018 06:56:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.136.249 with HTTP; Tue, 27 Mar 2018 06:56:47 -0700 (PDT) In-Reply-To: References: From: Bryan Bende Date: Tue, 27 Mar 2018 09:56:47 -0400 Message-ID: Subject: Re: ListSFTP incoming relationship To: dev@nifi.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'm not sure that would solve the problem because you'd still be limited to one directory. What most people are asking for is the ability to use a dynamic directory from an incoming flow file. I think we might be trying to fit two different use-cases into one processor which might not make sense. Scenario #1... There is a directory that is constantly receiving new data and has a significant amount of files, and I want to periodically find new files. This is what the current processors are optimized for. Scenario #2... There is a directory that is mostly static with a moderate/small number of files, and at points in my flow I want to dynamically perform a listing of this directory and retrieve the files. This is more geared towards the mentality of running a job/workflow. On Tue, Mar 27, 2018 at 9:36 AM, Otto Fowler wrot= e: > What if the changes where =E2=80=98on top of=E2=80=99 some base set of pr= operties, like > directory? > Like a filter, where if present from the incoming file will have the LIST= * > list only things > that match a name or attribute? > > > > On March 27, 2018 at 00:08:41, Joe Witt (joe.witt@gmail.com) wrote: > > Scott > > This idea has come up a couple of times and there is definitely > something intriguing to it. Where I think this idea stalls out though > is in implementation. > > While I agree that the other List* processors might similarly benefit > lets focus on ListFile. Today you tell ListFile what directory to > start looking for files in. It goes off scanning that directory for > hits and stores state about what it has already searched/seen. And it > is important to keep track of how much it has already scanned because > at times the search directory can be massive (100,000s of thousands or > more files and directories to scan for example). > > In the proposed model the directory to be scanned could be provided > dynamically by looking at an attribute of an incoming flowfile (or > other criteria can be provided - not just the directory to scan). In > this case the ListFile processor goes on scanning against that now. > What about the previous directory (or directories) it was told to > scan? Does it still track those too? What if it starts scanning the > newly provided directory, hasn't finished pulling all the data or new > data is continually arriving, and it is told to switch to another > directory. > > I think if those questions can get solid answers and someone invests > time in creating a PR then this could be pretty powerful. Would be > good to see a written description of the use case(s) for this too. > > Thanks > Joe > > On Mon, Mar 26, 2018 at 11:58 PM, scott wrote: >> Hello Devs, >> >> I would like to request a feature to a major processor, ListSFTP. But > before >> I do down the official road, I wanted to ask if anyone thought it was a >> terrible idea or impossible, etc. The request is to add support for an >> incoming relationship to the ListSFTP processor specifically, but I coul= d >> see it added to many of the commonly used head processes, such as > ListFile. >> I would envision functionality more like InvokeHTTP or ExecuteSQL, where > an >> incoming flow file could initiate the action, and the attributes in the >> incoming flow file could be used to configure the processor actions. It'= s >> the configuration aspect that most appeals to me, because it opens it up > to >> being centrally or dynamically configured. >> >> Thanks, >> >> Scott >>