From issues-return-81652-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Tue Jul 23 15:01:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id CBBEE1802C7 for ; Tue, 23 Jul 2019 17:01:01 +0200 (CEST) Received: (qmail 42503 invoked by uid 500); 23 Jul 2019 15:01:01 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 42489 invoked by uid 99); 23 Jul 2019 15:01:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Jul 2019 15:01:01 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 87DD8E2800 for ; Tue, 23 Jul 2019 15:01:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 45532265CC for ; Tue, 23 Jul 2019 15:01:00 +0000 (UTC) Date: Tue, 23 Jul 2019 15:01:00 +0000 (UTC) From: "Alessandro D'Armiento (JIRA)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (NIFI-6465) ListHDFS: skip last should be optional MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/NIFI-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro D'Armiento resolved NIFI-6465. ----------------------------------------- Resolution: Not A Problem > ListHDFS: skip last should be optional > -------------------------------------- > > Key: NIFI-6465 > URL: https://issues.apache.org/jira/browse/NIFI-6465 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework > Affects Versions: 1.9.2 > Reporter: Alessandro D'Armiento > Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > h2. Current Situation > From [official documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-hadoop-nar/1.9.2/org.apache.nifi.processors.hadoop.ListHDFS/index.html] > * Each time a listing is performed, the files with the latest timestamp will be excluded and picked up during the next execution of the processor. This is done to ensure that we do not miss any files, or produce duplicates, in the cases where files with the same timestamp are written immediately before and after a single execution of the processor. > h2. Improvement Proposal > * If we are calling the ListHDFS only after a certain operation which populates an HDFS directory has finished, it is pointless to skip the last file, and avoiding this behavior is tricky. > * A mandatory property "skip last" should be implemented in order to be able to actively decide whether or not this behavior is necessary, based on the use case. > * This is also particularly useful in combination with [NIFI-6462] -- This message was sent by Atlassian JIRA (v7.6.14#76016)