From issues-return-64657-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Sat Sep 15 19:36:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B8A37180629 for ; Sat, 15 Sep 2018 19:36:03 +0200 (CEST) Received: (qmail 38495 invoked by uid 500); 15 Sep 2018 17:36:02 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 38486 invoked by uid 99); 15 Sep 2018 17:36:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Sep 2018 17:36:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 63490C71FD for ; Sat, 15 Sep 2018 17:36:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 9AoQcB2gvJj2 for ; Sat, 15 Sep 2018 17:36:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id B2EB65F3E2 for ; Sat, 15 Sep 2018 17:36:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id EFD32E25B0 for ; Sat, 15 Sep 2018 17:36:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2BA1A23FA3 for ; Sat, 15 Sep 2018 17:36:00 +0000 (UTC) Date: Sat, 15 Sep 2018 17:36:00 +0000 (UTC) From: "Matt Burgess (JIRA)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (NIFI-5601) Add fragment.* attributes to GenerateTableFetch MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Matt Burgess created NIFI-5601: ---------------------------------- Summary: Add fragment.* attributes to GenerateTableFetch Key: NIFI-5601 URL: https://issues.apache.org/jira/browse/NIFI-5601 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess GenerateTableFetch will generate SQL statements in multiple flow files corresponding to "pages" of data for a single execution of the processor. QueryDatabaseTable does something similar, although it also does the fetching of the rows and adds fragment.* attributes to the outgoing flow files, in case a merge or other correlation needs to be done later. GenerateTableFetch (although it doesn't produce source data per se) is usually used in conjunction with ExecuteSQL to fetch the actual pages of rows, and sometimes these would need to be merged/correlated. To that end, this Jira proposes to add the same fragment.* attributes to the outgoing GTF flow files for use downstream. Note that if the SQL statements are distributed among a cluster (for parallel execution by ExecuteSQL), they will not be able to be merged, as different nodes may get different fragments and often no single node will get all the fragments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)