nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Burgess (JIRA)" <>
Subject [jira] [Created] (NIFI-5601) Add fragment.* attributes to GenerateTableFetch
Date Sat, 15 Sep 2018 17:36:00 GMT
Matt Burgess created NIFI-5601:

             Summary: Add fragment.* attributes to GenerateTableFetch
                 Key: NIFI-5601
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
            Reporter: Matt Burgess

GenerateTableFetch will generate SQL statements in multiple flow files corresponding to "pages"
of data for a single execution of the processor. QueryDatabaseTable does something similar,
although it also does the fetching of the rows and adds fragment.* attributes to the outgoing
flow files, in case a merge or other correlation needs to be done later.

GenerateTableFetch (although it doesn't produce source data per se) is usually used in conjunction
with ExecuteSQL to fetch the actual pages of rows, and sometimes these would need to be merged/correlated.
To that end, this Jira proposes to add the same fragment.* attributes to the outgoing GTF
flow files for use downstream.

Note that if the SQL statements are distributed among a cluster (for parallel execution by
ExecuteSQL), they will not be able to be merged, as different nodes may get different fragments
and often no single node will get all the fragments.

This message was sent by Atlassian JIRA

View raw message