nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-2735) Add processor to perform simple aggregations
Date Tue, 06 Sep 2016 14:59:21 GMT

    [ https://issues.apache.org/jira/browse/NIFI-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15467618#comment-15467618
] 

ASF GitHub Bot commented on NIFI-2735:
--------------------------------------

GitHub user mattyb149 opened a pull request:

    https://github.com/apache/nifi/pull/988

    NIFI-2735: Add AggregateValues processor for aggregate operations on flow file groups

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mattyb149/nifi aggregator

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/988.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #988
    
----
commit 0730a98bc67f905dfd9edf479b2fcc7b2426386d
Author: Matt Burgess <mattyb149@apache.org>
Date:   2016-09-06T14:51:28Z

    NIFI-2735: Add AggregateValues processor for aggregate operations on flow file groups

----


> Add processor to perform simple aggregations
> --------------------------------------------
>
>                 Key: NIFI-2735
>                 URL: https://issues.apache.org/jira/browse/NIFI-2735
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Extensions
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
>
> This is a proposal for a new processor (AggregateValues, for example) that can perform
simple aggregation operations such as count, sum, average, min, max, and concatenate, over
a set of "related" flow files. For example, when a JSON file is split on an array (using the
SplitJson processor), the total count of the splits, the index of each split, and the unique
indentifier (shared by each split) are stored as attributes in each flow file sent to the
"splits" relationship:
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitJson/index.html
> These attributes are the "fragment.*" attributes in the documentation for SplitText,
SplitXml, and SplitJson, for example.
> Such a processor could perform these operations for each flow file split from the original
document, and when all documents from a split have been processed, a flow file could be transferred
to an "aggregate" relationship containing attributes for the operation, aggregate value, etc.
> An interesting application of this (besides the actual aggregation operations) is that
you can use the "aggregate" relationship as an event trigger. For example if you need to wait
until all files from a group are processed, you can use AggregateValues and the "aggregate"
relationship to indicate downstream that the entire group has been processed. If there is
not a Split processor upstream, then the attributes (fragment.*) would have to be manipulated
by the data flow designer, but this can be accomplished with other processors (including the
scripting processors if necessary). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message