nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Percivall (JIRA)" <>
Subject [jira] [Created] (NIFI-1582) New processor to allow basic data science
Date Wed, 02 Mar 2016 19:07:18 GMT
Joseph Percivall created NIFI-1582:

             Summary: New processor to allow basic data science
                 Key: NIFI-1582
             Project: Apache NiFi
          Issue Type: New Feature
            Reporter: Joseph Percivall

This idea was sparked by a thread on the user list:
I expect that in the future I’ll need something a little more sophisticated but for now
my problem is very simple:
I want to be able to trigger an alert (only once) when an attribute in an incoming stream,
for instance, goes over a predefined threshold. The Processor should then trigger (only once
again) another trigger when the signal goes back to normal (below threshold). Basically a
RouteByAttribute but with memory.



Hello Claudio,

Your use-case actually could leverage a couple of recently added features to create a really
cool open-source processor. The two key features that were added are State Management and
the ability to reference processor specific variables in expression language. You can take
a look at RouteText to see both in action. 

By utilizing both you can create a processor that is configured with multiple Expression language
expressions. There would be dynamic properties which would accept expression language and
then store the evaluated value via state management. Then there would be a routing property
(that supports expression language) that could simply add an attribute to the flowfile with
the evaluated value which would allow it to be used by flowing processors for routing.

This would allow you to do your use-case where you store the value for the incoming stream
and route differently once you go over a threshold. It could even allow more complex use-cases.
One instance, I believe, would be possible is to have a running average and standard deviation
and route data to different locations based on it's standard deviation.

You can think of this like an UpdateAttribute with the ability to store and calculate variables
using expression language.

This message was sent by Atlassian JIRA

View raw message