nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owens, Mark" <jmow...@evoforge.org>
Subject RE: Re:[EXT] [DISCUSS] Predictive Analytics for NiFi Metrics
Date Mon, 19 Aug 2019 18:24:33 GMT
Hi Yolanda,



I've been working on a feature that appears to possibly overlap with the work you are pursuing.
Perhaps we should see if/should we try to coordinate our efforts. I've been updating NiFi
to predict the time to queue overflow for both flowfiles and bytes and displaying that information
in the GUI. For the initial attempt, I’ve been using a simple model of straight line prediction
over a sliding window of 15 minutes to predict when flows will fail. This estimate is then
displayed on both the NiFi Summary page under the connections tab and in the status history
graphs.  Below are examples of what would be displayed to the user.



[cid:image001.png@01D55696.E4CCD550]



The Connection tab contains a new column on the right that displays the prediction for both
flow files and data size. The user can select a maximum time at which specific times are no
longer displayed. In this example, if the prediction lies beyond 12 hours then the display
simply indicates that the flow is greater than 12 hours away from failure at the moment.



[cid:image002.png@01D55697.2C8AC500]



This display graphs the prediction for byte overflow over time. Note that if the estimate
is greater than the user provided maximum value of interest the graph maxes out at that time,
effectively indicating no overflow concerns.



[cid:image003.png@01D55697.965C27D0]



A similar display for flowfile count is displayed as well.



The current state of work can be found at https://github.com/jmark99/nifi/tree/time-to-overflow



I welcome your (or any others) feedback on this effort.



Thanks,
Mark



P.S. If the images are not displaying, they can be viewed at https://github.com/jmark99/nifi-images







-----Original Message-----
From: Yolanda Davis <yolanda.m.davis@gmail.com>
Sent: Monday, August 19, 2019 11:29 AM
To: dev@nifi.apache.org
Subject: Re:[EXT] [DISCUSS] Predictive Analytics for NiFi Metrics



Hello All,



I just wanted to follow up on the discussion we started a couple of weeks ago concerning an
analytics framework for NiFi metrics.  Working with Andy Christianson and Matt Burgess we
shaped our ideas and drafted a proposal for this feature on the Apache NiFi Wiki [1] . We've
also begun implementing some of these ideas in a feature branch (which is work in

progress) [2].  We’d appreciate any questions or feedback you may have.



Thanks,



-yolanda



[1] -

https://cwiki.apache.org/confluence/display/NIFI/Operational+Analytics+Framework+for+NiFi

[2] - https://github.com/apache/nifi/commits/analytics-framework



On Wed, Jul 31, 2019 at 9:58 AM Andy Christianson <aichrist@protonmail.com.invalid<mailto:aichrist@protonmail.com.invalid>>
wrote:



> As someone who operated a 24/7 mission-critical NiFi flow, this

> feature would have been a life saver. If I'm heading home on a Friday,

> it would be great to have some blinking red lights to let me know that

> the system predicts that it is going to experience backpressure

> sometime over the weekend, so that corrective action could be taken before leaving.

>

> Since there is support in the community for this, I created a JIRA to

> track the effort:

>

> https://issues.apache.org/jira/browse/NIFI-6510

>

> I also created a JIRA to track the remote protocol:

>

> https://issues.apache.org/jira/browse/NIFI-6511

>

>

> Regards,

>

> Andy

>

>

> Sent from ProtonMail, Swiss-based encrypted email.

>

> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> On Wednesday, July 31, 2019 6:57 AM, Arpad Boda <aboda@apache.org<mailto:aboda@apache.org>>
wrote:

>

> > If you could share a bit more details about your OPC and Modbus

> > usage,

> that

> > would be highly appreciated!

> >

> > On Wed, Jul 31, 2019 at 12:01 PM Craig Knell craig.knell@gmail.com<mailto:craig.knell@gmail.com>

> wrote:

> >

> > > Sounds. Great

> > > Let me know if you need some help

> > > Best regards

> > > Craig

> > >

> > > > On 31 Jul 2019, at 17:31, Arpad Boda aboda@cloudera.com.invalid<mailto:aboda@cloudera.com.invalid>

> wrote:

> > > > Craig,

> > > > OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and

> Modbus (

> > > > https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the

> way for

> > > > MiNiFi c++, hopefully both will be part of next release (0.7.0).

> > > > It's gonna be legen... wait for it! :) Regards, Arpad

> > > >

> > > > > On Wed, Jul 31, 2019 at 2:30 AM Craig Knell

> > > > > craig.knell@gmail.com<mailto:craig.knell@gmail.com>

> > > > > wrote:

> > > >

> > > > > Hi Folks

> > > > > That's our use case now. All our Models are run in python.

> > > > > Currently we send events to the ML via http, although this is

> > > > > not optimal

> > > >

> > > > > Our use case is edge ML where we want a light weight wrapper

> > > > > for Python code base.

> > > > > Jython however does not work with the code base I'm think of

> > > > > changing the interface to some thing like REDIS for

> pub/sub

> > > > > Id also like this to be a push deployment via minifi Also

> > > > > support for sensors via protocols via Modbus and OPC would be

> great

> > > > > Craig

> > > > >

> > > > > > On Wed, Jul 31, 2019 at 1:43 AM Joe Witt joe.witt@gmail.com<mailto:joe.witt@gmail.com>

> wrote:

> > > > > > Definitely something that I think would really help the

> community. It

> > > > > > might make sense to frame/structure these APIs such that an

> internal

> > > > > > option

> > > > > > could be available to reduce dependencies and get up and

> > > > > > running

> but

> > > > > > that

> > > >

> > > > > > also just as easily a remote implementation where the engine

> lives and

> > > > > > is

> > > >

> > > > > > managed externally could also be supported.

> > > > > > Thanks

> > > > > > On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto

> alopresto@apache.org<mailto:alopresto@apache.org>

> > > > > > wrote:

> > > > > >

> > > > > > > Yolanda,

> > > > > > > I think this sounds like a great idea and will be very

> > > > > > > useful

> to

> > > > > > > admins/users, as well as enabling some interesting

> > > > > > > next-level functionality

> > > > > >

> > > > > > > and insight generation. Thanks for putting this out there.

> > > > > > > Andy LoPresto

> > > > > > > alopresto@apache.org<mailto:alopresto@apache.org>

> > > > > > > alopresto.apache@gmail.com<mailto:alopresto.apache@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6

> > > > > > > 5A3F D3C4 BACE 3C6E F65B 2F7D

> EF69

> > > > > > >

> > > > > > > > On Jul 30, 2019, at 5:55 AM, Yolanda Davis <

> > > > > > > > yolanda.m.davis@gmail.com<mailto:yolanda.m.davis@gmail.com>>

> > > > > >

> > > > > > > wrote:

> > > > > > >

> > > > > > > > Hello Everyone,

> > > > > > > > I wanted to reach out to the community to discuss

> > > > > > > > potentially enhancing

> > > > > >

> > > > > > > > NiFi to include predictive analytics that can help
users

> assess and

> > > > > > > > predict

> > > > > > > > NiFi behavior and performance. Currently NiFi has
lots

> > > > > > > > of

> metrics

> > > > > > > > available

> > > > > > > > for areas including jvm and flow component usage (via

> component

> > > > > > > > status)

> > > > > >

> > > > > > > as

> > > > > > >

> > > > > > > > well as provenance data which NiFi makes available

> > > > > > > > either

> through

> > > > > > > > the UI

> > > > > >

> > > > > > > or

> > > > > > >

> > > > > > > > reporting tasks (for consumption by other systems).
Past

> discussions

> > > > > > > > in

> > > > > >

> > > > > > > the

> > > > > > >

> > > > > > > > community cite users shipping this data to applications

> > > > > > > > such

> as

> > > > > > > > Prometheus,

> > > > > > > > ELK stacks, or Ambari metrics for further analysis
in

> > > > > > > > order

> to

> > > > > > > > capture/review performance issues, detect anomalies,
and

> send alerts

> > > > > > > > or

> > > > > >

> > > > > > > > notifications. These systems are efficient in capturing

> > > > > > > > and

> helping

> > > > > > > > to

> > > > > >

> > > > > > > > analyze these metrics however it requires customization

> > > > > > > > work

> and

> > > > > > > > knowledge

> > > > > > > > of NiFi operations to provide meaningful analytics

> > > > > > > > within a

> flow

> > > > > > > > context.

> > > > > >

> > > > > > > > In speaking with Matt Burgess and Andy Christianson
on

> > > > > > > > this

> topic we

> > > > > > > > feel

> > > > > >

> > > > > > > > that there is an opportunity to introduce an analytics

> framework that

> > > > > > > > could

> > > > > > > > provide users reasonable predictions on key performance

> indicators

> > > > > > > > for

> > > > > >

> > > > > > > > flows, such as back pressure and flow rate, to help

> administrators

> > > > > > > > improve

> > > > > > > > operational management of NiFi clusters. This framework

> could offer

> > > > > > > > several key features:

> > > > > > > >

> > > > > > > > -   Provide a flexible internal analytics engine and
model

> api which

> > > > > > > >     supports the addition of or enhancement to onboard

> > > > > > > > models

> > > > > > > >

> > > > > > > > -   Support integration of remote or cloud based ML
models

> > > > > > > > -   Support both traditional and online (incremental)

> learning

> > > > > > > >     methods

> > > > > > > >

> > > > > >

> > > > > > > > -   Provide support for model caching (perhaps later

> inclusion into

> > > > > > > >     a

> > > > > > > >

> > > > > >

> > > > > > > > model repository or registry)

> > > > > > > >

> > > > > > > > -   UI enhancements to display prediction information
either

> in

> > > > > > > >     existing

> > > > > > > >

> > > > > >

> > > > > > > > summary data, new data visualizations, or directly

> > > > > > > > within the flow/canvas (where applicable) For an initial

> > > > > > > > target we thought that back pressure

> prediction would

> > > > > > > > be a

> > > > > >

> > > > > > > > good starting point for this initiative, given that
back

> pressure

> > > > > > > > detection

> > > > > > > > is a key indicator of flow performance and many of
the

> metrics

> > > > > > > > currently

> > > > > >

> > > > > > > > available would provide enough data points to create
a

> reasonable

> > > > > > > > performing model. We have some ideas on how this could

> > > > > > > > be

> achieved

> > > > > > > > however

> > > > > > > > we wanted to discuss this more with the community
to get

> thoughts

> > > > > > > > about

> > > > > >

> > > > > > > > tackling this work, especially if there are specific
use

> cases or

> > > > > > > > other

> > > > > >

> > > > > > > > factors that should be considered.

> > > > > > > > Looking forward to everyone's thoughts and input.

> > > > > > > > Thanks,

> > > > > > > > -yolanda

> > > > > > > > --

> > > > > > > > yolanda.m.davis@gmail.com<mailto:yolanda.m.davis@gmail.com>
@YolandaMDavis

> > > > >

> > > > > --

> > > > > Regards

> > > > > Craig Knell

> > > > > Mobile: +61 402 128 615

> > > > > Skype: craigknell

>

>

>



--

--

yolanda.m.davis@gmail.com<mailto:yolanda.m.davis@gmail.com>

@YolandaMDavis
Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message