nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: [DISCUSS] Predictive Analytics for NiFi Metrics
Date Tue, 30 Jul 2019 17:40:21 GMT
Yolanda, 

I think this sounds like a great idea and will be very useful to admins/users, as well as
enabling some interesting next-level functionality and insight generation. Thanks for putting
this out there. 

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jul 30, 2019, at 5:55 AM, Yolanda Davis <yolanda.m.davis@gmail.com> wrote:
> 
> Hello Everyone,
> 
> I wanted to reach out to the community to discuss potentially enhancing
> NiFi to include predictive analytics that can help users assess and predict
> NiFi behavior and performance. Currently NiFi has lots of metrics available
> for areas including jvm and flow component usage (via component status) as
> well as provenance data which NiFi makes available either through the UI or
> reporting tasks (for consumption by other systems). Past discussions in the
> community cite users shipping this data to applications such as Prometheus,
> ELK stacks, or Ambari metrics for further analysis in order to
> capture/review performance issues, detect anomalies, and send alerts or
> notifications.  These systems are efficient in capturing and helping to
> analyze these metrics however it requires customization work and knowledge
> of NiFi operations to provide meaningful analytics within a flow context.
> 
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for
> flows, such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters.  This framework could offer
> several key features:
> 
>   - Provide a flexible internal analytics engine and model api which
>   supports the addition of or enhancement to onboard models
>   - Support integration of remote or cloud based ML models
>   - Support both traditional and online (incremental) learning methods
>   - Provide support for model caching  (perhaps later inclusion into a
>   model repository or registry)
>   - UI enhancements to display prediction information either in existing
>   summary data, new data visualizations, or directly within the flow/canvas
>   (where applicable)
> 
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable
> performing model.  We have some ideas on how this could be achieved however
> we wanted to discuss this more with the community to get thoughts about
> tackling this work, especially if there are specific use cases or other
> factors that should be considered.
> 
> Looking forward to everyone's thoughts and input.
> 
> Thanks,
> 
> -yolanda
> 
> --
> yolanda.m.davis@gmail.com
> @YolandaMDavis


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message