nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <>
Subject [jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
Date Thu, 05 Sep 2019 19:04:00 GMT


ASF subversion and git services commented on NIFI-6510:

Commit de9e2cbb72ca24f979dc89b9ee283686437bc68d in nifi's branch refs/heads/analytics-framework
from Andrew I. Christianson
[;h=de9e2cb ]

NIFI-6510 Optimize imports

> Predictive Analytics for NiFi Metrics
> -------------------------------------
>                 Key: NIFI-6510
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Andrew Christianson
>            Assignee: Yolanda M. Davis
>            Priority: Major
>             Fix For: 1.10.0
>          Time Spent: 4h 10m
>  Remaining Estimate: 0h
> From Yolanda's email to the list:
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow component
usage (via component status) as well as provenance data which NiFi makes available either
through the UI or reporting tasks (for consumption by other systems). Past discussions in
the community cite users shipping this data to applications such as Prometheus, ELK stacks,
or Ambari metrics for further analysis in order to capture/review performance issues, detect
anomalies, and send alerts or notifications. These systems are efficient in capturing and
helping to analyze these metrics however it requires customization work and knowledge of NiFi
operations to provide meaningful analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel that there
is an opportunity to introduce an analytics framework that could provide users reasonable
predictions on key performance indicators for flows, such as back pressure and flow rate,
to help administrators improve operational management of NiFi clusters. This framework could
offer several key features:
> - Provide a flexible internal analytics engine and model api which supports the addition
of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model repository
or registry)
> - UI enhancements to display prediction information either in existing summary data,
new data visualizations, or directly within the flow/canvas (where applicable)
> For an initial target we thought that back pressure prediction would be a good starting
point for this initiative, given that back pressure detection is a key indicator of flow performance
and many of the metrics currently available would provide enough data points to create a reasonable
performing model. We have some ideas on how this could be achieved however we wanted to discuss
this more with the community to get thoughts about tackling this work, especially if there
are specific use cases or other factors that should be considered.{noformat}

This message was sent by Atlassian Jira

View raw message