From issues-return-84511-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Sun Sep 8 21:34:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1BA6A18062C for ; Sun, 8 Sep 2019 23:34:02 +0200 (CEST) Received: (qmail 42769 invoked by uid 500); 8 Sep 2019 21:34:01 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 42757 invoked by uid 99); 8 Sep 2019 21:34:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Sep 2019 21:34:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 979B6E026E for ; Sun, 8 Sep 2019 21:34:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 0E997780749 for ; Sun, 8 Sep 2019 21:34:00 +0000 (UTC) Date: Sun, 8 Sep 2019 21:34:00 +0000 (UTC) From: "ASF subversion and git services (Jira)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/NIFI-6510?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16925= 257#comment-16925257 ]=20 ASF subversion and git services commented on NIFI-6510: ------------------------------------------------------- Commit 4ff359abfc2b68a52522f722d15b5c2a9f91d288 in nifi's branch refs/heads= /analytics-framework from Yolanda M. Davis [ https://gitbox.apache.org/repos/asf?p=3Dnifi.git;h=3D4ff359a ] NIFI-6510 - rolled back last change and applied minNonNegative method > Predictive Analytics for NiFi Metrics > ------------------------------------- > > Key: NIFI-6510 > URL: https://issues.apache.org/jira/browse/NIFI-6510 > Project: Apache NiFi > Issue Type: Improvement > Reporter: Andrew Christianson > Assignee: Yolanda M. Davis > Priority: Major > Fix For: 1.10.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > From Yolanda's email to the list: > =C2=A0 > {noformat} > Currently NiFi has lots of metrics available for areas including jvm and = flow component usage (via component status) as well as provenance data whic= h NiFi makes available either through the UI or reporting tasks (for consum= ption by other systems). Past discussions in the community cite users shipp= ing this data to applications such as Prometheus, ELK stacks, or Ambari met= rics for further analysis in order to capture/review performance issues, de= tect anomalies, and send alerts or notifications. These systems are efficie= nt in capturing and helping to analyze these metrics however it requires cu= stomization work and knowledge of NiFi operations to provide meaningful ana= lytics within a flow context. > In speaking with Matt Burgess and Andy Christianson on this topic we feel= that there is an opportunity to introduce an analytics framework that coul= d provide users reasonable predictions on key performance indicators for fl= ows, such as back pressure and flow rate, to help administrators improve op= erational management of NiFi clusters. This framework could offer several k= ey features: > - Provide a flexible internal analytics engine and model api which suppor= ts the addition of or enhancement to onboard models > - Support integration of remote or cloud based ML models > - Support both traditional and online (incremental) learning methods > - Provide support for model caching (perhaps later inclusion into a model= repository or registry) > - UI enhancements to display prediction information either in existing su= mmary data, new data visualizations, or directly within the flow/canvas (wh= ere applicable) > For an initial target we thought that back pressure prediction would be a= good starting point for this initiative, given that back pressure detectio= n is a key indicator of flow performance and many of the metrics currently = available would provide enough data points to create a reasonable performin= g model. We have some ideas on how this could be achieved however we wanted= to discuss this more with the community to get thoughts about tackling thi= s work, especially if there are specific use cases or other factors that sh= ould be considered.{noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)