From issues-return-84604-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Mon Sep 9 15:39:20 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id AC87B18065C for ; Mon, 9 Sep 2019 17:39:19 +0200 (CEST) Received: (qmail 80262 invoked by uid 500); 9 Sep 2019 15:39:17 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 79950 invoked by uid 99); 9 Sep 2019 15:39:16 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Sep 2019 15:39:16 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id BD89BE3167 for ; Mon, 9 Sep 2019 15:39:15 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id EA5D57809A2 for ; Mon, 9 Sep 2019 15:39:14 +0000 (UTC) Date: Mon, 9 Sep 2019 15:39:14 +0000 (UTC) From: "ASF subversion and git services (Jira)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/NIFI-6510?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16925= 821#comment-16925821 ]=20 ASF subversion and git services commented on NIFI-6510: ------------------------------------------------------- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads= /master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=3Dnifi.git;h=3D8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpres= sure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventA= ccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection = versions * NIFI-6510 Remove redundant connection prediction interfaces as we can jus= t use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces= as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populat= ed * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interf= aces for models, refactored callers to use StatusAnalytics objects with con= nection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction p= ercentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also = changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected va= riable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use= non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regress= ion model based on r-squared values Added ordinary least squares model, which truly uses multivariable regressi= on. Refactor of interfaces to include more general interface for variate mo= dels (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for con= nection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics obj= ect. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.prope= rties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed = unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. = Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the = UI * Add multi-line tooltips with detail for connection queue back pressure gr= aphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection= summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the = connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable/disable property * NIFI-6510 - UI: handle the scenario where backpressure predictions are di= sabled (#3685) * NIFI-6510 - admin guide updates to further describe model functionality * NIFI-6510 - code quality fixes (if statement and constructor) * NIFI-6510 - log warnings when properties could not be retrieved. fixed in= correct property retrieval for score threshold * NIFI-6510 Extract out predictions into their own DTO * NIFI-6510 Optimize imports * NIFI-6510 Fix formatting * NIFI-6510 Optimize imports * NIFI-6510 Optimize imports * NIFI-6510 - Notice updates for Commons math and Caffeine * NIFI-6510 - UI updates to account for minor API changes for back pressure= predictions (#3697) * NIFI-6510 - Fix issue displaying estimated time to back pressure in conne= ction summary table when only one of the predictions is known. Signed-off-by: Matthew Burgess This closes #3705 * NIFI-6510 Rip out useless members * NIFI-6510 - dto updates to check for -1 value * NIFI-6510 - checkstyle fix * NIFI-6510 - rolled back last change and applied minNonNegative method * NIFI-6510 Rip out useless members > Predictive Analytics for NiFi Metrics > ------------------------------------- > > Key: NIFI-6510 > URL: https://issues.apache.org/jira/browse/NIFI-6510 > Project: Apache NiFi > Issue Type: Improvement > Reporter: Andrew Christianson > Assignee: Yolanda M. Davis > Priority: Major > Fix For: 1.10.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > From Yolanda's email to the list: > =C2=A0 > {noformat} > Currently NiFi has lots of metrics available for areas including jvm and = flow component usage (via component status) as well as provenance data whic= h NiFi makes available either through the UI or reporting tasks (for consum= ption by other systems). Past discussions in the community cite users shipp= ing this data to applications such as Prometheus, ELK stacks, or Ambari met= rics for further analysis in order to capture/review performance issues, de= tect anomalies, and send alerts or notifications. These systems are efficie= nt in capturing and helping to analyze these metrics however it requires cu= stomization work and knowledge of NiFi operations to provide meaningful ana= lytics within a flow context. > In speaking with Matt Burgess and Andy Christianson on this topic we feel= that there is an opportunity to introduce an analytics framework that coul= d provide users reasonable predictions on key performance indicators for fl= ows, such as back pressure and flow rate, to help administrators improve op= erational management of NiFi clusters. This framework could offer several k= ey features: > - Provide a flexible internal analytics engine and model api which suppor= ts the addition of or enhancement to onboard models > - Support integration of remote or cloud based ML models > - Support both traditional and online (incremental) learning methods > - Provide support for model caching (perhaps later inclusion into a model= repository or registry) > - UI enhancements to display prediction information either in existing su= mmary data, new data visualizations, or directly within the flow/canvas (wh= ere applicable) > For an initial target we thought that back pressure prediction would be a= good starting point for this initiative, given that back pressure detectio= n is a key indicator of flow performance and many of the metrics currently = available would provide enough data points to create a reasonable performin= g model. We have some ideas on how this could be achieved however we wanted= to discuss this more with the community to get thoughts about tackling thi= s work, especially if there are specific use cases or other factors that sh= ould be considered.{noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)