nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yolanda M. Davis (Jira)" <j...@apache.org>
Subject [jira] [Updated] (NIFI-6801) Connection analytics model should be unique across connections
Date Wed, 23 Oct 2019 15:02:00 GMT

     [ https://issues.apache.org/jira/browse/NIFI-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yolanda M. Davis updated NIFI-6801:
-----------------------------------
    Status: Patch Available  (was: In Progress)

> Connection analytics model should be unique across connections
> --------------------------------------------------------------
>
>                 Key: NIFI-6801
>                 URL: https://issues.apache.org/jira/browse/NIFI-6801
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.10.0
>            Reporter: Yolanda M. Davis
>            Assignee: Yolanda M. Davis
>            Priority: Major
>             Fix For: 1.11.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Connections should have Status Analytics Model instances that are unique for each connection
such that calculations for the model can be referred to or updated when new observations are
available.  Currently the model appears to be shared across connection objects which leads
to erroneous results, especially when a connection does not have enough observations to update
the model (since the older values of a previous connection may still be present). 
> This behavior can be seen when debugging is enabled for analytics and noting differences
between a Timer Driven Thread execution of predictions vs the NiFi Web Server thread predictions
for multiple connections (a flow with one connection would not exhibit this issue).  The
Time Driven thread may not retrieve enough connection status observations to refresh the model,
and when that occurs, it may still contain calculations from a previous model and use that
for predictions. Observers may see behavior where web server  and timer driven predictions
only have similar or matching predictions for one connection but others may be mismatched,
such as in the below (where connection BBBB  has similar prediction between threads but AAAA,
and CCCC do not): 
> {code:java}
> 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess
>>>> AAAA: predicted time to COUNT backpressure = 00:44:13 / 2653416
> 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess
>>>> AAAA: predicted time to BYTES backpressure = 01:09:13 / 4153511
> 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess
>>>> CCCC: predicted time to COUNT backpressure = 00:32:31 / 1951609
> 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess
>>>> CCCC: predicted time to BYTES backpressure = 00:32:28 / 1948504
> 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess
>>>> BBBB: predicted time to COUNT backpressure = 07:29:14 / 26954837
> 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23] o.a.nifi.reporting.StandardEventAccess
>>>> BBBB: predicted time to BYTES backpressure = 05:24:56 / 19496954 
> 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess
>>>> AAAA: predicted time to COUNT backpressure = 04:08:36 / 14916150
> 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess
>>>> AAAA: predicted time to BYTES backpressure = 04:17:49 / 15469243
> 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess
>>>> CCCC: predicted time to COUNT backpressure = 01:13:42 / 4422829
> 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess
>>>> CCCC: predicted time to BYTES backpressure = 00:57:15 / 3435109
> 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess
>>>> BBBB: predicted time to COUNT backpressure = 07:28:46 / 26926608
> 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2] o.a.nifi.reporting.StandardEventAccess
>>>> BBBB: predicted time to BYTES backpressure = 05:24:28 / 19468725
> {code}
>  A workaround for this is to increase the nifi.analytics.query.interval property or
decrease the nifi.components.status.snapshot.frequency property to ensure obtaining enough
observations for a refresh. However an appropriate fix is to ensure model instances are unique
for each connection and to ensure that by default timer driven threads will obtain enough
observations for predictions.
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message