nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arpad Boda <ab...@cloudera.com.INVALID>
Subject Re: [DISCUSS] Predictive Analytics for NiFi Metrics
Date Wed, 31 Jul 2019 09:31:04 GMT
Craig,

OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and Modbus (
https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the way for
MiNiFi c++, hopefully both will be part of next release (0.7.0).
It's gonna be legen... wait for it! :)

Regards,
Arpad

On Wed, Jul 31, 2019 at 2:30 AM Craig Knell <craig.knell@gmail.com> wrote:

> Hi Folks
>
> That's our use case now.  All our Models are run in python.
> Currently we send events to the ML via http, although this is not optimal
>
> Our use case is edge ML where we want a light weight wrapper for
> Python code base.
> Jython however does not work with the code base
> I'm think of changing the interface to some thing like REDIS for pub/sub
> Id also like this to be a push deployment via minifi
>
> Also support for sensors via protocols via Modbus and OPC would be great
>
> Craig
>
> On Wed, Jul 31, 2019 at 1:43 AM Joe Witt <joe.witt@gmail.com> wrote:
> >
> > Definitely something that I think would really help the community.  It
> > might make sense to frame/structure these APIs such that an internal
> option
> > could be available to reduce dependencies and get up and running but that
> > also just as easily a remote implementation where the engine lives and is
> > managed externally could also be supported.
> >
> > Thanks
> >
> >
> > On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto <alopresto@apache.org>
> wrote:
> >
> > > Yolanda,
> > >
> > > I think this sounds like a great idea and will be very useful to
> > > admins/users, as well as enabling some interesting next-level
> functionality
> > > and insight generation. Thanks for putting this out there.
> > >
> > > Andy LoPresto
> > > alopresto@apache.org
> > > alopresto.apache@gmail.com
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > > On Jul 30, 2019, at 5:55 AM, Yolanda Davis <
> yolanda.m.davis@gmail.com>
> > > wrote:
> > > >
> > > > Hello Everyone,
> > > >
> > > > I wanted to reach out to the community to discuss potentially
> enhancing
> > > > NiFi to include predictive analytics that can help users assess and
> > > predict
> > > > NiFi behavior and performance. Currently NiFi has lots of metrics
> > > available
> > > > for areas including jvm and flow component usage (via component
> status)
> > > as
> > > > well as provenance data which NiFi makes available either through
> the UI
> > > or
> > > > reporting tasks (for consumption by other systems). Past discussions
> in
> > > the
> > > > community cite users shipping this data to applications such as
> > > Prometheus,
> > > > ELK stacks, or Ambari metrics for further analysis in order to
> > > > capture/review performance issues, detect anomalies, and send alerts
> or
> > > > notifications.  These systems are efficient in capturing and helping
> to
> > > > analyze these metrics however it requires customization work and
> > > knowledge
> > > > of NiFi operations to provide meaningful analytics within a flow
> context.
> > > >
> > > > In speaking with Matt Burgess and Andy Christianson on this topic we
> feel
> > > > that there is an opportunity to introduce an analytics framework that
> > > could
> > > > provide users reasonable predictions on key performance indicators
> for
> > > > flows, such as back pressure and flow rate, to help administrators
> > > improve
> > > > operational management of NiFi clusters.  This framework could offer
> > > > several key features:
> > > >
> > > >   - Provide a flexible internal analytics engine and model api which
> > > >   supports the addition of or enhancement to onboard models
> > > >   - Support integration of remote or cloud based ML models
> > > >   - Support both traditional and online (incremental) learning
> methods
> > > >   - Provide support for model caching  (perhaps later inclusion into
> a
> > > >   model repository or registry)
> > > >   - UI enhancements to display prediction information either in
> existing
> > > >   summary data, new data visualizations, or directly within the
> > > flow/canvas
> > > >   (where applicable)
> > > >
> > > > For an initial target we thought that back pressure prediction would
> be a
> > > > good starting point for this initiative, given that back pressure
> > > detection
> > > > is a key indicator of flow performance and many of the metrics
> currently
> > > > available would provide enough data points to create a reasonable
> > > > performing model.  We have some ideas on how this could be achieved
> > > however
> > > > we wanted to discuss this more with the community to get thoughts
> about
> > > > tackling this work, especially if there are specific use cases or
> other
> > > > factors that should be considered.
> > > >
> > > > Looking forward to everyone's thoughts and input.
> > > >
> > > > Thanks,
> > > >
> > > > -yolanda
> > > >
> > > > --
> > > > yolanda.m.davis@gmail.com
> > > > @YolandaMDavis
> > >
> > >
>
>
>
> --
> Regards
>
> Craig Knell
> Mobile: +61 402 128 615
> Skype: craigknell
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message