nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: MiNiFi commits for h2o
Date Mon, 04 May 2020 16:38:13 GMT
Yep got ya.  I'm talking about lines like

https://github.com/apache/nifi-minifi-cpp/commit/6e5f96518764df7791519c0ee625a94a207ddc69#diff-fa43d8bfc508fc5db1c2a320e3d03cb5R33

Does that seem fine to you?

On Mon, May 4, 2020 at 12:29 PM Marc Parisi <phrocker@apache.org> wrote:

> Hey Joe,
>
> "Yeah for the three/commit James replied on in that link - i dont see how
> we
> can keep those.  They import from something that I doubt very much is
> compatible with ALv2/ASF guidelines."
>
> Sorry for confusing things but my statement was referring to the MiNiFi C++
> Python processors that don't import as java would. I've asked the original
> author to chime in here so that he can comment on the MiNiFi C++
> processors, especially on whether or not he feels the processors should
> stay and the other concerns raised. I'll continue to monitor this and
> create those follow up tickets when I am able to.
>
> Thanks,
> Marc
>
> On Mon, May 4, 2020 at 12:13 PM Joe Witt <joe.witt@gmail.com> wrote:
>
> > Yeah for the three/commit James replied on in that link - i dont see how
> we
> > can keep those.  They import from something that I doubt very much is
> > compatible with ALv2/ASF guidelines.
> >
> > On Mon, May 4, 2020 at 12:01 PM Marc Parisi <marc@parisi.io> wrote:
> >
> > > Afternoon Joe,
> > >
> > > "So in the case of minificpp we are certain there is no L&N concern
> > > regarding either our source OR convenience binary?"
> > >
> > > I believe that to be the case; however, per
> > > https://github.com/apache/nifi/pull/4242#issuecomment-622655276 I
> think
> > > that maybe this is moot and the author can convert those processors in
> > > MiNiFI to use that open source python lib. I'm not sure if he's on this
> > > chain so I will follow up on the MiNiFi tickets to ensure that is the
> > case
> > > when I am able to log in again ( have to fix some account issue ).
> > >
> > > "Lastly, why are the JIRAs still open if this is merged?  Can you
> please
> > > close them and assign to the proper fix version if these are 'done'?"
> > >
> > > Sorry, when I am able to log in ( account issue ) and make the above
> > > comment I will close the tx. I would prefer the fix be a follow on so
> > this
> > > is all tracked. The ticket fell off the radar ( as did the account
> issue,
> > > to be frank ), so when that is resolved I will take these actions. You
> > know
> > > me Joe, I'm like a dog who sees a squirrel, but I've put a note on my
> > > laptop to serve as a reminder.
> > >
> > > Thanks,
> > > Marc
> > >
> > > On Mon, May 4, 2020 at 11:49 AM Joe Witt <joe.witt@gmail.com> wrote:
> > >
> > > > Marc
> > > >
> > > > I'm not opposed to it staying if there is zero license issue.  I only
> > > > mentioned the maintenance as a 'additional thing to consider' - that
> is
> > > > more of a discussion item.  L&N isn't discussion - it is hard rule.
> > > >
> > > > Ok so in the minificpp case you're saying "there is no source or
> binary
> > > > dependency (that we'd package)" that isn't entirely ALv2/ASF
> compatible
> > > and
> > > > so if people want to use this they can do so by adding in the 'h2o'
> > bits
> > > at
> > > > their own effort, right?  In this line of thinking I still don't love
> > it
> > > > but it is arguably not a lot different that our components which talk
> > to
> > > > services you have to have a paid or otherwise agreement with like
> Cloud
> > > > services offering by AWS, Azure, GCP, etc..
> > > >
> > > > So in the case of minificpp we are certain there is no L&N concern
> > > > regarding either our source OR convenience binary?
> > > >
> > > > Lastly, why are the JIRAs still open if this is merged?  Can you
> please
> > > > close them and assign to the proper fix version if these are 'done'?
> > > >
> > > > Thanks
> > > >
> > > > On Mon, May 4, 2020 at 11:46 AM Marc Parisi <phrocker@apache.org>
> > wrote:
> > > >
> > > > > "We should find a way for vendors to offer extension points like
> this
> > > > > without having to take on the burden of maintenance. How can we
> > > possibly
> > > > do
> > > > > this well?  We're learning this lesson the hard way in NiFi itself
> > and
> > > > this
> > > > > is why the registry is being formulated."
> > > > >
> > > > > I think there is a fine line when walking vendor tie-in. Having the
> > > > > registry allows for an arm's length agreement with the Apache
> > > community.
> > > > I
> > > > > know others will disagree but my opinion is that extension points
> > that
> > > > > don't introduce licensing concerns from libraries should be fair
> game
> > > but
> > > > > the fact that a paid product requires a license for that code to
be
> > > > useful
> > > > > should not be a non-starter -- if that makes sense. I believe I
> > stated,
> > > > "I
> > > > > don't have a strong argument" as I don't really have a strong
> > opinion (
> > > > and
> > > > > certainly can be wrong too ) -- so I'm interested in hearing others
> > in
> > > > > regards to this.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 4, 2020 at 11:38 AM Marc Parisi <phrocker@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi Joe,
> > > > > >
> > > > > > When merging I did not use the NiFi processors to test as I
> already
> > > > have
> > > > > > tooling around H20 driven ML in my home. While I don't admittedly
> > use
> > > > it
> > > > > > very often, I thought the python processors were quite cool
and
> > could
> > > > > > certainly be useful for others. My rationale is that the
> > dependencies
> > > > are
> > > > > > not built into source or binary for MiNiFi C++. I would agree
on
> > the
> > > > Java
> > > > > > side that this would be unavoidable. While it would be
> preferential
> > > to
> > > > > have
> > > > > > the NiFi analogue I do not think it would be required when
> > > considering
> > > > > the
> > > > > > MiNiFi C++ processors; however, James should be the arbiter
of
> that
> > > > > > decision.
> > > > > >
> > > > > > While I disagree that the MiNiFi portion should be removed,
I
> don't
> > > > have
> > > > > a
> > > > > > strong argument to keep or remove the MiNiFI C++ code so I'd
be
> > happy
> > > > to
> > > > > > revert if the community feels the need to.
> > > > > >
> > > > > > Thanks,
> > > > > > Marc
> > > > > >
> > > > > > On Mon, May 4, 2020 at 10:50 AM Joe Witt <joe.witt@gmail.com>
> > wrote:
> > > > > >
> > > > > >> Team
> > > > > >>
> > > > > >> The below two commits raise serious concerns.
> > > > > >>
> > > > > >> I want to be clear here and point out that h2o is cool.
 Having
> > such
> > > > > >> integration is a neat concept and idea and one that certainly
> > > warrants
> > > > > >> determination on how best to do so.
> > > > > >>
> > > > > >> My issue is with these two commits as it relates to licensing
> and
> > > > > >> maintenance.
> > > > > >>
> > > > > >> License:
> > > > > >> We should support vendors like this wanting to bring their
> > > > capabilities
> > > > > >> into NiFi generally sure. But the licensing and mode of
use is
> > > > critical.
> > > > > >> In talking about this with the contributor for NiFi as well
it
> is
> > > > clear
> > > > > >> that at least some or important portions of this require
the
> user
> > to
> > > > > have
> > > > > >> a
> > > > > >> 'driverless ai license' so they can include their jar or
build
> > > > pipelines
> > > > > >> or
> > > > > >> whatnot.  Thus it isn't usable without that first. So it
might
> be
> > > the
> > > > > case
> > > > > >> that this stuff is source dependent only on ASF compatible
> > licenses
> > > in
> > > > > >> terms of source - but it certainly doesn't seem to be true
in
> > terms
> > > of
> > > > > >> binary dependencies. Where in the PR or JIRA is there any
> > discussion
> > > > or
> > > > > >> review of the licensing?  I see plenty from James to believe
> this
> > > > needs
> > > > > to
> > > > > >> be reverted.  Any contrary info?
> > > > > >> https://github.com/apache/nifi/pull/4242#issuecomment-622654527
> > > > > >>
> > > > > >>
> > > > > >> Maintenance:
> > > > > >> We should find a way for vendors to offer extension points
like
> > this
> > > > > >> without having to take on the burden of maintenance. How
can we
> > > > possibly
> > > > > >> do
> > > > > >> this well?  We're learning this lesson the hard way in NiFi
> itself
> > > and
> > > > > >> this
> > > > > >> is why the registry is being formulated.
> > > > > >>
> > > > > >> JIRA/hygiene:
> > > > > >> https://issues.apache.org/jira/browse/MINIFICPP-1199
> > > > > >> and
> > > > > >> https://issues.apache.org/jira/browse/MINIFICPP-1201
> > > > > >> Both are open.  Yet we've merged commits that claim to be
> against
> > > > each.
> > > > > >>
> > > > > >> I believe both of these commits need to be reverted as I
do not
> > > > believe
> > > > > >> the
> > > > > >> licensing considerations have been addressed.  I'd like
to see
> > > > > discussion
> > > > > >> on the above maintenance concern as well but that is less
> > pressing.
> > > > > >>
> > > > > >>
> > > > > >> Thanks
> > > > > >> Joe
> > > > > >>
> > > > > >>
> > > > > >> commit 7206c62240647520cf35649868d5d87903a256c2
> > > > > >> Author: James Medel <jamesmedel94@gmail.com>
> > > > > >> Date:   Wed Apr 29 12:38:04 2020 -0700
> > > > > >>
> > > > > >>     MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi
(#766)
> > > > > >>
> > > > > >>     MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi
(#766)
> > > > > >>
> > > > > >> commit 6e5f96518764df7791519c0ee625a94a207ddc69
> > > > > >> Author: James Medel <jamesmedel94@gmail.com>
> > > > > >> Date:   Wed Apr 29 12:37:00 2020 -0700
> > > > > >>
> > > > > >>     MINIFI-1199: Integrate H2O Driverless AI PSP in MiNiFi
> (#763)
> > > > > >>
> > > > > >>     * MINIFI-1199: Integrate H2O Driverless AI in MiNiFi
> > > > > >>
> > > > > >>     MiNiFi C++ and H2O Driverless AI Integration via Custom
> Python
> > > > > >> Processors:
> > > > > >>     Integrates MiNiFi C++ with H2O's Driverless AI by Using
> > > Driverless
> > > > > >> AI's
> > > > > >> Python Scoring Pipeline and MiNiFi's Custom Python Processors.
> > Uses
> > > > the
> > > > > >> Python Processors to execute the Python Scoring Pipeline
scorer
> to
> > > do
> > > > > >> batch
> > > > > >> scoring and real-time scoring for one or more predicted
labels
> on
> > > test
> > > > > >> data
> > > > > >> in the incoming flow file content. I would like to contribute
my
> > > > > >> processors
> > > > > >> to MiNiFi C++ as a new feature.
> > > > > >>
> > > > > >>     * Update H2oPspScoreBatches processor
> > > > > >>
> > > > > >>     This update includes passing "index=False" to
> > > > > >> "batch_scores_df.to_string(index=False)". By updating this
line
> of
> > > > code,
> > > > > >> we
> > > > > >> tell pandas to not include the DataFrame index when converting
> the
> > > > > >> DataFrame to a string. The reason for this update is that
we
> don't
> > > > want
> > > > > >> this extra column pandas tries to add to our predictions
frame,
> we
> > > > only
> > > > > >> want the predictions. Thus, later when the predictions get
saved
> > to
> > > a
> > > > > csv
> > > > > >> file, it will only include the predictions.
> > > > > >>
> > > > > >>     * Moved ConvertDsToCsv to h2o base dir
> > > > > >>
> > > > > >>     Since this ConvertDsToCsv python processor is used by
the
> > > > > >>     Python Scoring Pipeline and MOJO Scoring Pipeline
> processors,
> > > > > >>     I moved ConvertDsToCsv to h2o base dir for easier access
to
> > it.
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message