nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: NIFI-3641
Date Wed, 07 Jun 2017 02:57:57 GMT
Hi Mike,

I like the idea of adding gRPC as an option for NiFi to communicate
with other NiFi (s2s) or other server endpoint which can talk via
gRPC.

I had implemented HTTP for s2s before. It was not an easy task (at
least for me) to make the new protocol align with existing
terminology, behavior and the same level of support.
We need to implement both s2s client and server side and that would
require a huge effort.

I personally prefer starting with option 2 (enabling flow file sharing
to arbitrary external
services via gRPC). Probably start with a simple gRPC client processor
similar to InvokeHTTP.
Then we would expand our support by adding server side processor
similar to HandleHTTPRequest/Response.
After that, we could add support for gRPC way load-balancing, to
distribute requests among NiFi nodes in a cluster those are running
HandleGRPCRequest/Response.
At this point, we need a Load balancer described in this design document:
https://github.com/grpc/grpc/blob/master/doc/load-balancing.md

If we go through this route, we will have more detailed knowledge on
how gRPC works and more clear idea on how we can apply it to NiFi s2s
mechanism.

I maybe wrong since I just started reading about gRPC technology..

Thanks,
Koji

On Wed, Jun 7, 2017 at 11:29 AM, Michael Hogue
<michael.p.hogue89@gmail.com> wrote:
> Indeed, Tony. Thanks for clearing that up.
>
> The first option (gRPC for s2s) may offer an opportunity to leverage a
> library for some of the things you'd likely care about with distributed
> comms, such as load balancing, rather than implementing that ourselves.
> gRPC has pluggable load balancing, authentication, and transport protocols,
> so you're still free to provide your own implementation.
>
> I think the latter option (enabling flow file sharing to arbitrary external
> services via gRPC) may open a number of additional doors not previously
> available with tensorflow as the motivator.
>
> I'm happy to choose one of these things, but i thought it'd be wise to open
> the conversation to the dev list prior to starting.
>
> Thanks,
> Mike
>
> On Tue, Jun 6, 2017 at 8:22 PM Tony Kurc <trkurc@gmail.com> wrote:
>
>> Joe, the ticket [1] mentions tensorflow, implying, to some extent, that
>> option A from my email wouldn't cut the mustard.
>>
>> 1. https://issues.apache.org/jira/browse/NIFI-3641
>>
>> On Jun 6, 2017 8:15 PM, "Joe Witt" <joe.witt@gmail.com> wrote:
>>
>> > Mike, Tony,
>> >
>> > This certainly sounds interesting and from reading through the
>> > motivations/design behind it there are clearly some well thought out
>> > reasons for it.
>> >
>> > For site-to-site support I can see advantages for interoperability.
>> > For other factors it would be good to identify limitations of the
>> > current options (raw sockets/http) so that we can clearly and
>> > measurable improve gaps for certain use cases.  Would be good to hear
>> > any of those you have in mind.
>> >
>> > Outside of site-to-site it seems like it could make sense for a
>> > processor configured with a given gRPC/proto being able to talk to
>> > another service to share data.  Is that planned as part of this effort
>> > or is that just what the referenced JIRA was about?
>> >
>> > In either case this seems like one heck of an initial area to
>> > contribute to and a good discussion point!  Thanks
>> >
>> > Joe
>> >
>> > http://www.grpc.io/blog/principles
>> >
>> > On Tue, Jun 6, 2017 at 7:21 PM, Tony Kurc <trkurc@gmail.com> wrote:
>> > > Mike,
>> > > I think what you're saying is you are debating two options:
>> > >
>> > > A) gRPC as a transport mechanism and support the deployment use cases
>> > from
>> > > the HTTP s2s document [1] to include using nifi-specific peer selection
>> > in
>> > > the client if the destination is a cluster.
>> > > B) Building a different implementation with an additional deployment
>> > case,
>> > > which is sending from a client (in the diagrams as NiFi site-to-site
>> > > client) to a cluster which isn't NiFi and delegating peer selection "to
>> > > gRPC" which lightens what the receiving cluster would have to
>> implement?
>> > >
>> > > Sound pretty close to the decision you're looking for input on?
>> > >
>> > > 1.
>> > > https://cwiki.apache.org/confluence/display/NIFI/
>> > Support+HTTP%28S%29+as+a+transport+mechanism+for+Site-
>> > to-Site#SupportHTTP(S)asatransportmechanismforSite-
>> > to-Site-Deploymentexamples
>> > >
>> > > On Tue, Jun 6, 2017 at 2:28 PM, Michael Hogue <
>> > michael.p.hogue89@gmail.com>
>> > > wrote:
>> > >
>> > >> All,
>> > >>
>> > >>    As an initial dive into nifi i elected to take a stab at NIFI-3641
>> > >> <https://issues.apache.org/jira/browse/NIFI-3641> (in particular,
the
>> > >> site-to-site bit), but i've got a few high level questions before
>> taking
>> > >> off down a path.
>> > >>
>> > >>    Would it be more desirable to have a gRPC site-to-site mechanism
or
>> > the
>> > >> ability to generally interact with external gRPC services, such as
>> > tensor
>> > >> flow, in a processor? That might help guide the approach I take in
>> > >> addressing the issue. I'm not entirely sure it's possible to do both
>> > with
>> > >> the same solution since the current site-to-site API assumes the other
>> > end
>> > >> is a nifi. On top of that, gRPC provides a number of load balancing,
>> > remote
>> > >> gRPC server (i.e. peer) selection, and serialization mechanisms that
>> the
>> > >> current site-to-site implementation does itself.
>> > >>
>> > >>    What i'm looking for here are some thoughts and/or guidance on a
>> > >> recommended approach and intended goal.
>> > >>
>> > >> Thanks,
>> > >> Mike
>> > >>
>> >
>>

Mime
View raw message