nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: NIFI-3641
Date Wed, 07 Jun 2017 00:15:01 GMT
Mike, Tony,

This certainly sounds interesting and from reading through the
motivations/design behind it there are clearly some well thought out
reasons for it.

For site-to-site support I can see advantages for interoperability.
For other factors it would be good to identify limitations of the
current options (raw sockets/http) so that we can clearly and
measurable improve gaps for certain use cases.  Would be good to hear
any of those you have in mind.

Outside of site-to-site it seems like it could make sense for a
processor configured with a given gRPC/proto being able to talk to
another service to share data.  Is that planned as part of this effort
or is that just what the referenced JIRA was about?

In either case this seems like one heck of an initial area to
contribute to and a good discussion point!  Thanks

Joe

http://www.grpc.io/blog/principles

On Tue, Jun 6, 2017 at 7:21 PM, Tony Kurc <trkurc@gmail.com> wrote:
> Mike,
> I think what you're saying is you are debating two options:
>
> A) gRPC as a transport mechanism and support the deployment use cases from
> the HTTP s2s document [1] to include using nifi-specific peer selection in
> the client if the destination is a cluster.
> B) Building a different implementation with an additional deployment case,
> which is sending from a client (in the diagrams as NiFi site-to-site
> client) to a cluster which isn't NiFi and delegating peer selection "to
> gRPC" which lightens what the receiving cluster would have to implement?
>
> Sound pretty close to the decision you're looking for input on?
>
> 1.
> https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP%28S%29+as+a+transport+mechanism+for+Site-to-Site#SupportHTTP(S)asatransportmechanismforSite-to-Site-Deploymentexamples
>
> On Tue, Jun 6, 2017 at 2:28 PM, Michael Hogue <michael.p.hogue89@gmail.com>
> wrote:
>
>> All,
>>
>>    As an initial dive into nifi i elected to take a stab at NIFI-3641
>> <https://issues.apache.org/jira/browse/NIFI-3641> (in particular, the
>> site-to-site bit), but i've got a few high level questions before taking
>> off down a path.
>>
>>    Would it be more desirable to have a gRPC site-to-site mechanism or the
>> ability to generally interact with external gRPC services, such as tensor
>> flow, in a processor? That might help guide the approach I take in
>> addressing the issue. I'm not entirely sure it's possible to do both with
>> the same solution since the current site-to-site API assumes the other end
>> is a nifi. On top of that, gRPC provides a number of load balancing, remote
>> gRPC server (i.e. peer) selection, and serialization mechanisms that the
>> current site-to-site implementation does itself.
>>
>>    What i'm looking for here are some thoughts and/or guidance on a
>> recommended approach and intended goal.
>>
>> Thanks,
>> Mike
>>

Mime
View raw message